Cap'n Proto serialization/RPC system - core tools and C++ library

Hello,

this patch adds support for observing availability of urgent/out-of-band data by adding a new promise to FdObserver. The patch itself is simple and without side effects: Unless OBSERVE_URGENT is passed when constructing the FdObserver object, no new code is run. The implementation mirrors what was already there for the POLLIN event.

Right, so... why do we need this? A little backstory: I gave Cap'n Proto a shot as RPC server for my home automation system, which controls various devices through a Raspberry Pi B+'s GPIO ports and a i2c GPIO multiplexer. Cap'n Proto did the job flawlessly! However, in order to receive interrupts from the i2c GPIO multiplexer, I have to test for the POLLPRI flag on the interrupt port's fd. Linux's GPIO abstraction layer sets POLLPRI when an edge change is detected on a GPIO port. Extending KJ's UnixEventPort implementation appeared to be the most simple way to achieve this goal and at the same time integrate with the RPC server's event loop. POLLPRI is also used by TCP and many drivers for specialized hardware (such as Bluetooth receivers.)

I would appreciate if this change could be merged so that I do not have to maintain a separate fork.

I have included a test which verifies that the new event works correctly with Linux's loopback TCP/IP out-of-band message implementation.

Cheers, Oliver

clang 8 build failed with error:

[ 15%] Building CXX object modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/encoding.c++:242:19: error: no matching conversion for functional-style cast from 'const char8_t [4]' to 'kj::StringPtr'
    result.addAll(StringPtr(u8"\ufffd"));
                  ^~~~~~~~~~~~~~~~~~~~
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const char8_t [4]' to 'const kj::StringPtr' for 1st argument
class StringPtr {
      ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const char8_t [4]' to 'kj::StringPtr' for 1st argument
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:73:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'decltype(nullptr)' (aka 'nullptr_t') for 1st argument
  inline StringPtr(decltype(nullptr)): content("", 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:74:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const char *' for 1st argument
  inline StringPtr(const char* value): content(value, strlen(value) + 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:141:20: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'ArrayPtr<const char>' for 1st argument
  inline constexpr StringPtr(ArrayPtr<const char> content): content(content) {}
                   ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:499:19: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const kj::String' for 1st argument
inline StringPtr::StringPtr(const String& value): content(value.cStr(), value.size() + 1) {}
                  ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:83:10: note: candidate template ignored: substitution failure [with T = char8_t [4]]: no matching function for call to 'instance'
  inline StringPtr(const T& t): StringPtr(t.c_str()) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:72:10: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
  inline StringPtr(): content("", 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:75:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
  inline StringPtr(const char* value, size_t size): content(value, size + 1) {
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:78:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
  inline StringPtr(const char* begin, const char* end): StringPtr(begin, end - begin) {}
         ^
1 error generated.
modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/build.make:257: recipe for target 'modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o' failed
make[2]: *** [modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o] Error 1

This is mostly copied from node-capnproto and async-unix. This commit enables integration with Cocoa's event loop (CFRunLoop).

Notes:

• I haven't committed any tests yet (however I have tried the AsyncIo::SimpleNetwork test with it and it works both when using wait, and running CFRunLoop normally)
• I haven't modified the makefiles (requires -framework CoreFoundation to be added to LDFLAGS for libkj-async)
• I've had to add a temporary AsyncCocoaIoContext struct as AsyncIoContext currently has a unixEventPort field

Is this something you would merge into kj? It would be useful for anyone that wants to integrate kj/capnproto into an OSX/iOS application, and for an Objective-C wrapper of capnproto.

Implements a $Cxx.omitSchemas annotation that strips schema information from the binary as discussed in https://github.com/capnproto/capnproto/issues/1498

👋 just found the latest release 0.9.0 does not compile on Mojave (macOS 10.14), error log is below:

In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/mutex.h:28:
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE'
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
                                                               ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                         ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                                                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN'
      uint column = KJ_CALLER_COLUMN())
                    ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN'
#define KJ_CALLER_COLUMN() __builtin_COLUMN()
                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
            ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor
  const char* fileName;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor
  const char* function;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor
  uint lineNumber;
       ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor
  uint columnNumber;
       ^
5 errors generated.
In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.c++:22:
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE'
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
                                                               ^
make[2]: *** [src/kj/CMakeFiles/kj.dir/mutex.c++.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                         ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                                                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN'
      uint column = KJ_CALLER_COLUMN())
                    ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN'
#define KJ_CALLER_COLUMN() __builtin_COLUMN()
                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
            ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor
  const char* fileName;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor
  const char* function;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor
  uint lineNumber;
       ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor
  uint columnNumber;
       ^
5 errors generated.
make[2]: *** [src/kj/CMakeFiles/kj.dir/source-location.c++.o] Error 1

The cmake command

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/capnp/0.9.0 -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_VERBOSE_MAKEFILE=ON -Wno-dev -DBUILD_TESTING=OFF -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk

relates to https://github.com/Homebrew/homebrew-core/pull/83289

This is based on #1434, but I went ahead and made KJ treat unexpected EOF as an error -- even with OpenSSL 1.x -- since this is technically important to prevent truncation attacks.

(It turns out that lots of apps than use OpenSSL 1.x are vulnerable to truncation attacks, which is why OpenSSL 3 decided to strengthen the error reporting to make it clearer that this situation should be an error, but this was considered a breaking change. So KJ is not alone in having this bug, but I went ahead and fixed it even when using OpenSSL 1.x.)

NOTE: I'm flying blind here as my local Debian machine is still on OpenSSL 1.x. Gonna do a little CI-driven-development. I'll mark this no longer a draft once I get CI passing...

Hello and thank you for this nice tool. In the way I understood the docs and cli help, I should be able to import files from outside the current schema folder by passing the -I flag to the compile command. This fails with error: Import failed: <path to file> Copying the exact same file right next to my schema file and changing the import path to non absolute compiles fine.

I now tried for some time to figure it out, but I am unable to.

Here is a minimal example:

> uname -srmo
Linux 6.1.1-arch1-1 x86_64 GNU/Linux

> capnp --version
Cap n Proto version 0.10.2

> pwd
/tmp/capnp

> tree
.
├── A
│   ├── a.capnp
│   ├── b.capnp
│   └── c.capnp
└── B
    └── b.capnp

3 directories, 4 files

> cat A/a.capnp
@0x8e4986f5ac37ebbe;

using Bar = import "/tmp/capnp/B/b.capnp";

struct A {
	bs @0 :List(Bar.B);
}

> capnp compile -oc++ --verbose --import-path=/tmp/capnp/B A/a.capnp
A/a.capnp:3:20-42: error: Import failed: /tmp/capnp/B/b.capnp
A/a.capnp:6:14-17: error: Not defined: Bar
A/a.capnp:6:9-13: error: 'List' requires exactly one parameter.

> cat /tmp/capnp/B/b.capnp
@0xb7169ccabccfe69e;

struct B {
    name @0 :Text;
}

> ls -l
total 0
drwxr-xr-x 2 user user 100 Dec 27 23:55 A
drwxr-xr-x 2 user user  60 Dec 27 23:27 B

> ls -l A
total 12
-rw-r--r-- 1 user user 101 Dec 28 00:04 a.capnp
-rw-r--r-- 1 user user  54 Dec 27 23:29 b.capnp
-rw-r--r-- 1 user user  80 Dec 28 00:04 c.capnp

> ls -l B
total 4
-rw-r--r-- 1 user user 54 Dec 27 23:23 b.capnp

> cat A/c.capnp
@0x8e4986f5ac37ebbe;

using import "b.capnp".B;

struct A {
	bs @0 :List(B);
}

> capnp compile -oc++ --verbose --import-path=/tmp/capnp/B A/c.capnp

> diff A/b.capnp B/b.capnp

Test Plan

In internal repo for ASAN:

$ bazel test //src/ew/tests:sockets/[email protected]

Capnproto-specific test will come in a separate PR.

When bootstrapping a new Cap'n Proto implementation in another language, it can be useful to make the first code-generator in the same language as the language you're planning to write the code generator in (of course). But until the schema.capnp file has been bootstrapped at least once for the library being developed, this is difficult.

You may bootstrap a simple compiler in another language like C++, but then you won't be able to reuse any of the code and be forced to rewrite from scratch in the target language once up and running.

It'd be useful if another included output format existed like capnpc-capnp, to dump the input code generator request in another format that's easier to bootstrap in the target language. JSON is decently popular enough that it's well supported basically everywhere.

This PR adds capnpc-json, an included output plugin that simply dumps the entire code generator request in JSON format to stdout.

Hi!

I was investigating some unrelated issues with ASAN lately and in order to better understand real-world usages of swapcontext had a look at what you guys do.

First, it seems to me that this https://github.com/capnproto/capnproto/blob/fd508f325ed65f9d6cdf10a0f4511aa53b5659fc/c%2B%2B/src/kj/async.c%2B%2B#L1451-L1456 might lead to problems: https://godbolt.org/z/Ec5dd7a3f. Asan uses uc_stack.ss_size and uc_stack.ss_sp to know what and how much of memory to unpoison, so this hack tricks it.
Second, Asan doesn't intercept setcontext (but it does work with swapcontext), thus it doesn't unpoison memory whatsover: https://godbolt.org/z/da4bfE1ze
Third, this https://github.com/capnproto/capnproto/blob/fd508f325ed65f9d6cdf10a0f4511aa53b5659fc/c%2B%2B/src/kj/async.c%2B%2B#L1684-L1686 indeed doesn't leak fake stacks - it exhausts the mechanism instead. You see, there are only so much of fake stacks to use per thread: https://github.com/google/sanitizers/wiki/AddressSanitizerUseAfterReturn#memory-consumption, thus "leaking" them leads to this

If the given size class is fully used, the allocator will return 0 and thus regular stack will be used (i.e. stack-use-after-return detection will not work for the given function call)

which might render stack-use-after-return not working at all in later usages.

P.S. I am not at all familiar with the codebase, so could misinterpret something here and there.

P.P.S. After rereading this i realized it might sound a bit harsh, but definitely no offense intended - just trying to make an already great library a bit better, :heart:

list.h conditionally includes compat/std-iterator.h, but this is not included in the bazel library. This gives an error when KJ_STD_COMPAT is defined, since the file is not in the sandbox.