Cap'n Proto serialization/RPC system - core tools and C++ library

Hello,

this patch adds support for observing availability of urgent/out-of-band data by adding a new promise to FdObserver. The patch itself is simple and without side effects: Unless OBSERVE_URGENT is passed when constructing the FdObserver object, no new code is run. The implementation mirrors what was already there for the POLLIN event.

Right, so... why do we need this? A little backstory: I gave Cap'n Proto a shot as RPC server for my home automation system, which controls various devices through a Raspberry Pi B+'s GPIO ports and a i2c GPIO multiplexer. Cap'n Proto did the job flawlessly! However, in order to receive interrupts from the i2c GPIO multiplexer, I have to test for the POLLPRI flag on the interrupt port's fd. Linux's GPIO abstraction layer sets POLLPRI when an edge change is detected on a GPIO port. Extending KJ's UnixEventPort implementation appeared to be the most simple way to achieve this goal and at the same time integrate with the RPC server's event loop. POLLPRI is also used by TCP and many drivers for specialized hardware (such as Bluetooth receivers.)

I would appreciate if this change could be merged so that I do not have to maintain a separate fork.

I have included a test which verifies that the new event works correctly with Linux's loopback TCP/IP out-of-band message implementation.

Cheers, Oliver

clang 8 build failed with error:

[ 15%] Building CXX object modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/encoding.c++:242:19: error: no matching conversion for functional-style cast from 'const char8_t [4]' to 'kj::StringPtr'
    result.addAll(StringPtr(u8"\ufffd"));
                  ^~~~~~~~~~~~~~~~~~~~
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const char8_t [4]' to 'const kj::StringPtr' for 1st argument
class StringPtr {
      ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const char8_t [4]' to 'kj::StringPtr' for 1st argument
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:73:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'decltype(nullptr)' (aka 'nullptr_t') for 1st argument
  inline StringPtr(decltype(nullptr)): content("", 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:74:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const char *' for 1st argument
  inline StringPtr(const char* value): content(value, strlen(value) + 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:141:20: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'ArrayPtr<const char>' for 1st argument
  inline constexpr StringPtr(ArrayPtr<const char> content): content(content) {}
                   ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:499:19: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const kj::String' for 1st argument
inline StringPtr::StringPtr(const String& value): content(value.cStr(), value.size() + 1) {}
                  ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:83:10: note: candidate template ignored: substitution failure [with T = char8_t [4]]: no matching function for call to 'instance'
  inline StringPtr(const T& t): StringPtr(t.c_str()) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:72:10: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
  inline StringPtr(): content("", 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:75:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
  inline StringPtr(const char* value, size_t size): content(value, size + 1) {
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:78:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
  inline StringPtr(const char* begin, const char* end): StringPtr(begin, end - begin) {}
         ^
1 error generated.
modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/build.make:257: recipe for target 'modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o' failed
make[2]: *** [modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o] Error 1

This is mostly copied from node-capnproto and async-unix. This commit enables integration with Cocoa's event loop (CFRunLoop).

Notes:

• I haven't committed any tests yet (however I have tried the AsyncIo::SimpleNetwork test with it and it works both when using wait, and running CFRunLoop normally)
• I haven't modified the makefiles (requires -framework CoreFoundation to be added to LDFLAGS for libkj-async)
• I've had to add a temporary AsyncCocoaIoContext struct as AsyncIoContext currently has a unixEventPort field

Is this something you would merge into kj? It would be useful for anyone that wants to integrate kj/capnproto into an OSX/iOS application, and for an Objective-C wrapper of capnproto.

👋 just found the latest release 0.9.0 does not compile on Mojave (macOS 10.14), error log is below:

In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/mutex.h:28:
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE'
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
                                                               ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                         ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                                                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN'
      uint column = KJ_CALLER_COLUMN())
                    ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN'
#define KJ_CALLER_COLUMN() __builtin_COLUMN()
                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
            ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor
  const char* fileName;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor
  const char* function;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor
  uint lineNumber;
       ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor
  uint columnNumber;
       ^
5 errors generated.
In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.c++:22:
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE'
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
                                                               ^
make[2]: *** [src/kj/CMakeFiles/kj.dir/mutex.c++.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                         ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                                                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN'
      uint column = KJ_CALLER_COLUMN())
                    ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN'
#define KJ_CALLER_COLUMN() __builtin_COLUMN()
                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
            ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor
  const char* fileName;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor
  const char* function;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor
  uint lineNumber;
       ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor
  uint columnNumber;
       ^
5 errors generated.
make[2]: *** [src/kj/CMakeFiles/kj.dir/source-location.c++.o] Error 1

The cmake command

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/capnp/0.9.0 -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_VERBOSE_MAKEFILE=ON -Wno-dev -DBUILD_TESTING=OFF -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk

relates to https://github.com/Homebrew/homebrew-core/pull/83289

This does the following:

• Adds a new general function ConnectionReceiver::run() to facilitate an async accept loop.
• Adds a queue and a TaskSet to the TlsConnectionReceiver to accept on its inner ConnectionReceiver without blocking on SSL_accept().
• Adds several new tests for the TlsConnectionReceiver.
• Converts several other TLS tests to use convenience helpers.

This PR closes #523 by implementing a set of CMake package configuration files for the autotools build. Here are some of the more important details:

• The implementation re-uses the CapnProtoConfig.cmake.in file from the CMake build and adopts the same version compatibility policy.

• The implementation generates imported CMake targets for each .pc file installed whose main library file (.so or .a) exists. Listing a new .pc file in configure.ac should be sufficient to enable its imported target in the installed CMake package config.

• I made sure the installed package config files could handle a lite mode installation as well as a full installation. If it only finds capnp and kj libraries but no others, then it won't create imported executable targets, and the user will need to specify capnp and capnpc-c++ executable locations on the command line.

• I added CAPNP_-prefixed variables to CapnProtoConfig.cmake.in to be backwards compatible with FindCapnProto.cmake. (I believe this PR obviates FindCapnProto.cmake.)

• I changed the CI quick test to build the samples using CMake rather than manually, which gives us an implicit test of the new package config files.

@pqu, @kentonv, would you mind reviewing when you have some time? Note that this is my first foray into anything autotools-related. @Conan-Kudo and @thomaslee I'd welcome your feedback, too.

It might be useful to consumers to be able to poll for a bounded number of events in order to yield control of the thread over to other time-sensitive processing that is unable to run on another thread.

Added a test case to assert that events are only processed up to the max turn count.

Currently, when using the dynamic API via SchemaLoader, it is possible to convert dynamic objects obtained with a SchemaLoader to C++ objects (as long as the conversion is registered via loadCompiledTypeAndDependencies). However, the opposite direction, passing a dynamic object obtained from a statically-typed C++ object into structs / capabilities from the SchemaLoader, is currently not available.

This is because the SchemaLoader maintains its own copy of the schema, and therefore the C++ object's schema will compare unequal to the types requested by other dynamic structs and/or capabilities.

This pull requests adds a new method, SchemaLoader::getSchemaFor, which constructs a capnp::Schema object equivalent to the static schema in the loader. This is achieved simply by combining SchemaLoader::Impl::loadNative and SchemaLoader::Impl::makeBranded (which is done in a new method added to SchemaLoader:: Impl).

A small test is also added. I tested compilation and the test itself (it is part of capnp-tests the way I wrote it, right?) locally on MSVC 19, CLANG 15 (WSL) and GCC 9 (WSL).

Closed the old PR because I messed up the git history and didn't want to make review unnecessarily challenging.

This is going to be fairly dense, so I think it's better to break it into parts.

Phase 1 - Negotiating the Extension

This is all that's been done so far. Commits 1-3 represent a stage in extension negotiation, that is:

1. build the extension header for the request
2. parse the offers as the server + build a response
3. parse the servers response as the client

I added a test in commit 4, but since we're doing a lot of string parsing I think more tests are needed.

As of now, we:

• Drop all Sec-WebSocket-Extensions aside from permessage-deflate before sending the request to the server. The client retains information from its highest preference offer (ex. # bits to use for client_max_window_bits) so it can compare with what the server agrees to support.
• Parse permessage-deflate offers as the server, populate a struct CompressionParameters, then build a response with agreed parameters.
• Parse permessage-deflate in a response (as a client), potentially comparing the information the client retained with what the server agreed to support. The client also populates its own CompressionParameters.
• Handle invalid offers/agreements, i.e. decline offers as the server/fail connections as the client (could use tests here!)
• Handle whitespace between tokens correctly (implied *LWS) (could use tests here too, though the existing test somewhat covers)

One thing on my mind is I'm not sure what the best way to organize the code is. I've put new functions in an anonymous namespace so that they can't be accessed outside the file (we only really need these for this file anyways) since it looks like that's done elsewhere. AFAIK, this means we can only test the string parsing indirectly by passing Sec-WebSocket-Extensions headers in tests that create websocket client/servers.

Phase 2 - Basic Compression

Phase 3 - Using Compression Parameters

Implementing the idea of a realtime stream, as discussed in #976. ~The dynamic capabilities part and the tests are missing, but I figured that I would already ask for review with this milestone; no need to go further if I'm going in the wrong direction :slightly_smiling_face:.~ I implemented something for the dynamic capabilities, but honestly I don't really know how to use them so I mostly copied from the streams implementation.

Similar to the stream keyword, this can be used to signal a realtime stream:

write @0 (bytes :Data) -> realtime stream;

A "realtime stream" defines a Call that does not expect a Return (so the return type is always void). The idea is that some VatNetwork implementation could leverage this to send "unreliable" messages over e.g. UDP. Over a reliable transport (e.g. TCP), it just means that the RPC system does not keep track of those messages (which may be desirable in some situations?).

A "realtime call" is defined by setting isRealtime on the Call, and by using a QuestionId in the range [2^31, 2^32). The question id just increments from 2^31 until it reaches 2^32 - 1, at which point it starts from 2^31 again. It is assumed that this is safe because that huge interval will never be full of in-flight messages.

The approach in this PR is backwards-compatible. If the caller sends a realtime call, but the callee does not know about the concept of "realtime", then the callee will answer with a Return, and the caller will respond with a Finish (as usual). The only difference being that the caller, in this case, will not be keeping track of realtime messages.

Note that I tend to use "realtime" and "realtime stream" interchangeably (depending on what feels more readable). It feels right since a "realtime stream" is just a bunch of "realtime calls", without any flow control.

I tested the code with a small example and checked the traffic with wireshark and the capnp wireshark plugin (which works fine, but not perfectly, as can be seen in the screenshots below). The example sends 2 realtime calls, then one done(), then again 3 realtime calls, and done() again.

The first example below shows traffic when both the client and the server know the concept of realtime stream:

It is a bit difficult to see here, but the point is that we see finish(0) and finish(1) for the boostrap and done() calls, but no finish(2^31) for the realtime calls (because no returns were sent). Compare this to the same example, but when the server does not know about the concept of "realtime stream" (i.e. it knows that call as something like write @0 (bytes :Data) -> (); instead of write @0 (bytes :Data) -> realtime stream;).

This time we see that the server sends Return messages for realtime calls (with an id in the 2^31 range), and the realtime-aware client still answers with Finish, to be backwards-compatible.

prevent Warray-bounds, Wstringop-overflow warnings under gcc 12 in Release build:

In member function 'void capnp::compiler::NodeTranslator::StructLayout::HoleSet<UIntType>::addHolesAtEnd(UIntType, UIntType, UIntType) [with UIntType = unsigned char]',
    inlined from 'capnp::uint capnp::compiler::NodeTranslator::StructLayout::Group::DataLocationUsage::allocateFromHole(capnp::compiler::NodeTranslator::StructLayout::Group&, capnp::compiler::NodeTranslator::StructLayout::Union::DataLocation&, capnp::uint)' at c++/src/capnp/compiler/node-translator.c++:352:30,
    inlined from 'virtual capnp::uint capnp::compiler::NodeTranslator::StructLayout::Group::addData(capnp::uint)' at c++/src/capnp/compiler/node-translator.c++:496:63:
c++/src/capnp/compiler/node-translator.c++:118:23: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
  118 |         holes[lgSize] = offset;
      |         ~~~~~~~~~~~~~~^~~~~~~~
c++/src/capnp/compiler/node-translator.c++: In member function 'virtual capnp::uint capnp::compiler::NodeTranslator::StructLayout::Group::addData(capnp::uint)':
c++/src/capnp/compiler/node-translator.c++:71:14: note: at offset 6 into destination object 'capnp::compiler::NodeTranslator::StructLayout::HoleSet<unsigned char>::holes' of size 6
   71 |     UIntType holes[6];
      |              ^~~~~

In file included from c++/src/capnp/layout.h:31,
                 from c++/src/capnp/layout.c++:23:
In member function 'T* kj::ArrayPtr<T>::end() [with T = const capnp::word]',
    inlined from 'capnp::word* capnp::_::SegmentBuilder::allocate(capnp::SegmentWordCount)' at c++/src/capnp/arena.h:471:21,
    inlined from 'static capnp::word* capnp::_::WireHelpers::allocate(capnp::_::WirePointer*&, capnp::_::SegmentBuilder*&, capnp::_::CapTableBuilder*, capnp::SegmentWordCount, capnp::_::WirePointer::Kind, capnp::_::BuilderArena*)' at c++/src/capnp/layout.c++:488:36,
    inlined from 'static capnp::_::StructBuilder capnp::_::WireHelpers::initStructPointer(capnp::_::WirePointer*, capnp::_::SegmentBuilder*, capnp::_::CapTableBuilder*, capnp::_::StructSize, capnp::_::BuilderArena*)' at c++/src/capnp/layout.c++:1099:25,
    inlined from 'static capnp::_::OrphanBuilder capnp::_::OrphanBuilder::initStruct(capnp::_::BuilderArena*, capnp::_::CapTableBuilder*, capnp::_::StructSize)' at c++/src/capnp/layout.c++:3315:57:
c++/src/kj/common.h:1735:28: warning: array subscript 0 is outside array bounds of 'kj::ArrayPtr<const capnp::word> [0]' [-Warray-bounds]
 1735 |   inline T* end() { return ptr + size_; }
      |                            ^~~
c++/src/kj/common.h:1735:34: warning: array subscript 0 is outside array bounds of 'kj::ArrayPtr<const capnp::word> [0]' [-Warray-bounds]
 1735 |   inline T* end() { return ptr + size_; }
      |                                  ^~~~~