Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster. In fact, in benchmarks, Cap'n Proto is INFINITY TIMES faster than Protocol Buffers.
Cap'n Proto serialization/RPC system - core tools and C++ library
Overview
Comments
-
json: implement decode(... input, DynamicStruct::Builder output)
- type and field handlers are not implemented yet.
- top level json object must be struct.
This code covers the basic use case: to convert json string to capnp struct.
-
Let MSVC build without lite mode
This branch allows Visual Studio 2015 Update 3 to build everything except parser-dependent code (schema loading, compiler, compiler plugins) in non-lite mode. It is based on the current master branch. On Linux, all tests pass. On Windows, the status is still a bit WIP: 10 of the newly-enabled tests fail, and it builds with many warnings.
@kentonv, this is a rather large PR -- I'm happy to break it up into two or three smaller ones, if you'd like. I won't have time to work on the warnings/test failures and PtmfHelper port until next month, unfortunately, though I'm interested in doing so. I of course wouldn't be offended if you wanted to get some of it done to get a release out the door, too. :)
@nathanhourt, you might be interested in testing this with your codebase, though keep in mind you'll need a MinGW-built capnp.exe.
Notes:
- CMake now only builds parser-dependent code if CAPNP_BUILD_TOOLS is set to ON, which I made the default. We could name this option differently (CAPNP_NO_BUILD_TOOLS?), or we could default it to OFF, or what-have-you. I have no strong preference either way.
- Pointer-to-member-function address extraction is not yet ported, so stack traces will be unhelpful for the time being.
- Atomic operations are a dilemma. For now, I #defined the GNU builtins that RawSchema uses for its initializers to be simple non-atomic loads and stores, per a conversation I had with Kenton a while back. For the atomic sentWake in async-io-win32.{h,c++}, I'd prefer to use std::atomic to decrease the amount of special-casing between MSVC and MinGW.
- One of the overloads of AsyncInputStream::read() was undefined in async-io-win32.c++, which the linker complained about. I copied and pasted the implementation from async-io.c++.
- I had to work around the usual host of compiler bugs. Most of the workarounds turned out okay, but there are a couple (non-const copy capture in lambdas and hard-coding dependency/scope counts) that I wish I had better options for.
-
Websocket Compression (Phase Two)
This PR implements sending and receiving compressed messages. All
permessage-deflate
parameters are supported.I've written an
ew-test-bin
for testing and it seems like things are working. I compared my compressed results with a few examples in the compression RFC and things matched up so that's a good sign. That said, I want to add to our existing KJ tests, though I suspect it might require modifying the fake server a bit.There's still a couple
KJ_DBG
s sprinkled around for debugging reasons. TheContext
class is based on a similar class used by ourCompressionStream
implementation. -
Fix/complete CMake installation
This PR tries to address #973 and also fix an issue with a header installed to the wrong place. I tried to compare the install trees generated by autotools and CMake to make sure there was no functional differences.
I see that @harrishancock said he will do the CMake part here https://github.com/capnproto/capnproto/pull/978#pullrequestreview-397491907, but seeing the imminent release, I thought it was worth trying to do it before the release is official, in case the merge window was still open. My apology if this feels inappropriate.
-
capnproto 0.7.0 fails to build on gcc10 on i686 and armv7hl
When rebuilding capnproto 0.7.0 with GCC 10 in Fedora rawhide, I've encountered different failures on i686 and armv7hl.
i686
The
kj-heavy-tests-run
test suite failed with the following error:2: [ TEST ] filesystem-disk-test.c++:833: DiskFile holes 2: kj/filesystem-disk-test.c++:887: failed: expected file->stat().spaceUsed == meta.spaceUsed 2: stack: 56692dfc 566dd63b f7de4b8b f7d7e441 f7de6388 f7de6aee f7d8fe37 f7d906a5 f7d7e441 f7d8b618 f7de38da f7996ebd 5663b154 2: kj/filesystem-disk-test.c++:920: failed: expected file->stat().spaceUsed < meta.spaceUsed 2: stack: 566d29bc 566dd83a f7de4b8b f7d7e441 f7de6388 f7de6aee f7d8fe37 f7d906a5 f7d7e441 f7d8b618 f7de38da f7996ebd 5663b154 2: [ FAIL ] filesystem-disk-test.c++:833: DiskFile holes (5662 μs)
Full build log: capnproto-c++-0.7.0-5.fc33.i686-build.log.txt
armv7hl
The
kj-tests-run
test suite failed with the following error:1: [ TEST ] table-test.c++:786: large tree table 1: *** Received signal #11: Segmentation fault 1: stack: 1/5 Test #1: kj-tests-run .....................***Failed 1.29 sec
Full build log: capnproto-c++-0.7.0-5.fc33.armv7hl-build.log.txt
-
Add Coroutines TS support
This adds support for the Coroutines TS to libkj. Specifically, you can now co_await kj::Promise
objects from inside functions which themselves return kj::Promise objects. The integration is header-only, so the only difference in a Cap'n Proto build is that we might conditionally compile the coroutine tests, if the compiler and standard library both support the Coroutines TS.
I tested this with Clang/libc++ 12 and the latest VS2019.
-
For Clang with autotools / CMake it should be sufficient to configure the toolchain like so, then configure and build like normal: export CC=clang export CXX=clang++ export CXXFLAGS="-std=gnu++17 -fcoroutines-ts -stdlib=libc++" export LDFLAGS="-fcoroutines-ts -stdlib=libc++" ./configure && make check
-
For VS2019, add -DCMAKE_CXX_FLAGS="/std:c++17 /await /EHsc" to your regular CMake command line. (The /EHsc shuts up a compiler warning...)
VS2019 and GCC both apparently support C++20-standardized coroutines, but VS2019's version seems ICE-y, and I'm too lazy to figure out how to install GCC 10 on Debian Buster.
-
-
Add observing for urgent data to UnixEventPort::FdObserver
Hello,
this patch adds support for observing availability of urgent/out-of-band data by adding a new promise to FdObserver. The patch itself is simple and without side effects: Unless OBSERVE_URGENT is passed when constructing the FdObserver object, no new code is run. The implementation mirrors what was already there for the POLLIN event.
Right, so... why do we need this? A little backstory: I gave Cap'n Proto a shot as RPC server for my home automation system, which controls various devices through a Raspberry Pi B+'s GPIO ports and a i2c GPIO multiplexer. Cap'n Proto did the job flawlessly! However, in order to receive interrupts from the i2c GPIO multiplexer, I have to test for the POLLPRI flag on the interrupt port's fd. Linux's GPIO abstraction layer sets POLLPRI when an edge change is detected on a GPIO port. Extending KJ's UnixEventPort implementation appeared to be the most simple way to achieve this goal and at the same time integrate with the RPC server's event loop. POLLPRI is also used by TCP and many drivers for specialized hardware (such as Bluetooth receivers.)
I would appreciate if this change could be merged so that I do not have to maintain a separate fork.
I have included a test which verifies that the new event works correctly with Linux's loopback TCP/IP out-of-band message implementation.
Cheers, Oliver
-
clang 8 build falied
clang 8 build failed with error:
[ 15%] Building CXX object modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/encoding.c++:242:19: error: no matching conversion for functional-style cast from 'const char8_t [4]' to 'kj::StringPtr' result.addAll(StringPtr(u8"\ufffd")); ^~~~~~~~~~~~~~~~~~~~ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const char8_t [4]' to 'const kj::StringPtr' for 1st argument class StringPtr { ^ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const char8_t [4]' to 'kj::StringPtr' for 1st argument /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:73:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'decltype(nullptr)' (aka 'nullptr_t') for 1st argument inline StringPtr(decltype(nullptr)): content("", 1) {} ^ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:74:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const char *' for 1st argument inline StringPtr(const char* value): content(value, strlen(value) + 1) {} ^ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:141:20: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'ArrayPtr<const char>' for 1st argument inline constexpr StringPtr(ArrayPtr<const char> content): content(content) {} ^ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:499:19: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const kj::String' for 1st argument inline StringPtr::StringPtr(const String& value): content(value.cStr(), value.size() + 1) {} ^ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:83:10: note: candidate template ignored: substitution failure [with T = char8_t [4]]: no matching function for call to 'instance' inline StringPtr(const T& t): StringPtr(t.c_str()) {} ^ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:72:10: note: candidate constructor not viable: requires 0 arguments, but 1 was provided inline StringPtr(): content("", 1) {} ^ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:75:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided inline StringPtr(const char* value, size_t size): content(value, size + 1) { ^ /home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:78:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided inline StringPtr(const char* begin, const char* end): StringPtr(begin, end - begin) {} ^ 1 error generated. modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/build.make:257: recipe for target 'modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o' failed make[2]: *** [modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o] Error 1
-
Added eventloop integration with CFRunLoop (OSX/iOS)
This is mostly copied from node-capnproto and async-unix. This commit enables integration with Cocoa's event loop (CFRunLoop).
Notes:
- I haven't committed any tests yet (however I have tried the AsyncIo::SimpleNetwork test with it and it works both when using
wait
, and running CFRunLoop normally) - I haven't modified the makefiles (requires
-framework CoreFoundation
to be added to LDFLAGS for libkj-async) - I've had to add a temporary
AsyncCocoaIoContext
struct asAsyncIoContext
currently has a unixEventPort field
Is this something you would merge into kj? It would be useful for anyone that wants to integrate kj/capnproto into an OSX/iOS application, and for an Objective-C wrapper of capnproto.
- I haven't committed any tests yet (however I have tried the AsyncIo::SimpleNetwork test with it and it works both when using
-
Implement $Cxx.omitSchemas annotation
Implements a
$Cxx.omitSchemas
annotation that strips schema information from the binary as discussed in https://github.com/capnproto/capnproto/issues/1498 -
capnp 0.9.0 failed to compile on Majove
👋 just found the latest release 0.9.0 does not compile on Mojave (macOS 10.14), error log is below:
build error
In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/mutex.h:28: /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE' constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(), ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION' const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(), ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE' const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(), ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN' uint column = KJ_CALLER_COLUMN()) ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN' #define KJ_CALLER_COLUMN() __builtin_COLUMN() ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(), ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor const char* fileName; ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor const char* function; ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor uint lineNumber; ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor uint columnNumber; ^ 5 errors generated. In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.c++:22: /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE' constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(), ^ make[2]: *** [src/kj/CMakeFiles/kj.dir/mutex.c++.o] Error 1 make[2]: *** Waiting for unfinished jobs.... /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION' const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(), ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE' const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(), ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN' uint column = KJ_CALLER_COLUMN()) ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN' #define KJ_CALLER_COLUMN() __builtin_COLUMN() ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(), ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor const char* fileName; ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor const char* function; ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor uint lineNumber; ^ /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor uint columnNumber; ^ 5 errors generated. make[2]: *** [src/kj/CMakeFiles/kj.dir/source-location.c++.o] Error 1
The cmake command
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/capnp/0.9.0 -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_VERBOSE_MAKEFILE=ON -Wno-dev -DBUILD_TESTING=OFF -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
relates to https://github.com/Homebrew/homebrew-core/pull/83289
-
Own
::attach() and Promise ::attach() have opposite destruction ordering Discussed in https://github.com/cloudflare/workerd/pull/244#discussion_r1060835931.
This issue isn't urgent, but it's a footgun that would be nice to render safe somehow.
A simple mitigation might be to start with a deprecation warning for the variadic form of
kj::Promise<T>::attach()
. -
Fix bug in sendForPipeline().
On the receiving end, we need to make sure the
RpcCallContext
lives at least untilFinish
is received. Otherwise, when the context is destroyed, it will prematurely remove the answer table entry making pipelined calls fail, defeating the purpose.Unfortunately we'll have to wait for this to roll out fully before we can actually use
sendForPipeline()
. -
Implements HTTP Connect in http-over-capnp.
Test Plan
In internal repo for ASAN:
$ bazel test //src/ew/tests:sockets/sockets-tcp-test.ew-test-bin@force-process-sandboxing $ bazel test @capnp-cpp//src/capnp/compat:http-over-capnp-test --config=asan --test_timeout 5
-
Add capnpc-json for dumping code generator requests in json format
When bootstrapping a new Cap'n Proto implementation in another language, it can be useful to make the first code-generator in the same language as the language you're planning to write the code generator in (of course). But until the schema.capnp file has been bootstrapped at least once for the library being developed, this is difficult.
You may bootstrap a simple compiler in another language like C++, but then you won't be able to reuse any of the code and be forced to rewrite from scratch in the target language once up and running.
It'd be useful if another included output format existed like capnpc-capnp, to dump the input code generator request in another format that's easier to bootstrap in the target language. JSON is decently popular enough that it's well supported basically everywhere.
This PR adds capnpc-json, an included output plugin that simply dumps the entire code generator request in JSON format to stdout.
-
ASAN might be misbehaving
Hi!
I was investigating some unrelated issues with ASAN lately and in order to better understand real-world usages of
swapcontext
had a look at what you guys do.First, it seems to me that this https://github.com/capnproto/capnproto/blob/fd508f325ed65f9d6cdf10a0f4511aa53b5659fc/c%2B%2B/src/kj/async.c%2B%2B#L1451-L1456 might lead to problems: https://godbolt.org/z/Ec5dd7a3f. Asan uses
uc_stack.ss_size
anduc_stack.ss_sp
to know what and how much of memory to unpoison, so this hack tricks it.
Second, Asan doesn't interceptsetcontext
(but it does work with swapcontext), thus it doesn't unpoison memory whatsover: https://godbolt.org/z/da4bfE1ze
Third, this https://github.com/capnproto/capnproto/blob/fd508f325ed65f9d6cdf10a0f4511aa53b5659fc/c%2B%2B/src/kj/async.c%2B%2B#L1684-L1686 indeed doesn't leak fake stacks - it exhausts the mechanism instead. You see, there are only so much of fake stacks to use per thread: https://github.com/google/sanitizers/wiki/AddressSanitizerUseAfterReturn#memory-consumption, thus "leaking" them leads to thisIf the given size class is fully used, the allocator will return 0 and thus regular stack will be used (i.e. stack-use-after-return detection will not work for the given function call)
which might render stack-use-after-return not working at all in later usages.
P.S. I am not at all familiar with the codebase, so could misinterpret something here and there.
P.P.S. After rereading this i realized it might sound a bit harsh, but definitely no offense intended - just trying to make an already great library a bit better, :heart:
-
Cannot use capnp bazel library with KJ_STD_COMPAT
patches-welcomelist.h
conditionally includescompat/std-iterator.h
, but this is not included in the bazel library. This gives an error whenKJ_STD_COMPAT
is defined, since the file is not in the sandbox.
Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications
Zmeya Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serializ
Your binary serialization library
Bitsery Header only C++ binary serialization library. It is designed around the networking requirements for real-time data delivery, especially for ga
A C++11 library for serialization
cereal - A C++11 library for serialization cereal is a header-only C++11 serialization library. cereal takes arbitrary data types and reversibly turns
FlatBuffers: Memory Efficient Serialization Library
FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ
Simple C++ 20 Serialization Library that works out of the box with aggregate types!
BinaryLove3 Simple C++ 20 Serialization Library that works out of the box with aggregate types! Requirements BinaryLove3 is a c++20 only library.
CppSerdes is a serialization/deserialization library designed with embedded systems in mind
A C++ serialization/deserialization library designed with embedded systems in mind
Header-only library for automatic (de)serialization of C++ types to/from JSON.
fuser 1-file header-only library for automatic (de)serialization of C++ types to/from JSON. how it works The library has a predefined set of (de)seria
C++17 library for all your binary de-/serialization needs
blobify blobify is a header-only C++17 library to handle binary de-/serialization in your project. Given a user-defined C++ struct, blobify can encode
Yet another JSON/YAML/BSON serialization library for C++.
ThorsSerializer Support for Json Yaml Bson NEW Benchmark Results Conformance mac linux Performance max linux For details see: JsonBenchmark Yet anothe
Cista is a simple, high-performance, zero-copy C++ serialization & reflection library.
Simple C++ Serialization & Reflection. Cista++ is a simple, open source (MIT license) C++17 compatible way of (de-)serializing C++ data structures. Si
Bond is a cross-platform framework for working with schematized data. It supports cross-language de/serialization and powerful generic mechanisms for efficiently manipulating data. Bond is broadly used at Microsoft in high scale services.
Bond Bond is an open-source, cross-platform framework for working with schematized data. It supports cross-language serialization/deserialization and
Fast Binary Encoding is ultra fast and universal serialization solution for C++, C#, Go, Java, JavaScript, Kotlin, Python, Ruby, Swift
Fast Binary Encoding (FBE) Fast Binary Encoding allows to describe any domain models, business objects, complex data structures, client/server request
Yet Another Serialization
YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1
Binary Serialization
Binn Binn is a binary data serialization format designed to be compact, fast and easy to use. Performance The elements are stored with their sizes to
An implementation of the MessagePack serialization format in C / msgpack.org[C]
CMP CMP is a C implementation of the MessagePack serialization format. It currently implements version 5 of the MessagePack Spec. CMP's goal is to be
MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]
Introduction MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is: Simple and easy to use Secure agai
Yet Another Serialization
YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1
universal serialization engine
A Universal Serialization Engine Based on compile-time Reflection iguana is a modern, universal and easy-to-use serialization engine developed in c++1
A lightweight C++20 serialization framework
zpp::bits A modern C++20 binary serialization library, with just one header file. This library is a successor to zpp::serializer. The library tries to