Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications

Overview

Zmeya Build Status Build status codecov MIT

Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serialization library in the usual sense but rather a set of STL-like containers that entirely agnostic for their memory location and movable. As long as you use Zmeya data structures + other trivially_copyable, everything just works, and there is no deserialization cost. You can memory-map a serialized file and immediately start using your data. There are no pointers fixup, nor any other parsing/decoding needed. You can also serialize your data and send it over the network, there is no deserialization cost on the receiver side, or you can even use Zmeya for interprocess communication.

Features

  • Cross-platform compatible
  • Single header library (~550 lines of code for deserialization and extra 750 lines of code with serialization support enabled)
  • No code generation required: no IDL or metadata, just use your types directly
  • No macros
  • Heavily optimized for performance
  • No dependencies
  • Zmeya pointers are always 32-bits (configurable) regardless of the target platform pointer size

Zmeya library offering the following memory movable types

  • Pointer<T>
  • Array<T>
  • String
  • HashSet<Key>
  • HashMap<Key, Value>

Usage

Include the header file, and you are all set.

Usage example

Here is a simple usage example.

#include "Zmeya.h"

struct Test
{
  uint32_t someVar;
  zm::String name;
  zm::Pointer<Test> ptr;
  zm::Array<zm::String> arr;
  zm::HashMap<zm::String, float> hashMap;
};

int main()
{
   // load binary file to memory (using fread, mmap, etc)
   // no parsing/decoding needed
   const Test* test = (const Test*)loadBytesFromDisk("binaryFile.zm");  
   
   // use your loaded data
   printf("%s\n", test->name.c_str());
   for(const zm::String& str : test->arr)
   {
     printf("%s\n",str.c_str());
   }
   printf("key = %3.2f\n", test->hashMap.find("key", 0.0f));
   return 0;
}

You can always find more usage examples looking into unit test files. They are organized in a way to covers all Zmeya features and shows common usage patterns.

How it works

Zmeya movable containers' key idea is to use self-relative pointers instead of using “absolute” pointers provided by C++ by default. The idea is pretty simple; instead of using the absolute address, we are using offset relative to the pointer's memory address. i.e., target_address = uintptr_t(this) + offset

Which is perfectly representable by one of x86/ARM addressing modes addr = reg+reg Here is an example of generated assembly code https://godbolt.org/z/aTTW9E7o9 https://godbolt.org/z/xEqTYe44j

One of the problems of such offset-based addressing is the representation of the null pointer. The null pointer can't be safely represented like an offset since the absolute address 0 is always outside of the mapped region. So we decided to use offset 0 (pointer to self) as a special magic value that encodes null pointer.

#include <stdint.h>

template<typename T>
struct OffsetPtr {
    int32_t offset;
    T* get() const noexcept {
        return reinterpret_cast<T*>(uintptr_t(this) + offset);
    }
};

template<typename T>
struct Ptr {
    T* ptr;
    T* get() const noexcept {
        return ptr;
    }
};

int test1(const OffsetPtr<int>& ptr) {
    return *ptr.get();
}

int test2(const Ptr<int>& ptr) {
    return *ptr.get();
}
test1(OffsetPtr<int> const&):
        movsx   rax, DWORD PTR [rdi]
        mov     eax, DWORD PTR [rax+rdi]
        ret
        
test2(Ptr<int> const&):
        mov     rax, QWORD PTR [rdi]
        mov     eax, DWORD PTR [rax]
        ret

So there is no extra overhead from using such pointers in comparison with traditional pointers.

All other Zmeya containers are pretty much based on the same principles. i.e. zm::String is a self-relative pointer to const char* zm::Array<T> is a self-relative pointer to data + size zm::HashSet<Key> is made using two arrays (buckets and values) etc...

The only requirement is that we have to have all the data tightly packed in a single memory region or binary blob. Zmeya provides a convenient mechanism to build such a binary blob called zm::BlobBuilder. Blob builder is capable of convert all the standard STL containers to appropriate Zmeya movable containers. Blob builder also provides a mechanism to convert all the inner types (e.g., std::vector<std::string>) to Zmeya compatible type. And by default, Zmeya offers convertors/template specializations for all commonly used cases.

References

Boost::offset_ptr
https://www.boost.org/doc/libs/1_75_0/doc/html/interprocess/offset_ptr.html

Handmade Hero forum thread
https://hero.handmade.network/forums/code-discussion/t/487-serialization_techniques_with_memory_pooling

Relative Pointers article by Ginger Bill
https://www.gingerbill.org/article/2020/05/17/relative-pointers/

"The Blob and I" by Niklas Gray
https://bitsquid.blogspot.com/2010/02/blob-and-i.html

FlatBuffers by Google
https://google.github.io/flatbuffers/

Physics Optimization Strategies by Sergiy Migdalskiy (slides 56-68)
http://media.steampowered.com/apps/valve/2015/Migdalskiy_Sergiy_Physics_Optimization_Strategies.pdf

https://youtu.be/Nsf2_Au6KxU?t=1542

Relative Pointers by Jonathan Blow
https://www.youtube.com/watch?v=Z0tsNFZLxSU

You might also like...
Simple C++ 20 Serialization Library that works out of the box with aggregate types!

BinaryLove3 Simple C++ 20 Serialization Library that works out of the box with aggregate types! Requirements BinaryLove3 is a c++20 only library.

Yet another JSON/YAML/BSON serialization library for C++.
Yet another JSON/YAML/BSON serialization library for C++.

ThorsSerializer Support for Json Yaml Bson NEW Benchmark Results Conformance mac linux Performance max linux For details see: JsonBenchmark Yet anothe

Yet Another Serialization
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

An implementation of the MessagePack serialization format in C / msgpack.org[C]

CMP CMP is a C implementation of the MessagePack serialization format. It currently implements version 5 of the MessagePack Spec. CMP's goal is to be

MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]

Introduction MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is: Simple and easy to use Secure agai

Serialization framework for Unreal Engine Property System that just works!

DataConfig Serialization framework for Unreal Engine Property System that just works! Unreal Engine features a powerful Property System which implemen

Yet Another Serialization
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

universal serialization engine

A Universal Serialization Engine Based on compile-time Reflection iguana is a modern, universal and easy-to-use serialization engine developed in c++1

Owner
Sergey Makeev
Roblox Corporation, Sr. Technical director, Rendering
Sergey Makeev
C++17 library for all your binary de-/serialization needs

blobify blobify is a header-only C++17 library to handle binary de-/serialization in your project. Given a user-defined C++ struct, blobify can encode

Tony Wasserka 247 Oct 20, 2022
Header-only library for automatic (de)serialization of C++ types to/from JSON.

fuser 1-file header-only library for automatic (de)serialization of C++ types to/from JSON. how it works The library has a predefined set of (de)seria

null 50 Oct 20, 2022
Fast Binary Encoding is ultra fast and universal serialization solution for C++, C#, Go, Java, JavaScript, Kotlin, Python, Ruby, Swift

Fast Binary Encoding (FBE) Fast Binary Encoding allows to describe any domain models, business objects, complex data structures, client/server request

Ivan Shynkarenka 646 Dec 1, 2022
Binary Serialization

Binn Binn is a binary data serialization format designed to be compact, fast and easy to use. Performance The elements are stored with their sizes to

null 381 Nov 24, 2022
CppSerdes is a serialization/deserialization library designed with embedded systems in mind

A C++ serialization/deserialization library designed with embedded systems in mind

Darren V Levine 79 Nov 5, 2022
Cista is a simple, high-performance, zero-copy C++ serialization & reflection library.

Simple C++ Serialization & Reflection. Cista++ is a simple, open source (MIT license) C++17 compatible way of (de-)serializing C++ data structures. Si

Felix Gündling 1.1k Dec 1, 2022
Simple Binary Encoding (SBE) - High Performance Message Codec

Simple Binary Encoding (SBE) SBE is an OSI layer 6 presentation for encoding and decoding binary application messages for low-latency financial applic

Real Logic 2.7k Nov 30, 2022
Cap'n Proto serialization/RPC system - core tools and C++ library

Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except

Cap'n Proto 9.4k Dec 1, 2022
A C++11 library for serialization

cereal - A C++11 library for serialization cereal is a header-only C++11 serialization library. cereal takes arbitrary data types and reversibly turns

iLab @ USC 3.4k Nov 22, 2022
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Google 19.4k Nov 27, 2022