Binary Serialization

Related tags

Serialization binn
Overview

Binn

Build Status Tests Stable

Binn is a binary data serialization format designed to be compact, fast and easy to use.

Performance

The elements are stored with their sizes to increase the read performance.

The library uses zero-copy when reading strings, blobs and containers.

The strings are null terminated so when read the library returns a pointer to them inside the buffer, avoiding memory allocation and data copying.

Data Types

The Binn format supports all these:

Primitive data types:

  • null
  • boolean (true and false)
  • integer (up to 64 bits signed or unsigned)
  • floating point numbers (IEEE single and double precision)
  • string
  • blob (binary data)
  • user defined

Containers:

  • list
  • map (numeric key associative array)
  • object (text key associative array)

Format

The elements are stored in this way:

boolean, null:
[type]

int, float (storage: byte, word, dword or qword):
[type][data]

string, blob:
[type][size][data]

list, object, map:
[type][size][count][data]

Example Structure

A json data such as {"hello":"world"} is serialized in binn as:

  \xE2           // type = object (container)
  \x11           // container total size
  \x01           // key/value pairs count
  \x05hello      // key
  \xA0           // type = string
  \x05world\x00  // value (null terminated)

You can check the complete specification

Usage Example

Writing

binn *obj;

// create a new object
obj = binn_object();

// add values to it
binn_object_set_int32(obj, "id", 123);
binn_object_set_str(obj, "name", "John");
binn_object_set_double(obj, "total", 2.55);

// send over the network or save to a file...
send(sock, binn_ptr(obj), binn_size(obj));

// release the buffer
binn_free(obj);

Reading

int id;
char *name;
double total;

id = binn_object_int32(obj, "id");
name = binn_object_str(obj, "name");
total = binn_object_double(obj, "total");

More examples

You can find more usage examples here and in the examples folder

Other Implementations

Feel free to make a wrapper for your preferred language. Then inform us so we can list it here.

How to use

  1. Including the binn.c file in your project; or
  2. Including the static library in your project; or
  3. Linking to the binn library:

On Linux and MacOSX:

gcc myapp.c -lbinn

On Windows:

Include the binn-3.0.lib in your MSVC project or use MinGW:

gcc myapp.c -lbinn-3.0

Compiling the Library

On Linux and MacOSX:

git clone https://github.com/liteserver/binn
cd binn
make
sudo make install

It will create the file libbinn.so.3.0 on Linux and libbinn.3.dylib on MacOSX

On Windows:

Use the included Visual Studio project in the src/win32 folder or compile it using MinGW:

git clone https://github.com/liteserver/binn
cd binn
make

Both will create the file binn-3.0.dll

Static library

To generate a static library:

make static

It will create the file libbinn.a

On Android:

Check for pre-compiled binaries in the android-binn-native project

Regression Tests

On Linux, MacOSX and Windows (MinGW):

cd binn
make test

On Windows (Visual Studio):

Use the included project in the test/win32 folder

Reliability

The current version (3.0) is stable and production ready

As it is cross-platform, data can be transferred between little-endian and big-endian devices

Licence

Apache 2.0

Contact

Questions, suggestions, support: contact AT litereplica DOT io

Comments
  • SIGSEGV in binn_load() when fed with random data

    SIGSEGV in binn_load() when fed with random data

    ASAN:DEADLYSIGNAL
    =================================================================
    ==19372==ERROR: AddressSanitizer: SEGV on unknown address 0x7ffdef02fd7f (pc 0x5567a79c235b bp 0x7ffd966db680 sp 0x7ffd966db660 T0)
        #0 0x5567a79c235a in AdvanceDataPos /home/mannol/Documents/Programming/B/src/binn.c:456
        #1 0x5567a79c5443 in binn_is_valid /home/mannol/Documents/Programming/B/src/binn.c:1237
        #2 0x5567a79c1f03 in binn_load /home/mannol/Documents/Programming/B/src/binn.c:340
        #3 0x5567a79ade2c in message_from_binn_raw /home/mannol/Documents/Programming/B/src/message.c:800
        #4 0x5567a7936b53 in test_message_fuzz /home/mannol/Documents/Programming/B/src/tests.c:877
        #5 0x5567a79275f5 in run_test /home/mannol/Documents/Programming/B/src/tests.c:50
        #6 0x5567a79bd57d in list_for_each /home/mannol/Documents/Programming/B/src/list.c:62
        #7 0x5567a792851f in run_tests /home/mannol/Documents/Programming/B/src/tests.c:3202
        #8 0x5567a794a1b8 in main /home/mannol/Documents/Programming/B/src/tests.c:3289
        #9 0x7f909e67d3f0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x203f0)
        #10 0x5567a7927389 in _start (/home/mannol/Documents/Programming/B/build/D/tests+0x2c389)
    
    AddressSanitizer can not provide additional info.
    SUMMARY: AddressSanitizer: SEGV /home/mannol/Documents/Programming/B/src/binn.c:456 in AdvanceDataPos
    ==19372==ABORTING
    

    EDIT: Another one in a similar place:

    ASAN:DEADLYSIGNAL
    =================================================================
    ==19481==ERROR: AddressSanitizer: SEGV on unknown address 0x7ffed27d8404 (pc 0x55c48d1c240b bp 0x7ffea42b34a0 sp 0x7ffea42b3320 T0)
        #0 0x55c48d1c240a in binn_is_valid /home/mannol/Documents/Programming/B/src/binn.c:1223
        #1 0x55c48d1bef03 in binn_load /home/mannol/Documents/Programming/B/src/binn.c:340
        #2 0x55c48d1aae51 in message_from_binn_raw /home/mannol/Documents/Programming/B/src/message.c:800
        #3 0x55c48d133b53 in test_message_fuzz /home/mannol/Documents/Programming/B/src/tests.c:877
        #4 0x55c48d1245f5 in run_test /home/mannol/Documents/Programming/B/src/tests.c:50
        #5 0x55c48d1ba57d in list_for_each /home/mannol/Documents/Programming/B/src/list.c:62
        #6 0x55c48d12551f in run_tests /home/mannol/Documents/Programming/B/src/tests.c:3202
        #7 0x55c48d1471b8 in main /home/mannol/Documents/Programming/B/src/tests.c:3289
        #8 0x7fba7fa233f0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x203f0)
        #9 0x55c48d124389 in _start (/home/mannol/Documents/Programming/B/build/D/tests+0x2c389)
    
    AddressSanitizer can not provide additional info.
    SUMMARY: AddressSanitizer: SEGV /home/mannol/Documents/Programming/B/src/binn.c:1223 in binn_is_valid
    ==19481==ABORTING
    

    Steps to reproduce (FUZZ):

    while (1)
    {
        uint8_t data[15000];
        // fill data[] with random bytes
        binn obj[1];
        binn_load((void*)data, obj); // << wait until it crashes
    }
    
    opened by mannol 20
  • Reserved #defines

    Reserved #defines

    Compiling binn on native IBM compiler.

    "binn.c", line 36.9: 1506-236 (W) Macro name _BIG_ENDIAN has been redefined.
    "binn.c", line 37.10: 1506-296 (S) #include file <machine/endian.h> not found.
    

    In the case of the native compiler, the symbols in question are defined as:

    #define __BIG_ENDIAN__ 1
    #define _BIG_ENDIAN 1
    
    

    Also, note that symbols with a leading underscore are considered reserved and should not be defined. Might want to make your endian detection code platform independent.

    "Each name that begins with an underscore is reserved to the implementation for use as a name in the global namespace."

    opened by csstup 10
  • binn_map() and binn_create(BINN_MAP) create different outputs

    binn_map() and binn_create(BINN_MAP) create different outputs

    This code produces hex output: e10f02000000022001000000032001

    uint8_t out[1000];
    binn* obj = binn_map();
    binn_map_set_int16(obj, 2, 1);
    binn_map_set_int16(obj, 3, 1);
    memcpy(out, binn_ptr(obj), binn_size(obj)); //  << the said output
    

    However, this code produces the following hex output: 306000009dc8e10f02000000022001

    uint8_t out[1000];
    binn obj[1];
    binn_create(obj, BINN_MAP, 1000, out);
    binn_map_set_int16(obj, 2, 1);
    binn_map_set_int16(obj, 3, 1);
    // now the out[] contains the said output
    

    The consequence:

    // in the first case
    binn_load((void*)out); // << works
    binn_load((void*)(out + 6)); // << fails
    
    // in the second case
    binn_load((void*)out); // << fails
    binn_load((void*)(out + 6)); // << works
    

    From the outputs I see the data is arranged in different order: 1: e10f02000000022001000000032001 2: 306000009dc8e10f02000000022001

    Is that normal? If so, how could one know when the data starts without calling binn_load() twice?

    opened by mannol 6
  • Compilation warnings

    Compilation warnings

    There variables are "maybe uninitialized" in these contexts. Nice lib btw!

    binn.c: In function ‘copy_int_value’: binn.c:2123:21: warning: ‘vint64’ may be used uninitialized in this function [-Wmaybe-uninitialized] *(int *)pdest = (int) vint64; ^~~~~~~~~~~~ binn.c:2139:30: warning: ‘vuint64’ may be used uninitialized in this function [-Wmaybe-uninitialized] *(unsigned int *)pdest = (unsigned int) vuint64; ^~~~~~~~~~~~~~~~~~~~~~ binn.c: In function ‘AddValue’: binn.c:717:6: warning: ‘vint’ may be used uninitialized in this function [-Wmaybe-uninitialized] if (vint >= INT32_MIN) { ^ binn.c:669:10: note: ‘vint’ was declared here int64 vint;

    opened by mannol 5
  • Blob size implementation always uses 4-byte block

    Blob size implementation always uses 4-byte block

    Hello,

    In specification, blob size can be 1 or 4-byte block. However, as far as I understand your code, blob size always uses 4-byte block. (I know very basic C).

    If that is true, I think your implementation should be updated, to support 1-byte block.

    May I ask if you could check?

    Thank you,

    opened by ghost 4
  • Missing magic number in specification

    Missing magic number in specification

    Hi,

    After reading source code, I found the magic number 0x1F22B11F. However specification doesn't mention it. Would you please update specification?

    Thank you,

    opened by ghost 4
  • How to serialize arrays?

    How to serialize arrays?

    Hello. I've created a small example to understand how binn works, but I'm not sure how to properly serialize arrays using the library. How should I serialize an array of uint8_t or uint16_t?

    I'm currently using binn_object_set_object but this doesn't look correct, in fact I get nothing on the other side but null. I'm also serializing a size_t as a uint64, I don't know if there is a better way to do with in binn.

    The example I created is below. I basically serialize some data, save it to a buffer, and then use that buffer to do the inverse operation. I'm hoping to use binn to transfer data using sockets, but I figured the buffer would be enough for testing.

    #include <stdio.h>
    #include <stdlib.h>
    #include <stdint.h>
    #include <inttypes.h>
    #include <string.h>
    
    #include "binn.h"
    
    // to test
    struct my_struct {
    	uint32_t num;
    	size_t size;
    	uint8_t buf[];
    };
    
    static uint8_t *host_to_buffer()
    {
    	binn *obj;
    	uint8_t *buf;
    	uint16_t a[] = {0x0001, 0x5502, 0x4409, 0x33A1, 0x22FC};
    	size_t size = sizeof(a);
    	
    	obj = binn_object();
    	
    	binn_object_set_uint32(obj, "uint", 12345);
    	binn_object_set_str(obj, "str", "hello world");
    	binn_object_set_uint64(obj, "size", size);
    	binn_object_set_object(obj, "array", a);  // what is the correct call?
    	
    	buf = malloc(binn_size(obj));
    	memcpy(buf, binn_ptr(obj), binn_size(obj));
    	
    	binn_free(obj);
    	
    	return buf;
    }
    
    static void buffer_to_host(uint8_t *buf)
    {
    	binn *obj = binn_open(buf);
    	
    	uint32_t num = binn_object_uint32(obj, "uint");
    	char *str = binn_object_str(obj, "str");
    	size_t size = binn_object_uint64(obj, "size");
    	uint16_t *a = binn_object_object(obj, "array");
    	
    	// print deseralization results
    	printf("num=%"PRIu32", str=%s\n", num, str);
    	if (a == NULL) {
    		fprintf(stderr, "array is null\n");
    	} else {
    		for (int i = 0; i < size/sizeof(uint16_t); i++)
    			printf("%04"PRIx16" ", a[i]);
    		printf("\n");
    	}
    	
    	
    	binn_free(obj);
    	free(buf);
    }
    
    int main()
    {
    	// save some data to a buffer using binn; then deserialize the data from buf
    	uint8_t *buf = host_to_buffer();
    	buffer_to_host(buf);
    	return 0;
    }
    
    opened by andrade 4
  • How to properly load and read a binn object from a data buffer without too many allocations?

    How to properly load and read a binn object from a data buffer without too many allocations?

    Hi, having troubles about figuring out how to use binn without wasting too much memory since i'm into a resource-limited environment (retrocomputing, DOS), and i want to know if opening and reading data also makes additional allocations, since i'm a bit confused how the inner works in binn.

    this is my code for example:

        int mapsize = 0;
        char* mapdata = vfs_load(mapfile, &mapsize);
        if(mapdata == NULL)
            return -1;
    
        binn* maproot = binn_open(mapdata);
        binn_release(maproot);
        free(mapdata);
    

    binn_open just allocates just the binn structure and nothing else and fetches the data directly from the buffer without copying?, and if is for string (i assume i have to copy them with memcpy, before i free the data buffer) and objects too? i use mostly binn for reading map data, not writing.

    question 
    opened by hydexon 3
  • Iterate over binn obj example

    Iterate over binn obj example

    I am having trouble understanding how to use binn_object_foreach. It would be great if someone could provide an example. I would like to be able to iterate through keys in the serialized data.

    Thanks.

    question 
    opened by Mango-kid 2
  • Setting length of buffer

    Setting length of buffer

    Hi

    Firstly thanks for a great module, however I have one problem I need to encode the first 4 bytes with an uint32_t. I have tried setting the first field of object to 0 and then try resetting to correct size after packing however second function call fails. Here is some code of what I am doing.

    binn *obj;
    
    obj = binn_object();
    
    binn_object_set_uint32(obj, "length", 0);
    
    // add more fields to object
    
    int size = binn_size(obj);
    binn_object_set_uint32(obj, "length", size);
    

    Thanks Simon

    opened by simon-p-r 2
  • Qt  Creator can't compile header file (just 1 function)

    Qt Creator can't compile header file (just 1 function)

    Hello developers,

    I've cloned from this commit: 7b62ace826bd6c04ab03cdfa99be134e6abd44d2

    Then I included both binn.c and binn.h files into my Qt project. Unfortunately only one single function is preventing Qt from compiling my project. It's at line 262:

    .../binn.h:262: error: expected ‘,’ or ‘...’ before ‘(’ token
     void   APIENTRY binn_set_alloc_functions(void* (*new_malloc)(size_t), void* (*new_realloc)(void*,size_t), void (*new_free)(void*));
                                                                 ^
    

    I don't need that function, so commenting it out will compile successfully.

    I have a note that I'm new to both C and C++.

    May I ask if you could give me some hint to solve this issue?

    Thank you,

    opened by ghost 2
  • unable to create dll in visual studio 2019

    unable to create dll in visual studio 2019

    Hi,

    i am trying to create dll file to use in c# project. i am getting following error:

    cc -Wall -c src/binn.c process_begin: CreateProcess(NULL, cc -Wall -c src/binn.c, ...) failed. make (e=2): The system cannot find the file specified. make: *** [binn.o] Error 2

    i followed the step given in readme

    git clone https://github.com/liteserver/binn cd binn make

    i have GNU Make 3.81 installed on my system, Windows 10.

    Also i tried to compile the code in src/win32 using visual studio 2019, it created binn-1.0.dll, but i am unable to add this file in c# project. i am getting following error

    A reference to the "binn-1.0.dll" could not be added.Please make sure that the file is accessible and that it is a valid assembly or COM component.

    it will be helpful if anybody can share steps for adding dll, or if i can add this dll for c# project or not ? Also as mention in readme

    Both will create the file binn-3.0.dll

    but when i build the project the output file is binn-1.0.dll, is this old repo am i doing something wrong ? if any further info is required from my end please inform

    opened by theElementZero 0
  • Fix missing header for size_t

    Fix missing header for size_t

    Without this standard C header it doesn't compile with following error:

    src\binn.h(259): error C2081: 'size_t': name in formal parameter list illegal

    Library itself (binn.c) compiles normally because it includes other standard headers before binn.h so binn.h receives size_t definition. But if you include binn.h in some other C file (like in writing.c) and don't include other headers, it won't compile. stddef.h is standard header so I think it should be okay to include that. All compilers should support it.

    opened by funbiscuit 0
  • Array serialization

    Array serialization

    I've seen there was an issue about array serialization and it was "solved" by using lists instead. But it is quite inefficient in both space and speed, especially on embedded devices (while this library seems to be the best for embedded). To give an example, I send data (array of 100000 uint16) over network split to packets of about 512 bytes each. If I use lists, I need about 600 packets, while I can fit into 400 if I use blobs and interpret them on the other side. This is a 50% difference. In my opinion this library will benefit a lot by adding support of lists of fixed type (and fixed size of each item). Do you think it will be possible? It may be even a small variation to list interpretation where values will be stored one after another and not storing values of different size or type than first item. Currently I use objects with two keys: t and d. t stores type of each element while d stores blob of an array. And I'm almost fine with it (it doesn't handle endianness but I don't need that). But native support would be a lot better.

    new feature 
    opened by funbiscuit 1
  • Performance Benchmark

    Performance Benchmark

    Hi, I was just curious if there are any performance benchmark numbers for binn?

    For example, how do common operations compare to flatbuffers (flatcc) or maybe json libraries or any other serialization libraries. Like on the flatbuffers google page there is this: https://google.github.io/flatbuffers/flatbuffers_benchmarks.html

    Currently I'm just playing around with storing structs in a serialized format in a key/value database. Not super worried about performance and really like the simplicity of binn. But it would be kind of cool to see how it stacks up vs larger libraries and such.

    Thanks for the nice library!

    question 
    opened by Mimerr 2
  • Nested objects

    Nested objects

    Ciao there,

    I'm new to this project and I'd like to ask whether or not it is possible to have nested objects. For example:

    { "message":{ "type":"QUERY", "timestamp":1234569900, "source":{ "id":1, "name":"NN", "actor":"AGT" } } }

    Something like: binn_object_append_object("source", obj_source, obj_message) Could you provide a code example?

    Thanks.

    question 
    opened by sfhacker 1
Owner
null
Binary Serialization

Binn Binn is a binary data serialization format designed to be compact, fast and easy to use. Performance The elements are stored with their sizes to

null 383 Dec 23, 2022
Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications

Zmeya Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serializ

Sergey Makeev 99 Dec 24, 2022
C++17 library for all your binary de-/serialization needs

blobify blobify is a header-only C++17 library to handle binary de-/serialization in your project. Given a user-defined C++ struct, blobify can encode

Tony Wasserka 247 Dec 8, 2022
Microsoft 2.5k Dec 31, 2022
Cap'n Proto serialization/RPC system - core tools and C++ library

Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except

Cap'n Proto 9.5k Jan 1, 2023
A C++11 library for serialization

cereal - A C++11 library for serialization cereal is a header-only C++11 serialization library. cereal takes arbitrary data types and reversibly turns

iLab @ USC 3.4k Jan 3, 2023
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Google 19.7k Jan 9, 2023
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

niXman 596 Jan 7, 2023
An implementation of the MessagePack serialization format in C / msgpack.org[C]

CMP CMP is a C implementation of the MessagePack serialization format. It currently implements version 5 of the MessagePack Spec. CMP's goal is to be

Charlie Gunyon 290 Dec 31, 2022
MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]

Introduction MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is: Simple and easy to use Secure agai

Nicholas Fraser 419 Jan 1, 2023
Simple C++ 20 Serialization Library that works out of the box with aggregate types!

BinaryLove3 Simple C++ 20 Serialization Library that works out of the box with aggregate types! Requirements BinaryLove3 is a c++20 only library.

RedSkittleFox 14 Sep 2, 2022
CppSerdes is a serialization/deserialization library designed with embedded systems in mind

A C++ serialization/deserialization library designed with embedded systems in mind

Darren V Levine 79 Nov 5, 2022
Serialization framework for Unreal Engine Property System that just works!

DataConfig Serialization framework for Unreal Engine Property System that just works! Unreal Engine features a powerful Property System which implemen

null 81 Dec 19, 2022
Header-only library for automatic (de)serialization of C++ types to/from JSON.

fuser 1-file header-only library for automatic (de)serialization of C++ types to/from JSON. how it works The library has a predefined set of (de)seria

null 51 Dec 17, 2022
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

niXman 455 Sep 7, 2021
universal serialization engine

A Universal Serialization Engine Based on compile-time Reflection iguana is a modern, universal and easy-to-use serialization engine developed in c++1

qicosmos 711 Jan 7, 2023
Yet another JSON/YAML/BSON serialization library for C++.

ThorsSerializer Support for Json Yaml Bson NEW Benchmark Results Conformance mac linux Performance max linux For details see: JsonBenchmark Yet anothe

Loki Astari 281 Dec 10, 2022
Cista is a simple, high-performance, zero-copy C++ serialization & reflection library.

Simple C++ Serialization & Reflection. Cista++ is a simple, open source (MIT license) C++17 compatible way of (de-)serializing C++ data structures. Si

Felix Gündling 1.1k Jan 2, 2023
A lightweight C++20 serialization framework

zpp::bits A modern C++20 binary serialization library, with just one header file. This library is a successor to zpp::serializer. The library tries to

Eyal Z 299 Dec 31, 2022