MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]

Overview

Introduction

MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is:

The core of MPack contains a buffered reader and writer, and a tree-style parser that decodes into a tree of dynamically typed nodes. Helper functions can be enabled to read values of expected type, to work with files, to allocate strings automatically, to check UTF-8 encoding, and more.

The MPack code is small enough to be embedded directly into your codebase. Simply download the amalgamation package and add mpack.h and mpack.c to your project.

The MPack featureset can be customized at compile-time to set which features, components and debug checks are compiled, and what dependencies are available.

Build Status

Travis-CI AppVeyor Coveralls.io
Build Status Build Status Build Status

The Node API

The Node API parses a chunk of MessagePack data into an immutable tree of dynamically-typed nodes. A series of helper functions can be used to extract data of specific types from each node.

// parse a file into a node tree
mpack_tree_t tree;
mpack_tree_init_filename(&tree, "homepage-example.mp", 0);
mpack_tree_parse(&tree);
mpack_node_t root = mpack_tree_root(&tree);

// extract the example data on the msgpack homepage
bool compact = mpack_node_bool(mpack_node_map_cstr(root, "compact"));
int schema = mpack_node_i32(mpack_node_map_cstr(root, "schema"));

// clean up and check for errors
if (mpack_tree_destroy(&tree) != mpack_ok) {
    fprintf(stderr, "An error occurred decoding the data!\n");
    return;
}

Note that no additional error handling is needed in the above code. If the file is missing or corrupt, if map keys are missing or if nodes are not in the expected types, special "nil" nodes and false/zero values are returned and the tree is placed in an error state. An error check is only needed before using the data.

The above example decodes into allocated pages of nodes. A fixed node pool can be provided to the parser instead in memory-constrained environments. For maximum performance and minimal memory usage, the Expect API can be used to parse data of a predefined schema.

The Write API

The Write API encodes structured data to MessagePack.

// encode to memory buffer
char* data;
size_t size;
mpack_writer_t writer;
mpack_writer_init_growable(&writer, &data, &size);

// write the example on the msgpack homepage
mpack_start_map(&writer, 2);
mpack_write_cstr(&writer, "compact");
mpack_write_bool(&writer, true);
mpack_write_cstr(&writer, "schema");
mpack_write_uint(&writer, 0);
mpack_finish_map(&writer);

// finish writing
if (mpack_writer_destroy(&writer) != mpack_ok) {
    fprintf(stderr, "An error occurred encoding the data!\n");
    return;
}

// use the data
do_something_with_data(data, size);
free(data);

In the above example, we encode to a growable memory buffer. The writer can instead write to a pre-allocated or stack-allocated buffer, avoiding the need for memory allocation. The writer can also be provided with a flush function (such as a file or socket write function) to call when the buffer is full or when writing is done.

If any error occurs, the writer is placed in an error state. The writer will flag an error if too much data is written, if the wrong number of elements are written, if the data could not be flushed, etc. No additional error handling is needed in the above code; any subsequent writes are ignored when the writer is in an error state, so you don't need to check every write for errors.

Note in particular that in debug mode, the mpack_finish_map() call above ensures that two key/value pairs were actually written as claimed, something that other MessagePack C/C++ libraries may not do.

Comparison With Other Parsers

MPack is rich in features while maintaining very high performance and a small code footprint. Here's a short feature table comparing it to other C parsers:

MPack
(v1.1)
msgpack-c
(v3.2.0)
CMP
(v18)
CWPack
(v1.1)
No libc requirement
Growable memory writer ✓*
File I/O helpers ✓*
Propagating errors
Incremental parser
Tree stream parser
Compound size tracking
Automatic compound size

A larger feature comparison table is available here which includes descriptions of the various entries in the table.

This benchmarking suite compares the performance of MPack to other implementations of schemaless serialization formats. MPack outperforms all JSON and MessagePack libraries (except CWPack), and in some tests MPack is several times faster than RapidJSON for equivalent data.

Why Not Just Use JSON?

Conceptually, MessagePack stores data similarly to JSON: they are both composed of simple values such as numbers and strings, stored hierarchically in maps and arrays. So why not just use JSON instead? The main reason is that JSON is designed to be human-readable, so it is not as efficient as a binary serialization format:

  • Compound types such as strings, maps and arrays are delimited, so appropriate storage cannot be allocated upfront. The whole object must be parsed to determine its size.

  • Strings are not stored in their native encoding. Special characters such as quotes and backslashes must be escaped when written and converted back when read.

  • Numbers are particularly inefficient (especially when parsing back floats), making JSON inappropriate as a base format for structured data that contains lots of numbers.

  • Binary data is not supported by JSON at all. Small binary blobs such as icons and thumbnails need to be Base64 encoded or passed out-of-band.

The above issues greatly increase the complexity of the decoder. Full-featured JSON decoders are quite large, and minimal decoders tend to leave out such features as string unescaping and float parsing, instead leaving these up to the user or platform. This can lead to hard-to-find platform-specific and locale-specific bugs, as well as a greater potential for security vulnerabilites. This also significantly decreases performance, making JSON unattractive for use in applications such as mobile games.

While the space inefficiencies of JSON can be partially mitigated through minification and compression, the performance inefficiencies cannot. More importantly, if you are minifying and compressing the data, then why use a human-readable format in the first place?

Testing MPack

The MPack build process does not build MPack into a library; it is used to build and run the unit tests. You do not need to build MPack or the unit testing suite to use MPack.

See test/README.md for information on how to test MPack.

Comments
  • Initial work on generic mpack_write fixes #34

    Initial work on generic mpack_write fixes #34

    As discussed in #34 for now there is only the generic writer with C++ overloading. I added a c11=1 target to scons (for now). Before merging I can continue working on this pull request and we can discuss in #34. Let me know what you think.

    opened by xor-gate 13
  • Usage in Linux kernel modules

    Usage in Linux kernel modules

    Hi, we are using this nice library within Linux kernel modules. Since we only need the writer, a colleague took only the necessary code and changed/removed stuff to make it compile.

    This works but is of course not that nice. I haven't found any clean way of configuring the project accordingly.

    Did I miss something?

    best regards, Matthias

    opened by mloy 8
  • Possible encoding problems with differing double/int64 endianness

    Possible encoding problems with differing double/int64 endianness

    Hi Nicholas,

    I'm working on an ARM platform(little-endian) with your library. I was using doubles and floats, but this probably also counts for other types in mpack. The items in the the packet are endian swapped and not sent big-endian as described in the spec. https://github.com/msgpack/msgpack/blob/master/spec.md#formats-float

    When I send them to a server (i.e. x86), the values are also wrong. Using a union for mapping to uint64 and double is valid on big-endian systems, but invalid on little-endian. https://github.com/ludocode/mpack/blob/8c5bd668b515ded9e8e8e5a2f48033d2c3f6bd75/src/mpack/mpack-writer.c#L377-L384

    Perhaps it is a good idea to check for little-endian and swap the bytes when necessary?

    opened by rikvdh 8
  • MPACK_HAS_GENERIC autodetection is bugged with clang and/or C++ compiler

    MPACK_HAS_GENERIC autodetection is bugged with clang and/or C++ compiler

    With clang 3.7 the generic autodetection is bugged. Because of the following issue: See http://clang-developers.42468.n3.nabble.com/Generic-compiles-fine-in-c-source-td4037233.html

    opened by xor-gate 7
  • Undefined configuration and compiler definitions cause warnings

    Undefined configuration and compiler definitions cause warnings

    Hi Nicholas,

    I was compiling Mpack for a non-Linux platform today (non-GNU) and ran into a big list of warnings (46 to be exact). Since we want to compile our code with -Werror we really want to fix it. But we need your opinion on this.

    See some of our warnings below this. We are using the amalgamation package, so line-numbers are crazy, but you will get the point. The compiler moans about the check within #if while the variable is not defined.

    In the example of the first warning:

    #if MPACK_STDLIB
    

    should become:

    #if defined(MPACK_STDLIB) && MPACK_STDLIB
    

    but this creates a mess in the code. From my point of view, you want to use:

    #ifdef MPACK_STDLIB
    

    or

    #if defined(MPACK_STDLIB)
    

    but this changes the behavior when configuring Mpack, only setting config to 0 wouldn't suffice. You need to disable the config option completely.

    We are using GCC 4.9.3 Linaro arm-none-eabi cross-tools.

    I can fix this and make a PR for you. But (since it is your lib) I need your input on this.

    Thanks,

    Rik

    /3rdparty/mpack/src/mpack/mpack.h:78:5: warning: "MPACK_STDLIB" is not defined [-Wundef]
     #if MPACK_STDLIB
         ^
    /3rdparty/mpack/src/mpack/mpack.h:82:5: warning: "MPACK_STDIO" is not defined [-Wundef]
     #if MPACK_STDIO
         ^
    /3rdparty/mpack/src/mpack/mpack.h:85:5: warning: "MPACK_SETJMP" is not defined [-Wundef]
     #if MPACK_SETJMP
         ^
    /3rdparty/mpack/src/mpack/mpack.h:113:7: warning: "_MSC_VER" is not defined [-Wundef]
     #elif _MSC_VER
           ^
    /3rdparty/mpack/src/mpack/mpack.h:192:5: warning: "MPACK_STDLIB" is not defined [-Wundef]
     #if MPACK_STDLIB
         ^
    /3rdparty/mpack/src/mpack/mpack.h:224:5: warning: "MPACK_READ_TRACKING" is not defined [-Wundef]
     #if MPACK_READ_TRACKING && (!defined(MPACK_READER) || !MPACK_READER)
         ^
    /3rdparty/mpack/src/mpack/mpack.h:227:5: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
     #if MPACK_WRITE_TRACKING && (!defined(MPACK_WRITER) || !MPACK_WRITER)
         ^
    /3rdparty/mpack/src/mpack/mpack.h:231:9: warning: "MPACK_STDIO" is not defined [-Wundef]
         #if MPACK_STDIO
             ^
    /3rdparty/mpack/src/mpack/mpack.h:234:9: warning: "MPACK_READ_TRACKING" is not defined [-Wundef]
         #if MPACK_READ_TRACKING
             ^
    /3rdparty/mpack/src/mpack/mpack.h:237:9: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
         #if MPACK_WRITE_TRACKING
             ^
    /3rdparty/mpack/src/mpack/mpack.h:559:5: warning: "MPACK_READ_TRACKING" is not defined [-Wundef]
     #if MPACK_READ_TRACKING || MPACK_WRITE_TRACKING
         ^
    /3rdparty/mpack/src/mpack/mpack.h:559:28: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
     #if MPACK_READ_TRACKING || MPACK_WRITE_TRACKING
                                ^
    /3rdparty/mpack/src/mpack/mpack.h:780:5: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
     #if MPACK_WRITE_TRACKING
         ^
    /3rdparty/mpack/src/mpack/mpack.h:827:9: warning: "MPACK_SETJMP" is not defined [-Wundef]
         #if MPACK_SETJMP
             ^
    /3rdparty/mpack/src/mpack/mpack.h:833:9: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
         #if MPACK_WRITE_TRACKING
             ^
    /3rdparty/mpack/src/mpack/mpack.h:882:5: warning: "MPACK_STDIO" is not defined [-Wundef]
     #if MPACK_STDIO
         ^
    /3rdparty/mpack/src/mpack/mpack.h:906:5: warning: "MPACK_SETJMP" is not defined [-Wundef]
     #if MPACK_SETJMP
         ^
    /3rdparty/mpack/src/mpack/mpack.h:1203:5: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
     #if MPACK_WRITE_TRACKING
         ^
    /3rdparty/mpack/src/mpack/mpack.h:1298:5: warning: "MPACK_READ_TRACKING" is not defined [-Wundef]
     #if MPACK_READ_TRACKING
         ^
    /3rdparty/mpack/src/mpack/mpack.h:1351:9: warning: "MPACK_SETJMP" is not defined [-Wundef]
         #if MPACK_SETJMP
    
    opened by rikvdh 7
  • mpack_writer should not set flag to mpack_error_io when buffer to small

    mpack_writer should not set flag to mpack_error_io when buffer to small

    I think current behaviour is a bit weird when a static buffer is not big enough and more is tried to be written and the context is destroyed. The error flag is expected to be mpack_error_too_big but it is stated in the documentation (and emitted) currently as mpack_error_io.

    opened by xor-gate 6
  • Build for kernel module

    Build for kernel module

    This relates to: https://github.com/ludocode/mpack/issues/80

    The request contains everything I need to change to have it compiled withing a linux kernel module. Commit c92c2b3 looks not that nice to me. I think this one needs some refurbishment.

    opened by mloy 5
  • Support building with CMake or similar

    Support building with CMake or similar

    Hi,

    It would be really handy to be able to build this with CMake or some other standard build system, as Visual Studio .sln files aren't readily usable on Linux. I can't see any way to compile this on Linux currently, which is a shame, as I'd otherwise love to use this library.

    opened by martingkelly 5
  • Created a JUCE module for mpack.

    Created a JUCE module for mpack.

    This makes integrating mpack in C++ JUCE projects really swift and easy.

    Implementation Notes

    • JUCE Unity Build Module Format
    • The module code is explicitly designed to be CRLF to support MSVC. This is a consistent design choice for JUCE projects everywhere.

    Testing Question

    I'm open to ideas for how you would like to have these files be automatically tested as part of your CI.

    • You would definitely need a clone of JUCE from its master or develop branch.
    • You would need to build its Projucer app.
    • You will need to have a Projucer-based project ready with your module.
    • You can run the Projucer from the command-line:
      • Just resave the project.
        • pathToProjucer --resave path/to/MyProject.jucer

    I'm not sure what you would want to see run as part of this project. Although, JUCE provides a UnitTest class which could be integrated into the module, and selectively enabled via a module configuration macro.

    MSVC Warning Notes

    There are two types of warnings that come up. Seeing that I don't have the entire context for why the code is the way it is, I'm not totally sure how you would prefer tackling them so I'll leave that up to you!

    • C4127 is a common occurrence.
      • Example: if (sizeof(unsigned int) == 4)
    • There are a handful of truncation warnings (C4310).
      • mpack_write_byte_element(writer, (char)0xc0);
        • 0xc0 is 192 in decimal, causing overflow and therefore UB.

    Module Usage Example in JUCE's Projucer

    image

    opened by jrlanglois 5
  • Improved c89 compatibility

    Improved c89 compatibility

    Heyo!

    Currently your code fails to compile when c_std=gnu89. However, the only C99 feature that you appear to use is the ability to declare variables inside for loops. This PR moves the declarations outside the for loops, and compiles when c_std=gnu89

    Cheers!

    opened by dirkson 5
  • Overly used of inline keyword

    Overly used of inline keyword

    Hi,

    I'm integrating mpack into microcontroller/firmware and the use of inline is a no-go because it explodes the footprint of the firmware by crazy. The "magic" of inline is also not as you expect and the linux kernel coding style has some word about it:

    See Chapter 15: The inline disease https://www.kernel.org/doc/Documentation/CodingStyle

    opened by xor-gate 5
  • ESP32-C3 rebooting when executing functions from the MPack library

    ESP32-C3 rebooting when executing functions from the MPack library

    Hello, I'm testing the MPack library in an ESP32-C3 microcontroller. Below the code I'm using for testing (I added the MPack functions in the app_main function):

    #include <stdio.h>
    #include <stdint.h>
    #include <stddef.h>
    #include <string.h>
    
    #include "esp_wifi.h"
    #include "esp_system.h"
    #include "nvs_flash.h"
    #include "esp_event.h"
    #include "esp_netif.h"
    #include "protocol_examples_common.h"
    
    #include "freertos/FreeRTOS.h"
    #include "freertos/task.h"
    #include "freertos/semphr.h"
    #include "freertos/queue.h"
    
    #include "lwip/sockets.h"
    #include "lwip/dns.h"
    #include "lwip/netdb.h"
    
    #include "esp_log.h"
    #include "mqtt_client.h"
    
    #include "mpack.h"
    
    void print_data(char [], size_t);
    
    static const char *TAG = "MQTT_EXAMPLE";
    
    static void log_error_if_nonzero(const char *message, int error_code)
    {
        if (error_code != 0) {
            ESP_LOGE(TAG, "Last error %s: 0x%x", message, error_code);
        }
    }
    
    /*
     * @brief Event handler registered to receive MQTT events
     *
     *  This function is called by the MQTT client event loop.
     *
     * @param handler_args user data registered to the event.
     * @param base Event base for the handler(always MQTT Base in this example).
     * @param event_id The id for the received event.
     * @param event_data The data for the event, esp_mqtt_event_handle_t.
     */
    static void mqtt_event_handler(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data)
    {
        ESP_LOGD(TAG, "Event dispatched from event loop base=%s, event_id=%d", base, event_id);
        esp_mqtt_event_handle_t event = event_data;
        esp_mqtt_client_handle_t client = event->client;
        int msg_id;
        switch ((esp_mqtt_event_id_t)event_id) {
        case MQTT_EVENT_CONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");
            msg_id = esp_mqtt_client_publish(client, "/topic/qos1", "data_3", 0, 1, 0);
            ESP_LOGI(TAG, "sent publish successful, msg_id=%d", msg_id);
    
            msg_id = esp_mqtt_client_subscribe(client, "/topic/qos0", 0);
            ESP_LOGI(TAG, "sent subscribe successful, msg_id=%d", msg_id);
    
            msg_id = esp_mqtt_client_subscribe(client, "/topic/qos1", 1);
            ESP_LOGI(TAG, "sent subscribe successful, msg_id=%d", msg_id);
    
            msg_id = esp_mqtt_client_unsubscribe(client, "/topic/qos1");
            ESP_LOGI(TAG, "sent unsubscribe successful, msg_id=%d", msg_id);
            break;
        case MQTT_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_DISCONNECTED");
            break;
    
        case MQTT_EVENT_SUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_SUBSCRIBED, msg_id=%d", event->msg_id);
            msg_id = esp_mqtt_client_publish(client, "/topic/qos0", "data", 0, 0, 0);
            ESP_LOGI(TAG, "sent publish successful, msg_id=%d", msg_id);
            break;
        case MQTT_EVENT_UNSUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_UNSUBSCRIBED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_PUBLISHED:
            ESP_LOGI(TAG, "MQTT_EVENT_PUBLISHED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_DATA:
            ESP_LOGI(TAG, "MQTT_EVENT_DATA");
            printf("TOPIC=%.*s\r\n", event->topic_len, event->topic);
            printf("DATA=%.*s\r\n", event->data_len, event->data);
            break;
        case MQTT_EVENT_ERROR:
            ESP_LOGI(TAG, "MQTT_EVENT_ERROR");
            if (event->error_handle->error_type == MQTT_ERROR_TYPE_TCP_TRANSPORT) {
                log_error_if_nonzero("reported from esp-tls", event->error_handle->esp_tls_last_esp_err);
                log_error_if_nonzero("reported from tls stack", event->error_handle->esp_tls_stack_err);
                log_error_if_nonzero("captured as transport's socket errno",  event->error_handle->esp_transport_sock_errno);
                ESP_LOGI(TAG, "Last errno string (%s)", strerror(event->error_handle->esp_transport_sock_errno));
    
            }
            break;
        default:
            ESP_LOGI(TAG, "Other event id:%d", event->event_id);
            break;
        }
    }
    
    static void mqtt_app_start(void)
    {
        esp_mqtt_client_config_t mqtt_cfg = {
            .uri = CONFIG_BROKER_URL,
        };
    #if CONFIG_BROKER_URL_FROM_STDIN
        char line[128];
    
        if (strcmp(mqtt_cfg.uri, "FROM_STDIN") == 0) {
            int count = 0;
            printf("Please enter url of mqtt broker\n");
            while (count < 128) {
                int c = fgetc(stdin);
                if (c == '\n') {
                    line[count] = '\0';
                    break;
                } else if (c > 0 && c < 127) {
                    line[count] = c;
                    ++count;
                }
                vTaskDelay(10 / portTICK_PERIOD_MS);
            }
            mqtt_cfg.uri = line;
            printf("Broker url: %s\n", line);
        } else {
            ESP_LOGE(TAG, "Configuration mismatch: wrong broker url");
            abort();
        }
    #endif /* CONFIG_BROKER_URL_FROM_STDIN */
    
        esp_mqtt_client_handle_t client = esp_mqtt_client_init(&mqtt_cfg);
        /* The last argument may be used to pass data to the event handler, in this example mqtt_event_handler */
        esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID, mqtt_event_handler, NULL);
        esp_mqtt_client_start(client);
    }
    
    void app_main(void)
    {
        ESP_ERROR_CHECK(nvs_flash_init());
        ESP_ERROR_CHECK(esp_netif_init());
        ESP_ERROR_CHECK(esp_event_loop_create_default());
    
        /* This helper function configures Wi-Fi or Ethernet, as selected in menuconfig.
         * Read "Establishing Wi-Fi or Ethernet Connection" section in
         * examples/protocols/README.md for more information about this function.
         */
        //ESP_ERROR_CHECK(example_connect());
    
        mqtt_app_start();
    
    	// encode to memory buffer
    	char *data;
    	char data_r[256] = {0};
    	size_t size;
    	int len = 0;
    	mpack_writer_t writer;
    	mpack_writer_init_growable(&writer, &data, &size);
    
    	// write the example on the msgpack homepage
    	mpack_build_map(&writer);
    	mpack_write_cstr(&writer, "compact");
    	mpack_write_bool(&writer, true);
    	mpack_write_cstr(&writer, "schemaaaaa");
    	mpack_write_uint(&writer, 7);
    	mpack_complete_map(&writer);
    
            len = writer.position;
    
    	printf("Print 1: %u %u\n", len, (*writer.end));
    	printf("Print 2: %u %u\n", data[1], data[2]);
    
    	// finish writing
    	if (mpack_writer_destroy(&writer) != mpack_ok) {
    		  fprintf(stderr, "An error occurred encoding the data!\n");
    		  return;
    	}
    	else
    	{
    
    	}
    
    
    
    	// use the data
    
    	free(data);
    }
    
    
    void print_data(char data_in[], size_t length)
    {
    	for (size_t i = 0; i < length; i++)
    	{
    		printf("%X ", data_in[i]);
    	}
    }
    

    When I try to print the data in line printf("Print 2: %u %u\n", data[1], data[2]); I the microcontroller reboots:

    Print 1: 0 249
    Guru Meditation Error: Core  0 panic'ed (Load access fault). Exception was unhandled.
    
    Stack dump detected
    Core  0 register dump:
    MEPC    : 0x42007944  RA      : 0x42007942  SP      : 0x3fc91b90  GP      : 0x3fc8be00  
    0x42007944: app_main at /home/msurca/esp/IoTliftv1_0_3/tcp/build/../main/app_main.c:196 (discriminator 2)
    
    0x42007942: app_main at /home/msurca/esp/IoTliftv1_0_3/tcp/build/../main/app_main.c:196 (discriminator 2)
    
    TP      : 0x3fc7513c  T0      : 0x4005890e  T1      : 0x20000000  T2      : 0x3fc68a28  
    S0/FP   : 0x00000000  S1      : 0x00000000  A0      : 0x0000000f  A1      : 0x3fc917c8  
    A2      : 0x00000000  A3      : 0x00000001  A4      : 0x3fc8e000  A5      : 0x00000000  
    A6      : 0x42005cf4  A7      : 0x0000000a  S2      : 0x00000000  S3      : 0x00000000  
    0x42005cf4: console_write at /home/msurca/esp/esp-idf/components/vfs/vfs_console.c:71
    
    S4      : 0x00000000  S5      : 0x00000000  S6      : 0x00000000  S7      : 0x00000000  
    S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000  
    T3      : 0x00000000  T4      : 0x3ff00000  T5      : 0x00000000  T6      : 0x3fd34413  
    MSTATUS : 0x00001881  MTVEC   : 0x40380001  MCAUSE  : 0x00000005  MTVAL   : 0x00000002  
    0x40380001: _vector_table at ??:?
    
    MHARTID : 0x00000000  
    
    
    Backtrace:
    
    
    0x42007944 in app_main () at ../main/app_main.c:196
    196		printf("Print 2: %u %u\n", data[1], data[2]);
    #0  0x42007944 in app_main () at ../main/app_main.c:196
    #1  0x4205e806 in main_task (args=<optimized out>) at /home/msurca/esp/esp-idf/components/freertos/FreeRTOS-Kernel/portable/port_common.c:124
    #2  0x40387226 in prvTaskExitError () at /home/msurca/esp/esp-idf/components/freertos/FreeRTOS-Kernel/portable/riscv/port.c:132
    Backtrace stopped: frame did not save the PC
    ELF file SHA256: a7c731be59522570
    
    Rebooting...
    

    Also, if I change the content of "schema" by "schema12345" the end of the string is also 249. So, I am not sure if this is the function I should use to get the length of the data in the buffer. If I use the mpack_writer_buffer_used function the length I always get is 0.

    Finally, if I comment the line mpack_write_cstr(&writer, "compact");the program crashes and the microcontroller restarts. If the line mpack_build_map(&writer); is followed by the function mpack_write_cstr the program can be executed without crashing.

    Which could be the reason of these problems?

    opened by marco711 0
  • Warning about incompatible int types in mpack_snprintf calls

    Warning about incompatible int types in mpack_snprintf calls

    I copied the "amalgamation package" sources to my project and compiled it with a STM32 CubeIDE compiler. Got some warnings about incompatible types:

    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 427 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 477 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 489 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 492 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 544 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 547 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 556 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 559 C/C++ Problem
    

    Quickfix seems to be to replace all %u with %lu in the affected mpack_snprintf calls from above, where the compiler is complaining. Don't know if this could be fixed in the source repo here and still be platform independent?

    opened by Franky1 0
  • comparison with libmpack

    comparison with libmpack

    There's another project (since 2016) with a similar name: https://github.com/libmpack/libmpack

    Would be useful to see it in your comparison table:

    https://github.com/ludocode/mpack#comparison-with-other-parsers

    I also wonder if the names are too similar? It looks like https://github.com/ludocode/mpack generates its library file as libmpack, which may conflict with https://github.com/libmpack/libmpack .

    documentation only 
    opened by justinmk 1
  • Stream reader with manual memory management?

    Stream reader with manual memory management?

    Hi, I really like how well this library is developed and might want to use it in a project. However, I'm not sure if the node reader API can be used the way I'd like. To my understanding, the API is able to:

    • let the user handle memory management: With mpack_tree_init_pool(), the library does not allocate any memory by itself. Both the input data and the nodes memory is allocated by the user.
    • parse a continuous stream: With mpack_tree_init_stream() I can use the library for processing a TCP stream of MessagePack data.

    However, it seems that these two features cannot be combined. The stream reader always allocates its own buffer for the input data, and for the nodes. So I cannot manage the memory by myself. Especially for environments where memory fragmentation needs to be minimized, that's a real drawback.

    Actually for me it would be good enough if I could simply allocate a large enough buffer for the data and for the nodes once, which shall then be used as-is without any reallocation (memory is available, but should not fragment too much during runtime). Probably this is (theoretically) possible even with mpack_tree_init_stream(), but there's still a problem: The documentation says that the reader cannot be reset if it is in error state. So in that case I have to destroy and create a new reader, which will also reallocate (and possibly fragment) the memory.

    Is my understanding about that correct? Or is there a way to recover from a stream reader error without any memory reallocation?

    By the way, somehow it is also a bit cumbersome that reading the data has to be implemented by a callback. I understand that for directly reading from a blocking TCP socket it works perfectly. But if the data is received by some other software component and then forwarded in blocks to some kind of "data processor", it is very cumbersome. It would be much simpler if I could allocate my own data buffer, and let the parser process it block by block.

    Example (simplified):

    char buffer[1024];
    size_t bufferSize = 0;
    mpack_tree_t tree;
    
    void init() {
        // let the stream parser know the start address of the data
        mpack_tree_init_stream(&tree, &buffer[0], ...);
    }
    
    void processTcpData(const char* data, size_t len) {
        // append new data to existing data
        memcpy(&buffer[bufferSize], data, len);
        bufferSize += len;
    
        // continue parsing (only the new data)
        mpack_tree_add_data(&tree, len);
    
        // process all new objects from the start of the buffer
        while (mpack_tree_parse(&tree)) {
            mpack_node_t node = mpack_tree_root(&tree);
            processNode(node);
        }
    
        // remove the processed data from the buffer, move any additional data to the buffer start
        size_t processedBytes = mpack_tree_remove_processed_data(&tree);
        bufferSize -= processedBytes;
    }
    

    Maybe the example is not perfectly correct, but I hope you get the idea :slightly_smiling_face: Such an API would be very useful for me. Do you know if something like that is possible, either with mpack or with any other C/C++ library?

    Thanks in advance.

    feature request 
    opened by ubruhin 2
  • builder page allocation leak and segfault

    builder page allocation leak and segfault

    hi, I ran into a segfault with the following code:

    mpack_writer_t writer;
    
    char *output_data = nullptr;
    size_t output_size = 0;
    mpack_writer_init_growable(&writer, &output_data, &output_size);
    
    
    const int N_OUTER = 10;
    const int N_INNER = 100;
    
    mpack_start_array(&writer, N_OUTER);
    for (int k=0; k<N_OUTER; k++) {
    	mpack_build_map(&writer);
    	mpack_write_str(&writer, "key", 3);
    	mpack_write_u64(&writer, k*k);
    	mpack_write_str(&writer, "value", 5);
    	mpack_start_array(&writer, N_INNER);
    	for (int i=0; i<N_INNER; i++) {
    		mpack_build_map(&writer);
    		mpack_write_str(&writer, "value0", 6);
    		mpack_write_i64(&writer, i);
    		mpack_write_str(&writer, "value1", 6);
    		mpack_write_i64(&writer, i*k);
    		mpack_complete_map(&writer);
    	}
    	mpack_finish_array(&writer);
    	mpack_complete_map(&writer); // <- segfault here
    }
    mpack_finish_array(&writer);
    
    
    mpack_writer_destroy(&writer);
    

    At the end of the 5th outer loop, the call to mpack_complete_map() will corrupt the internal memory of the builder, resulting in a segfault. Some mild debugging traced the issue back to the page allocation mechanism. According to the logs, we might be freeing one page and using the next without initializing it.

    The problem vanishes when writing different integer values. It seems to be triggered only by very specific alignment conditions.

    I would appreciate it ,if you could confirm the findings or give me a clue of any mismatching calls in my code. Thanks!

    fixed; needs better tests 
    opened by momentarylapse 4
  • mpack crashes without a matching call to mpack_complete_*

    mpack crashes without a matching call to mpack_complete_*

    Taking the example from the README:

    #include <stdlib.h>
    #include <stdio.h>
    #include <mpack.h>
    
    int main()
    {
        char* data;
        size_t size;
        mpack_writer_t writer;
        mpack_writer_init_growable(&writer, &data, &size);
    
        // write the example on the msgpack homepage
        mpack_build_map(&writer);
        mpack_write_cstr(&writer, "compact");
        mpack_write_bool(&writer, true);
        mpack_write_cstr(&writer, "schema");
        mpack_write_uint(&writer, 0);
        //mpack_complete_map(&writer);
    
        // finish writing
        if (mpack_writer_destroy(&writer) != mpack_ok) {
            fprintf(stderr, "An error occurred encoding the data!\n");
            return 1;
        }
    
        // use the data
        printf("%.*s", (int)size, data);
        free(data);
        return 0;
    }
    
    glopes /tmp $ gcc -o t -g -I/tmp a.c mpack.c && valgrind -q ./t
    ==14417== Invalid free() / delete / delete[] / realloc()
    ==14417==    at 0x4C37DAD: realloc (vg_replace_malloc.c:1192)
    ==14417==    by 0x108D37: mpack_realloc (mpack.h:1842)
    ==14417==    by 0x10ABAB: mpack_growable_writer_teardown (mpack.c:1208)
    ==14417==    by 0x10B420: mpack_writer_destroy (mpack.c:1499)
    ==14417==    by 0x108C95: main (a.c:21)
    ==14417==  Address 0x52360b0 is 48 bytes inside a block of size 4,096 alloc'd
    ==14417==    at 0x4C32FB5: malloc (vg_replace_malloc.c:380)
    ==14417==    by 0x10DC1F: mpack_builder_begin (mpack.c:2470)
    ==14417==    by 0x10DCDF: mpack_builder_build (mpack.c:2495)
    ==14417==    by 0x10E1EC: mpack_build_map (mpack.c:2711)
    ==14417==    by 0x108C32: main (a.c:13)
    An error occurred encoding the data!
    

    This makes it impossible to use error chaining. In this case, I omitted the call to mpack_complete_map altogether, but it can also be skipped because of an earlier error. In fact, it's a problem even with:

    mpack_build_map(&writer);
    mpack_write_cstr(&writer, "foo");
    mpack_complete_map(&writer);
    mpack_writer_destroy(&writer);
    
    fixed; needs better tests 
    opened by cataphract 3
Releases(v1.1)
  • v1.1(Aug 23, 2021)

    New Features:

    • Maps and arrays can now be built dynamically without specifying their size up front. See mpack_build_map() and mpack_build_array().

    New Platforms:

    • Compiling as gnu89 is now supported. (See #68, #69)

    • Compiling in the Linux kernel is now possible using a standalone configuration file. (See #80)

    • Compiling for AVR (e.g. Arduino) and other microcontrollers is now supported. MPack now compiles cleanly on platforms with 16-bit int and size_t. (See #74, #79)

    • float and/or double can now be disabled individually on platforms with limited floating point support. If float is supported but not double, MessagePack doubles can be converted to float. (See #74, #79)

    • MPack now builds cleanly under /W4 with Visual Studio 2015, 2017 and 2019 build tools.

    Bug Fixes and Other Changes:

    • An mpack-defaults.h sample configuration is no longer provided.

    • Replaced SCons unit test buildsystem and XCode/VS projects with Python+Ninja.

    • Fixed an issue where write overloads could be erroneously defined in C++ without MPACK_WRITER (#66).

    • Fixed some minor portability issues.

    Source code(tar.gz)
    Source code(zip)
    mpack-amalgamation-1.1.tar.gz(407.12 KB)
  • v1.0(Oct 21, 2018)

    A number of breaking API changes have been made for the 1.0 release. Please take note of these changes when upgrading.

    Breaking Changes:

    • The Node API now separates tree initialization from parsing. After calling one of the mpack_tree_init() functions, you must explicitly call mpack_tree_parse() before accessing any nodes.

    • The configuration file mpack-config.h is now optional, and requires MPACK_HAS_CONFIG in order to be included. This means you must define MPACK_HAS_CONFIG when upgrading or your config file will be ignored!

    • Extension types are now disabled by default. You must define MPACK_EXTENSIONS to use them.

    • mpack_tag_t is now considered an opaque type to prevent future breakage when changing its layout. Compatibility is maintained for this release, but this may change in future releases.

    New Features:

    • The Node API can now parse multiple messages from a data source. mpack_tree_parse() can be called repeatedly to parse each message.

    • The Node API can now parse messages indefinitely from a continuous stream. A tree can be initialized with mpack_tree_init_stream() to receive a callback for more data.

    • The Node API can now parse messages incrementally from a non-blocking stream. Call mpack_tree_try_parse() with a non-blocking read function to start and resume parsing. It will return true when a complete message has become available.

    • The stdio helpers now allow reading from a FILE*. _init_file() functions have been renamed to _init_filename(). (The old names will continue to work for a few more versions.)

    • The Node API now returns a node of "missing" type instead of "nil" type for optional map lookups. This allows the caller to tell the difference between a key having value nil and a missing key.

    • The writer now supports a v4 compatibility mode. Call mpack_writer_set_version(writer, mpack_version_v4); to encode without using the raw8, bin and ext types. (This requires MPACK_COMPATIBILITY.)

    • The timestamp type has been implemented. A timestamp is a signed number of nanoseconds since the Unix epoch (1970-01-01T00:00:00Z). (This requires MPACK_EXTENSIONS.)

    Bug Fixes and Other Changes:

    • Fixed an allocation bug when closing a growable writer without having written anything (#58).

    • The reader's skip function is no longer ignored under MPACK_OPTIMIZE_FOR_SIZE.

    Source code(tar.gz)
    Source code(zip)
    mpack-amalgamation-1.0.tar.gz(372.69 KB)
  • v0.8.2(Apr 10, 2016)

    Changes:

    • Fixed incorrect element tracking in mpack_write_tag()
    • Added type-generic writer functions mpack_write() and mpack_write_kv()
    • Added mpack_write_object_bytes() to insert pre-encoded MessagePack into a larger message
    • Enabled strings in all builds by default
    • Fixed unit test errors under -ffast-math
    • Fixed some compiler warnings
    Source code(tar.gz)
    Source code(zip)
    mpack-amalgamation-0.8.2.tar.gz(294.93 KB)
  • v0.8.1(Jan 11, 2016)

  • v0.8(Jan 5, 2016)

  • v0.7.1(Nov 29, 2015)

    Changes:

    • Removed mpack_reader_destroy_cancel() and mpack_writer_destroy_cancel(). You must now flag an error (such as mpack_error_data) in order to cancel reading.
    • Added many code size optimizations. MPACK_OPTIMIZE_FOR_SIZE is no longer experimental.
    • Improved and reorganized Writer documentation
    • Made writer flag mpack_error_too_big instead of mpack_error_io if writing too much data without a flush callback
    • Added optional skip callback and optimized mpack_discard()
    • Fixed various compiler and code analysis warnings
    • Optimized speed and memory usage
    Source code(tar.gz)
    Source code(zip)
    mpack-amalgamation-0.7.1.tar.gz(358.77 KB)
  • v0.7(Oct 31, 2015)

  • v0.6(Sep 27, 2015)

    Changes:

    • setjmp/longjmp support has been replaced by error callbacks. You can safely longjmp or throw C++ exceptions out of error callbacks. Be aware of local variable invalidation rules regarding setjmp if you use it. See the documentation for mpack_reader_error_t and issue #19 for more details.
    • All inline functions in the MPack API are no longer static. A single non-inline definition of each inline function is emitted, so they behave like normal functions with external linkage.
    • Configuration options can now be pre-defined before including mpack-config.h, so you can customize MPack by defining these in your build system rather than editing the configuration file.
    Source code(tar.gz)
    Source code(zip)
    mpack-amalgamation-0.6.tar.gz(319.69 KB)
  • v0.5.1(Aug 16, 2015)

  • v0.5(Jul 28, 2015)

    Changes:

    • mpack_node_t is now a handle, so it should be passed by value, not by pointer. Porting to the new version should be as simple as replacing mpack_node_t* with mpack_node_t in your code.
    • Various other minor API changes have been made.
    • Major performance improvements were made across all aspects of MPack.
    Source code(tar.gz)
    Source code(zip)
    mpack-amalgamation-0.5.tar.gz(315.12 KB)
  • v0.4(Jun 21, 2015)

  • v0.3(Apr 7, 2015)

  • v0.2(Mar 17, 2015)

Owner
Nicholas Fraser
Nicholas Fraser
MessagePack implementation for C and C++ / msgpack.org[C/C++]

msgpack for C/C++ It's like JSON but smaller and faster. Overview MessagePack is an efficient binary serialization format, which lets you exchange dat

MessagePack 2.6k Dec 5, 2022
Msgpack11 - A tiny MessagePack library for C++11 (msgpack.org[C++11])

What is msgpack11 ? msgpack11 is a tiny MsgPack library for C++11, providing MsgPack parsing and serialization. This library is inspired by json11. Th

Masahiro Wada 99 Nov 4, 2022
Your binary serialization library

Bitsery Header only C++ binary serialization library. It is designed around the networking requirements for real-time data delivery, especially for ga

Mindaugas Vinkelis 760 Dec 1, 2022
Microsoft 2.5k Nov 28, 2022
Cap'n Proto serialization/RPC system - core tools and C++ library

Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except

Cap'n Proto 9.4k Dec 1, 2022
A C++11 library for serialization

cereal - A C++11 library for serialization cereal is a header-only C++11 serialization library. cereal takes arbitrary data types and reversibly turns

iLab @ USC 3.4k Dec 2, 2022
Fast Binary Encoding is ultra fast and universal serialization solution for C++, C#, Go, Java, JavaScript, Kotlin, Python, Ruby, Swift

Fast Binary Encoding (FBE) Fast Binary Encoding allows to describe any domain models, business objects, complex data structures, client/server request

Ivan Shynkarenka 646 Dec 1, 2022
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Google 19.4k Nov 27, 2022
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

niXman 583 Nov 18, 2022
Binary Serialization

Binn Binn is a binary data serialization format designed to be compact, fast and easy to use. Performance The elements are stored with their sizes to

null 381 Nov 24, 2022
Simple C++ 20 Serialization Library that works out of the box with aggregate types!

BinaryLove3 Simple C++ 20 Serialization Library that works out of the box with aggregate types! Requirements BinaryLove3 is a c++20 only library.

RedSkittleFox 14 Sep 2, 2022
Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications

Zmeya Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serializ

Sergey Makeev 97 Nov 2, 2022
CppSerdes is a serialization/deserialization library designed with embedded systems in mind

A C++ serialization/deserialization library designed with embedded systems in mind

Darren V Levine 79 Nov 5, 2022
Serialization framework for Unreal Engine Property System that just works!

DataConfig Serialization framework for Unreal Engine Property System that just works! Unreal Engine features a powerful Property System which implemen

null 80 Nov 27, 2022
Header-only library for automatic (de)serialization of C++ types to/from JSON.

fuser 1-file header-only library for automatic (de)serialization of C++ types to/from JSON. how it works The library has a predefined set of (de)seria

null 50 Oct 20, 2022
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

niXman 455 Sep 7, 2021
C++17 library for all your binary de-/serialization needs

blobify blobify is a header-only C++17 library to handle binary de-/serialization in your project. Given a user-defined C++ struct, blobify can encode

Tony Wasserka 247 Oct 20, 2022
universal serialization engine

A Universal Serialization Engine Based on compile-time Reflection iguana is a modern, universal and easy-to-use serialization engine developed in c++1

qicosmos 677 Dec 2, 2022
Yet another JSON/YAML/BSON serialization library for C++.

ThorsSerializer Support for Json Yaml Bson NEW Benchmark Results Conformance mac linux Performance max linux For details see: JsonBenchmark Yet anothe

Loki Astari 279 Oct 27, 2022