Minimalist protocol buffer decoder and encoder in C++

Overview

protozero

Minimalistic protocol buffer decoder and encoder in C++.

Designed for high performance. Suitable for writing zero copy parsers and encoders with minimal need for run-time allocation of memory.

Low-level: this is designed to be a building block for writing a very customized decoder for a stable protobuf schema. If your protobuf schema is changing frequently or lazy decoding is not critical for your application then this approach offers no value: just use the C++ API that can be generated with the Google Protobufs protoc program.

Travis Build Status Appveyor Build Status Coverage Status Packaging status

Depends

  • C++11 compiler
  • CMake
  • Some tests depend on the Google Protobuf library, but use of Protozero doesn't need it

How it works

The protozero code does not read .proto files used by the usual Protobuf implementations. The developer using protozero has to manually "translate" the .proto description into code. This means there is no way to access any of the information from the .proto description. This results in a few restrictions:

  • The names of the fields are not available.
  • Enum names are not available, you'll have to use the values they are defined with.
  • Default values are not available.
  • Field types have to be hardcoded. The library does not know which types to expect, so the user of the library has to supply the right types. Some checks are made using assert(), but mostly the user has to take care of that.

The library will make sure not to overrun the buffer it was given, but basically all other checks have to be made in user code!

Documentation

You have to have a working knowledge of how protocol buffer encoding works.

The build process will also build the Doxygen-based reference documentation if you have Doxygen installed. Then open doc/html/index.html in your browser to read it.

Endianness

Protozero uses a very simplistic test to check the byte order of the system it compiles on. If this check is wrong, you'll get test failures. If this is the case, please open an issue and tell us about your system.

Building tests

Extensive tests are included. Build them using CMake:

mkdir build
cd build
cmake ..
make

Call ctest to run the tests.

The unit and reader tests are always build, the writer tests are only build if the Google Protobuf library is found when running CMake.

See test/README.md for more details about the test.

Coverage report

To get a coverage report set CXXFLAGS and LDFLAGS before calling CMake:

CXXFLAGS="--coverage" LDFLAGS="--coverage" cmake ..

Then call make as usual and run the tests using ctest.

If you are using g++ use gcov to generate a report (results are in *.gcov files):

gcov -lp $(find test/ -name '*.o')

If you are using clang++ use llvm-cov instead:

llvm-cov gcov -lp $(find test/ -name '*.o')

If you are using g++ you can use gcovr to generate nice HTML output:

mkdir -p coverage
gcovr . -r SRCDIR --html --html-details -o coverage/index.html

Open coverage/index.html in your browser to see the report.

Clang-tidy

After the CMake step, run

make clang-tidy

to check the code with clang-tidy. You might have to set CLANG_TIDY in CMake config.

Cppcheck

For extra checks with Cppcheck you can, after the CMake step, call

make cppcheck

Installation

After the CMake step, call make install to install the include files in /usr/local/include/protozero.

If you are using CMake to build your own software, you can copy the file cmake/FindProtozero.cmake and use it in your build. See the file for details.

Who is using Protozero?

Are you using Protozero? Tell us! Send a pull request with changes to this README.

Issues
  • string_view writer

    string_view writer

    It is not uncommon to preallocate I/O buffers of fixed-length in low-latency applications to achieve maximum performance. Protozero is a great library, but does not currently work well with fixed-length buffers.

    Once string_view becomes widely available in c++17, it would be helpful to support the zero-copy use case. The writer class could accept a string_view and an offset and either throw an exception or set an error flag and switch to no-ops to prevent a buffer overflow when the allocated space has been consumed.

    enhancement 
    opened by tkohlman 16
  • Optimizing varint

    Optimizing varint

    Decoding varints is a major bottleneck. When doing profiling, the decode_varint function always shows up high in the list. I have done some experiments which show that over half the varints in OSM PBF files are 1-byte varints. Optimizing this specific case speeds up reading of those files by about 10%! I optimized this by using a wrapper function around decode_varint that checks for this specific case and handles it locally and otherwise hands work over to the original decode_varint function. I believe the compiler inlines the outer function which reduces the overhead for this common case. Forcing the compiler to inline the whole function didn't do much, the function is too big to be inlined anyway.

    There are some more things we need to figure out here:

    • Is the OSM PBF unusual in some way that it has so many 1-byte varints? Part of this is that Protobuf uses varints for the field tags (which say which type of field something is). If the tag is <15, this will be encoded in 1 byte varint. I suspect that most Protobuf formats will do this, because this optimization is specifically mention in the Protobuf documentation. And OSM PBF uses delta encoding of IDs etc. to get small integers in several places, so this also helps. But do other formats have similar rations? We have to check how vector tiles fare here.
    • Does it make sense to optimize this, or could we make some other optimization that would help in all cases. The varint decoder we use is from folly, I suspect they have done their homework on this, but we don't know their exact circumstances.
    • Can we do some general optimization that just works in most cases, or do we want the programmer to give hints to the library, such as what size integer to expect or so.

    I looked at the SIMD speedup things mentioned in #20 and #27, but that code is huge. Folly has some code to speed up multiple varints stored together which could help with repeated packed varints, but this will not help with the very common case of the varint-encoded tag.

    enhancement 
    opened by joto 14
  • Byte order

    Byte order

    The fixed size non-varint number types float, double, fixed32, and fixed64 are stored little-endian in Protobuf. There is no special handling in phf.hpp, so it will only work on little-endian machines.

    opened by joto 13
  • Tests fail

    Tests fail

    FreeBSD 12.0 amd64, clang 6.0.1, protozero 1.6.5, protobuf 3.6.1.

    unit_tests fail:

    % ./unit/unit_tests 
    
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    unit_tests is a Catch v1.12.0 host application.
    Run with -? for options
    
    -------------------------------------------------------------------------------
    varint
      encode/decode int32
    -------------------------------------------------------------------------------
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/unit/test_varint.cpp:12
    ...............................................................................
    
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/unit/test_varint.cpp:25: FAILED:
      REQUIRE_FALSE( item.next() )
    with expansion:
      !true
    
    -------------------------------------------------------------------------------
    varint
      encode/decode uint32
    -------------------------------------------------------------------------------
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/unit/test_varint.cpp:28
    ...............................................................................
    
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/unit/test_varint.cpp:41: FAILED:
      REQUIRE_FALSE( item.next() )
    with expansion:
      !true
    
    -------------------------------------------------------------------------------
    varint
      encode/decode uint64
    -------------------------------------------------------------------------------
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/unit/test_varint.cpp:44
    ...............................................................................
    
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/unit/test_varint.cpp:57: FAILED:
      REQUIRE_FALSE( item.next() )
    with expansion:
      item.next()
    due to unexpected exception with message:
      invalid tag exception
    
    ===============================================================================
    test cases:      49 |      48 passed | 1 failed
    assertions: 1052438 | 1052435 passed | 3 failed
    

    reader_tests fail:

    ...
    -------------------------------------------------------------------------------
    check assert on non-fixed access to fixed64
    -------------------------------------------------------------------------------
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/t/wrong_type_access/reader_test_cases.cpp:18
    ...............................................................................
    
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/t/wrong_type_access/reader_test_cases.cpp:18: FAILED:
    due to unexpected exception with message:
      could not open: 'fixed64/data-zero'
    
    -------------------------------------------------------------------------------
    check assert on non-string access to string
    -------------------------------------------------------------------------------
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/t/wrong_type_access/reader_test_cases.cpp:31
    ...............................................................................
    
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/t/wrong_type_access/reader_test_cases.cpp:31: FAILED:
    due to unexpected exception with message:
      could not open: 'string/data-string'
    
    -------------------------------------------------------------------------------
    check assert on non-fixed access to fixed32
    -------------------------------------------------------------------------------
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/t/wrong_type_access/reader_test_cases.cpp:44
    ...............................................................................
    
    /ssd/portstrees/sdl_gfx/devel/protozero/work/protozero-1.6.5/test/t/wrong_type_access/reader_test_cases.cpp:44: FAILED:
    due to unexpected exception with message:
      could not open: 'fixed32/data-zero'
    
    ===============================================================================
    test cases:  182 |  16 passed | 166 failed
    assertions: 1070 | 624 passed | 446 failed
    
    

    Most fail because they don't specify .pbf file extension. Full log: https://pastebin.com/zxUgJFy5

    opened by AMDmi3 12
  • Alignment faults running tests on ARM

    Alignment faults running tests on ARM

    I'm seeing a SIGBUS fault running the tests on an ARM system. The backtrace is:

    #0  Catch::ExpressionLhs<float const&>::captureExpression<(Catch::Internal::Operator)0, float> (this=0xbeffe2cc, [email protected]=0xbeffe388, [email protected]: 17.3400002) at /usr/include/catch/internal/catch_expression_lhs.hpp:90
    #1  0x0006d0ec in Catch::ExpressionLhs<float const&>::operator==<float> ([email protected]: 7.73589176e+34, this=0xbeffe2c4) at /usr/include/catch/internal/catch_expression_lhs.hpp:35
    #2  ____C_A_T_C_H____T_E_S_T____4 () at test/t/repeated_packed_float/test_cases.cpp:23
    #3  0x00042458 in Catch::FreeFunctionTestCase::invoke (this=<optimized out>) at /usr/include/catch/internal/catch_test_case_registry_impl.hpp:116
    #4  Catch::TestCase::invoke (this=<optimized out>) at /usr/include/catch/internal/catch_test_case_info.hpp:169
    #5  Catch::RunContext::invokeActiveTestCase (this=0xbeffed24) at /usr/include/catch/internal/catch_runner_impl.hpp:298
    #6  Catch::RunContext::runCurrentTest (redirectedCerr="", redirectedCout="", this=0xbeffed24) at /usr/include/catch/internal/catch_runner_impl.hpp:270
    #7  Catch::RunContext::runTest (testCase=..., this=0xbeffed24) at /usr/include/catch/internal/catch_runner_impl.hpp:108
    #8  Catch::Runner::runTests (this=0x25be4 <Catch::FatalConditionHandler::handleSignal(int)>, [email protected]=0xbefff178) at /usr/include/catch/catch_runner.hpp:59
    #9  0x00042ef8 in Catch::Session::run ([email protected]=0xbefff344) at /usr/include/catch/catch_runner.hpp:186
    #10 0x00015324 in Catch::Session::run (argv=<optimized out>, argc=<optimized out>, this=0xbefff344) at /usr/include/catch/catch_runner.hpp:166
    #11 main (argc=<optimized out>, argv=<optimized out>) at /usr/include/catch/internal/catch_default_main.hpp:15
    

    The faulting instruction is a vldr instruction loading the float referenced by *it_pair.first when loading the float from repeated_packed_float/data-one because the address returned is only two byte aligned, but NEON instructions like that require four byte alignment for floats.

    I don't think there's an easy fix here - obviously having the actual reader do anything to guarantee alignment is a potentially large performance hit. The alternative is to have the tests copy the value out to an aligned location before testing it.

    I guess the question is, what is the pbf_reader supposed to be guaranteeing about the iterators it returns for packed values?

    opened by tomhughes 12
  • Some tests fail on FreeBSD 11.1 with clang/libc++

    Some tests fail on FreeBSD 11.1 with clang/libc++

    When built with gcc/libstdc++ the tests pass, but when build with clang/libc++ (which is the default on FreeBSD):

    Test project /ssd/portstrees/batchports-mem/devel/protozero/work/protozero-1.6.2                                                                                                                                                                                     
        Start 1: pbf-decoder-no-args                                                                                                                                                                                                                                     
    1/9 Test #1: pbf-decoder-no-args ..............   Passed    0.00 sec                                                                                                                                                                                                 
        Start 2: pbf-decoder-help                                                                                                                                                                                                                                        
    2/9 Test #2: pbf-decoder-help .................   Passed    0.00 sec                                                                                                                                                                                                 
        Start 3: pbf-decoder-empty                                                                                                                                                                                                                                       
    3/9 Test #3: pbf-decoder-empty ................   Passed    0.00 sec                                                                                                                                                                                                 
        Start 4: pbf-decoder-data                                                                                                                                                                                                                                        
    4/9 Test #4: pbf-decoder-data .................   Passed    0.00 sec                                                                                                                                                                                                 
        Start 5: pbf-decoder-vt                                                                                                                                                                                                                                          
    5/9 Test #5: pbf-decoder-vt ...................   Passed    0.39 sec                                                                                                                                                                                                 
        Start 6: pbf-decoder-fail                                                                                                                                                                                                                                        
    6/9 Test #6: pbf-decoder-fail .................   Passed    0.01 sec                                                                                                                                                                                                 
        Start 7: pbf-decoder-fail-msg                                                                                                                                                                                                                                    
    7/9 Test #7: pbf-decoder-fail-msg .............   Passed    0.01 sec                                                                                                                                                                                                 
        Start 8: reader_tests                                                                                                                                                                                                                                            
    8/9 Test #8: reader_tests .....................   Passed    0.03 sec                                                                                                                                                                                                 
        Start 9: unit_tests                                                                                                                                                                                                                                              
    9/9 Test #9: unit_tests .......................***Failed    0.06 sec                                                                                                                                                                                                 
    
    % test/unit/unit_tests 
                                                                                                                                                                                                                                                                         
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                                                      
    unit_tests is a Catch v1.12.0 host application.                                                                                                                                                                                                                      
    Run with -? for options                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                         
    -------------------------------------------------------------------------------                                                                                                                                                                                      
    next() should throw when illegal tag is encountered                                                                                                                                                                                                                  
    -------------------------------------------------------------------------------                                                                                                                                                                                      
    test/unit/test_basic.cpp:40                                                                                                                                                                                                                                          
    ...............................................................................                                                                                                                                                                                      
                                                                                                                                                                                                                                                                         
    test/unit/test_basic.cpp:60: FAILED:                                                                                                                                                                                                                                 
      REQUIRE_THROWS_AS( item.next(), const protozero::invalid_tag_exception& )                                                                                                                                                                                          
    because no exception was thrown where one was expected:                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                         
    -------------------------------------------------------------------------------                                                                                                                                                                                      
    next() works when a legal tag is encountered                                                                                                                                                                                                                         
    -------------------------------------------------------------------------------                                                                                                                                                                                      
    test/unit/test_basic.cpp:63                                                                                                                                                                                                                                          
    ...............................................................................                                                                                                                                                                                      
                                                                                                                                                                                                                                                                         
    test/unit/test_basic.cpp:83: FAILED:                                                                                                                                                                                                                                 
      REQUIRE( item.next() )                                                                                                                                                                                                                                             
    with expansion:                                                                                                                                                                                                                                                      
      false                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                         
    ===============================================================================                                                                                                                                                                                      
    test cases:      37 |      35 passed | 2 failed                                                                                                                                                                                                                      
    assertions: 1051517 | 1051515 passed | 2 failed                                                                    
    

    Haven't investigated further yet.

    opened by AMDmi3 10
  • array-bounds error in nested_testcase.pb.cc

    array-bounds error in nested_testcase.pb.cc

    I'm trying to build protozero 1.6.8 with protobuf 3.7.0 and gcc 9.1.0. The prior version 1.6.7 has had worked, this one fails at compilation:

    In member function ‘void TestNested::Sub::SharedCtor()’,
        inlined from ‘TestNested::Sub::Sub()’ at /build/protozero/src/protozero-1.6.8/build/test/t/nested/nested_testcase.pb.cc:412:13,
        inlined from ‘void InitDefaultsSub_nested_5ftestcase_2eproto()’ at /build/protozero/src/protozero-1.6.8/build/test/t/nested/nested_testcase.pb.cc:410:1:
    /build/protozero/src/protozero-1.6.8/build/test/t/nested/nested_testcase.pb.cc:432:11: error: ‘void* memset(void*, int, size_t)’ offset [33, 36] from the object at ‘TestNested::_Sub_default_instance_’ is out of the bounds of referenced subobject ‘TestNested::Sub::subsub_’ with type ‘TestNested::SubSub*’ at offset 24 [-Werror=array-bounds]
      432 |   ::memset(&subsub_, 0, static_cast<size_t>(
          |   ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      433 |       reinterpret_cast<char*>(&i_) -
          |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      434 |       reinterpret_cast<char*>(&subsub_)) + sizeof(i_));
          |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    In member function ‘void TestNested::Test::SharedCtor()’,
        inlined from ‘TestNested::Test::Test()’ at /build/protozero/src/protozero-1.6.8/build/test/t/nested/nested_testcase.pb.cc:720:13,
        inlined from ‘void InitDefaultsTest_nested_5ftestcase_2eproto()’ at /build/protozero/src/protozero-1.6.8/build/test/t/nested/nested_testcase.pb.cc:718:1:
    /build/protozero/src/protozero-1.6.8/build/test/t/nested/nested_testcase.pb.cc:740:11: error: ‘void* memset(void*, int, size_t)’ offset [33, 36] from the object at ‘TestNested::_Test_default_instance_’ is out of the bounds of referenced subobject ‘TestNested::Test::sub_’ with type ‘TestNested::Sub*’ at offset 24 [-Werror=array-bounds]
      740 |   ::memset(&sub_, 0, static_cast<size_t>(
          |   ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      741 |       reinterpret_cast<char*>(&i_) -
          |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      742 |       reinterpret_cast<char*>(&sub_)) + sizeof(i_));
          |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    At global scope:
    cc1plus: error: unrecognized command line option ‘-Wno-covered-switch-default’ [-Werror]
    cc1plus: all warnings being treated as errors
    make[2]: *** [test/CMakeFiles/writer_tests.dir/build.make:387: test/CMakeFiles/writer_tests.dir/t/nested/nested_testcase.pb.cc.o] Error 1
    make[1]: *** [CMakeFiles/Makefile2:167: test/CMakeFiles/writer_tests.dir/all] Error 2
    make: *** [Makefile:141: all] Error 2
    
    opened by chazanov 7
  • Type-safe named access to protobuf fields.

    Type-safe named access to protobuf fields.

    One problem with Protozero has always been that it doesn't read the .proto file so it doesn't know about symbolic names of messages and fields and it can't make sure you are accessing the protobuf data using the right types. There is an attempt of fixing some of these problems with the pbf_message and pbf_builder classes, but it is somewhat ugly and only solves half of the problems.

    I tried out a new approach in the v2-experimental branch. It uses some template and macro magic to make most of what we want possible. It doesn't read the .proto file, but it allows putting the same information into the C++ file using macros in a way that is very easy to translate from the .proto file:

    PROTOZERO_MESSAGE(Sub) {
      PROTOZERO_FIELD(string, s, 1);
    };
    
    PROTOZERO_MESSAGE(Test) {
      PROTOZERO_FIELD(fixed32, f, 1);
      PROTOZERO_FIELD(int64, i, 2);
      PROTOZERO_FIELD(int64, j, 3);
      PROTOZERO_MESSAGE_FIELD(Sub, submessage, 5);
      PROTOZERO_FIELD(string, s, 8);
      PROTOZERO_FIELD(uint32, u, 4);
      PROTOZERO_PACKED_FIELD(sint32, d, 7);
    };
    

    Access is then through macros called FIELD_TAG() and FIELD_VALUE():

    protozero::message<Test> item(buffer);
    while (item.next()) {
        switch (item.tag()) {
            case FIELD_TAG(item, f): {
                auto value = FIELD_VALUE(item, f);
                ...
                break;
            }
            case FIELD_TAG(item, submessage): {
                auto subitem = FIELD_VALUE(item, submessage);
                subitem.next();
                auto str = FIELD_VALUE(subitem, s);
                ...
                break;
            }
        }
    }
    

    The code in Protozero is too complicated for my liking, but there are probably some simplifications and cleanups we can do. But using this is, I think, rather straightforward. And if the PROTOZERO_MESSAGE() and PROTOZERO_FIELD() macros are correct messages and fields are accessed by name and it is not possible any more to get the types wrong.

    What do you think?

    /cc @springmeyer @daniel-j-h

    enhancement question 
    opened by joto 7
  • First try at integrating Fuzz Testing

    First try at integrating Fuzz Testing

    This is a first try at Coverage Based Fuzz Testing (plus Data-flow-guided fuzzing) using LLVM's libFuzzer.

    It's building a driver for (at the moment only) the varint zigzag functions, then compiles it instrumenting the binary with coverage information. It then spins up the test driver (that gets a main from libFuzzer), dumping the corpus that changes control-flow into the fuzz/corpus directory. (This can be kept across runs to speed up fuzzing)

    It's best to compile the driver with multiple sanitizers (ubsan, asan, msan, ..), and fuzz the library with those. Set env var

    env FUZZ_SANITIZER=
    

    for this to e.g. undefined or address or memory, respectively.

    Note: -jobs=N can be passed to the driver, letting it fork and run N jobs in parallel; not doing this at the moment.

    I tried this with LLVM 3.8 / libc++ on Linux as it needs a fairly recent LLVM release; 3.7 should work, too, but I haven't tested this.

    Disclaimer: I'm already doing a pull request although it's only fuzzing the varint zigzag functions to get more traction for fuzzing across projects.

    References:

    • http://llvm.org/docs/LibFuzzer.html
    • http://llvm.org/releases/3.8.0/docs/LibFuzzer.html
    • https://www.youtube.com/watch?v=qTkYDA0En6U
    • https://github.com/Project-OSRM/osrm-backend/pull/2251

    /cc @TheMarex @joto @artemp @springmeyer @kkaefer @ericfischer

    env FUZZ_SANITIZER='undefined,integer' CC='clang' CXX='clang++' CXXFLAGS="-stdlib=libc++" LDFLAGS="-stdlib=libc++" make fuzz
    
    ./fuzz/varint -use_traces=1 fuzz/corpus
    Seed: 1864466908
    PreferSmall: 1
    0      READ   units: 1 exec/s: 0
    1      INITED cov: 32 bits: 32 units: 1 exec/s: 0
    2      NEW    cov: 39 bits: 39 indir: 1 units: 2 exec/s: 0 L: 64 MS: 0
    112    NEW    cov: 40 bits: 46 indir: 1 units: 3 exec/s: 0 L: 64 MS: 0
    559    NEW    cov: 66 bits: 72 indir: 1 units: 4 exec/s: 0 L: 1 MS: 3 EraseByte-CrossOver-EraseByte-
    637    NEW    cov: 66 bits: 74 indir: 1 units: 5 exec/s: 0 L: 2 MS: 1 InsertByte-
    639    NEW    cov: 66 bits: 79 indir: 1 units: 6 exec/s: 0 L: 3 MS: 3 InsertByte-ShuffleBytes-InsertByte-
    642    NEW    cov: 66 bits: 86 indir: 1 units: 7 exec/s: 0 L: 4 MS: 1 CrossOver-
    653    NEW    cov: 66 bits: 87 indir: 1 units: 8 exec/s: 0 L: 23 MS: 2 ShuffleBytes-CrossOver-
    1364   NEW    cov: 66 bits: 93 indir: 1 units: 9 exec/s: 0 L: 27 MS: 3 CrossOver-CrossOver-CrossOver-
    1417   NEW    cov: 66 bits: 94 indir: 1 units: 10 exec/s: 0 L: 31 MS: 1 CrossOver-
    3473   NEW    cov: 66 bits: 101 indir: 1 units: 11 exec/s: 0 L: 62 MS: 2 ChangeByte-CrossOver-
    4098   NEW    cov: 69 bits: 104 indir: 1 units: 12 exec/s: 0 L: 14 MS: 2 EraseByte-CrossOver-
    4602   NEW    cov: 72 bits: 107 indir: 1 units: 13 exec/s: 0 L: 28 MS: 1 CrossOver-
    5337   NEW    cov: 72 bits: 114 indir: 1 units: 14 exec/s: 0 L: 44 MS: 1 CrossOver-
    2097152        pulse  cov: 72 bits: 114 indir: 1 units: 14 exec/s: 699050
    4194304        pulse  cov: 72 bits: 114 indir: 1 units: 14 exec/s: 599186
    8388608        pulse  cov: 72 bits: 114 indir: 1 units: 14 exec/s: 599186
    
    opened by daniel-j-h 7
  • check_swap_1(-1) fails on various architectures

    check_swap_1(-1) fails on various architectures

    The Debian package build for protozero 1.2.0 failed on arm64, armel, powerpc, ppc64el, s390x & ppc64 (both big and little endian architectures) with the same test failure:

    tests is a Catch v1.2.1 host application.
    Run with -? for options
    
    -------------------------------------------------------------------------------
    byte swapping
    -------------------------------------------------------------------------------
    test/t/endian/test_cases.cpp:42
    ...............................................................................
    
    test/t/endian/test_cases.cpp:45: FAILED:
      REQUIRE( -1 == check_swap_1(-1) )
    with expansion:
    
    ===============================================================================
    test cases:   86 |   85 passed | 1 failed
    assertions: 4510 | 4509 passed | 1 failed
    

    Please advise how to best help troubleshoot this issue.

    opened by sebastic 6
  • fast and safe varint impl

    fast and safe varint impl

    Creating a ticket to document which varint implementation we are using and why.

    Original one was from https://github.com/haberman/upb. I also experimented with an optimized one from folly, but removed it in #1 since we did not need two and I saw no distinct benefit from the folly impl.

    But then @joto found a bug (I don't see a specific ticket on this) in the upb impl and switched formally to the folly impl in https://github.com/mapbox/pbf.hpp/commit/bbfddea68bc5873ba694ce6076cc9fa9f24e564f.

    Tonight I ran the old version of pbf (used in mbgl) through https://scan.coverity.com which found:

    screen shot 2015-05-03 at 10 58 58 pm

    @joto - is this related perhaps to the bug you found?

    Next actions:

    • [x] remove the upb license note since we are no longer using the ubp impl.
    • [x] keep this ticket open until I have a chance to run the upcoming pbf through coverity to ensure it is clean.
    opened by springmeyer 6
  • Improve compatibility with various STL flavors

    Improve compatibility with various STL flavors

    In our STL std::string::iterator is not a class but rather a raw pointer. Suggested approach works as std::advance accepts Distance type as a template parameter.

    opened by georgthegreat 0
  • CMakeLists.txt: protobuf is only needed for tests

    CMakeLists.txt: protobuf is only needed for tests

    Don't check for protobuf if tests are disabled. As a side effect, this will avoid a build failure if clang-tidy and protobuf are found but tests are disabled

    Fix #109

    Signed-off-by: Fabrice Fontaine [email protected]

    opened by ffontaine 0
  • Configuration error when tests are disabled but clang_tidy is found

    Configuration error when tests are disabled but clang_tidy is found

    writer_tests used in add_dependencies does only exist if tests are enabled. Add an additional AND BUILD_TESTING:

    if(CLANG_TIDY AND PROTOBUF_FOUND AND BUILD_TESTING)
        message(STATUS "Looking for clang-tidy - found ${CLANG_TIDY}")
        add_custom_target(clang-tidy
            ${CLANG_TIDY}
            -p ${CMAKE_BINARY_DIR}
            ${CMAKE_SOURCE_DIR}/test/*.cpp
            ${CMAKE_SOURCE_DIR}/test/t/*/reader_test_cases.cpp
            ${CMAKE_SOURCE_DIR}/test/t/*/writer_test_cases.cpp
            ${CMAKE_SOURCE_DIR}/test/unit/*.cpp
            ${CMAKE_SOURCE_DIR}/tools/*.cpp
        )
        add_dependencies(clang-tidy writer_tests)
    else()
        message(STATUS "Looking for clang-tidy - not found")
        message(STATUS "  Build target 'clang-tidy' will not be available.")
    endif()
    
    
    opened by kelteseth 0
  • Multiple installed versions

    Multiple installed versions

    Hi, libosmium seems to required a newer libprotozero as there is in Debian/Stretch. So cloning protozero to the directory solved the issue. But only half way. If libprotozero-dev from Stretch is installed it gets priority over the locally cloned and the build fails (Or the required version).

    It seems the included cmake/FindProtozero.cmake is not capable of searching all possible locations to satisfy the required version. It prefers /usr/include/protozero over ../protozero/ although /usr/include does not satisfy the REQUIRED version.

    Flo

    opened by flohoff 3
  • Getopt library on Windows

    Getopt library on Windows

    Building/Testing tools/pbf-decoder is disabled on Windows because of the missing getopt library. We should figure something out here.

    See also https://github.com/mapbox/vtzero/issues/20.

    opened by joto 1
  • Optimize pbf_reader::get_packed_bool

    Optimize pbf_reader::get_packed_bool

    pbf_reader::get_packed_bool() currently calls get_packed_int32(). But with a one-byte bool there will never be any endianness or alignment issues so we could optimize this case.

    enhancement 
    opened by joto 2
Releases(v1.7.1)
  • v1.7.1(Jan 10, 2022)

    Changed

    • Don't build tests if the standard CMake BUILD_TESTING variable is set to off.
    • Now needs CMake 3.5.0 or greater.
    • Update included catch2 framework to current version v2.13.8.
    • Only enable clang-tidy make target if protobuf was found.
    • Allow setting C++ version to compile with in CMake config.

    Fixed

    • Fixes undefined behaviour in float and double byteswap.
    • Add missing includes of "config.hpp".
    • Avoid narrowing conversion by doing an explicit static_cast.
    Source code(tar.gz)
    Source code(zip)
  • v1.7.0(Jun 8, 2020)

    Added

    • Support for buffer types other that std::string. pbf_writer is now just a typedef for basic_pbf_writer<std::string>. Other buffer types can be used with basic_pbf_writer. See doc/advanced.md for details.

    Changed

    • Switched to catch2 for testing.
    • Some minor tweaks.

    Fixed

    • Removed some undefined behaviour.
    Source code(tar.gz)
    Source code(zip)
  • v1.6.8(Aug 15, 2019)

  • v1.6.7(Feb 21, 2019)

  • v1.6.6(Feb 20, 2019)

  • v1.6.5(Feb 5, 2019)

  • v1.6.4(Nov 8, 2018)

    Added

    • Add function data() to get the not yet read data from a pbf_reader.
    • New add_packed_fixed() template function for pbf_writer.
    • New length_of_varint() helper function calculates how long a varint would be for a specified value.

    Changed

    • More consistent implementation of operators as free friend functions.

    Fixed

    • Fixed some zigzag encoding tests on MSVC.
    • Add extra cast so we do an xor with unsigned ints.
    • No more bitwise operations on signed integers in varint decoder.
    • No more bitwise operations on signed integers in zigzag encoder/decoder.
    Source code(tar.gz)
    Source code(zip)
  • v1.6.3(Jul 17, 2018)

    Changed

    • Moved byteswap_inplace functions from detail into protozero namespace. They can be useful outsize protozero.
    • More asserts and unit tests and small cleanups.
    Source code(tar.gz)
    Source code(zip)
  • v1.6.2(Mar 9, 2018)

    Changed

    • Update included catch.hpp to v1.12.0.
    • Move basic unit tests into their own directory (test/unit).
    • Improved clang-tidy config and fixed some code producing warnings.

    Fixed

    • Buffer overflow in pbf-decoder tool.
    Source code(tar.gz)
    Source code(zip)
  • v1.6.1(Nov 16, 2017)

    Added

    • Document internal handling of varints.
    • Add aliases for fixed iterators, too.

    Changed

    • The const_fixed_iterator is now a random access iterator making code using it potentially more performant (for instance when using std::distance)
    • Overloads std::distance for the varint and svarint iterators. This is better than the workaround with the rage_size function used before.

    Fixed

    • Rename .proto files in some tests to be unique. This solves a problem when building with newer versions of the Google Protobuf library.
    • Floating point comparisons in tests are now always correctly done using Approx().
    Source code(tar.gz)
    Source code(zip)
  • v1.6.0(Oct 24, 2017)

    Added

    • Comparison functions (<, <=, >, >=) for data_view. Allows use in std::map for instance.
    • Tool pbf-decoder for decoding raw messages. This has limited use for normal users, but it can be used for fuzzing.

    Changed

    • Protozero now uses CMake to build the tests etc. This does not affect simple users of the library, but if you are using CMake yourself you might want to use the cmake/FindProtozero.cmake module provided. The README contains more information about build options.
    • Moved data_view class from types.hpp into its own header file data_view.hpp.
    • Implementation of the const_fixed_iterator to use only a single pointer instead of two.
    • Made operator== and operator!= on data_view constexpr.
    • The pbf_reader constructor taking a std::pair is deprecated. Use one of the other constructors instead.

    Fixed

    • Varints where the last byte was larger than what would fit in 64bit were triggering undefined behaviour. This can only happen when the message being decoded was corrupt in some way.
    • Do not assert when reading too long varints for bools any more. A valid encoder should never generate varints with more than one byte for bools, but if they are longer that's not really a problem, so just handle it.
    • Throw exception if the length of a packed repeated field of a fixed-length type is invalid. The length must always be a multiple of the size of the underlying type. This can only happen if the data is corrupted in some way, a valid encoder would never generate data like this.
    • Throw an exception when reading invalid tags. This can only happen if the data is corrupted in some way, a valid encoder would never generate invalid tags.
    Source code(tar.gz)
    Source code(zip)
  • v1.5.3(Sep 22, 2017)

    Added

    • More documentation.
    • New size() method on iterator range used for packed repeated fields to find out how many elements there are in the range. This is much faster compared to the std::difference() call you had to do before, because the varints don't have to be fully decoded. See Advanced Topics for details.

    Changed

    • Updated clang-tidy settings in Makefiles and fixed a lot of minor issues reported by clang-tidy.
    • Update included catch.hpp to version 1.10.0.
    • Miscellaneous code cleanups.
    • Support for invalid state in pbf_writer and packed_repeated_fields. This fixes move construction and move assignement in pbf_writer and disables the copy construction and copy assignement which don't have clear semantics. It introduces an invalid or empty state in the pbf_writer, pbf_builder, and packed_repeated_fields classes used for default-constructed, moved from, or committed objects. There is a new commit() function for pbf_writer and the packed_repeated_fields which basically does the same as the destructor but can be called explicitly.

    Fixed

    • The empty() method of the iterator range now returns a bool instead of a size_t.
    Source code(tar.gz)
    Source code(zip)
  • v1.5.2(Jun 30, 2017)

    Added

    • Add missing two-parameter version of pbf_message::next() function.
    • Add data_view::empty() function.
    • Add missing versions of add_bytes(), add_string(), and add_message() to pbf_builder.

    Changed

    • Clarify include file usage in tutorial.
    • Updated included Catch unit test framework to version 1.9.6 and updated tests to work with the current version.
    • Make some constructors explicit (best practice to avoid silent conversions).

    Fixed

    • Important bugfix in data_view equality operator. The equality operator is actually never used in the protozero code itself, but users of protozero might use it. This is a serious bug that could lead to buffer overrun type problems.
    Source code(tar.gz)
    Source code(zip)
  • v1.5.1(Jan 14, 2017)

  • v1.5.0(Jan 12, 2017)

    Added

    • Add add_bytes_vectored() methods to pbf_writer and pbf_builder. This allows single-copy scatter-gather type adding of data that has been prepared in pieces to a protobuf message.
    • New functions to check the tag and wire type at the same time: Two parameter version of pbf_reader::next() and pbf_reader::tag_and_type() can be used together with the free function tag_and_type() to easily and quickly check that not only the tag but also the wire type is correct for a field.

    Changed

    • packed_field_* classes now work with pbf_builder.
    • Reorganized documentation. Advanced docs are now under doc/advanced.md.

    Fixed

    • packed_field class is now non-copyable because data can get corrupted if you copy it around.
    • Comparison operators of data_view now have const& parameters.
    • Make zigzag encoding/decoding functions constexpr.
    Source code(tar.gz)
    Source code(zip)
  • v1.4.5(Nov 18, 2016)

  • v1.4.4(Nov 15, 2016)

  • v1.4.3(Nov 15, 2016)

  • v1.4.2(Aug 27, 2016)

  • v1.4.1(Aug 24, 2016)

    Fixed

    • GCC 4.8 compile fixed

    Added

    • New ability to dynamically require the module as a node module to ease building against from other node C++ modules.
    Source code(tar.gz)
    Source code(zip)
  • v1.4.0(Jul 22, 2016)

    Changed

    • Use more efficient new skip_varint() function when iterating over packed varints.
    • Split decode_varint() function into two functions speeding up the common case where a varint is only one byte long.
    • Introduce new class iterator_range used instead of std::pair of iterators. This way the objects can be used in range-based for loops. Read UPGRADING.md for details.
    • Introduce new class data_view and functions using and returning it. Read UPGRADING.md for details.
    Source code(tar.gz)
    Source code(zip)
Owner
Mapbox
Mapbox is the location data platform for mobile and web applications. We're changing the way people move around cities and explore our world.
Mapbox
Quite OK Image (QOI) format encoder/decoder

This project implements encoding and decoding the "Quite OK Image" (QOI) format in the Ć programming language. Ć can be automatically translated to pu

Piotr Fusik 38 Jun 29, 2022
Second life for famous JPEGView - fast and tiny viewer/editor for JPEG, BMP, PNG, WEBP, TGA, GIF and TIFF images with a minimalist GUI and base image processing.

JPEGView-Image-Viewer-and-Editor Updated Dec 07 2021. Version 1.1.1.0 has been released. Download link1, link2 added. Second life for famous JPEGView

Ann Hatt 22 Jul 3, 2022
An STL-compatible, legible ring buffer in C++11.

baudvine/ringbuf Overview This is a header-only ring/circular buffer implementation in C++11, with the following goals: Reasonably performant. Readabl

Dominic van Berkel 11 May 14, 2022
Buffer reader/builder for C

ubuf ubuf is a simple interface for reading/writing binary data. It handles automatically expanding the buffer and byte order for you, but that's abou

adrian 2 Jan 10, 2022
Example code for the research paper "Masked Software Occlusion Culling"; implements an efficient alternative to the hierarchical depth buffer algorithm.

MaskedOcclusionCulling This code accompanies the research paper "Masked Software Occlusion Culling", and implements an efficient alternative to the hi

null 528 Jun 10, 2022
Gfx - A minimalist and easy to use graphics API.

gfx gfx is a minimalist and easy to use graphics API built on top of Direct3D12/HLSL intended for rapid prototyping. It supports: Full shader reflecti

Guillaume Boissé 236 Jun 24, 2022
F3D - Fast and minimalist 3D viewer

F3D - Fast and minimalist 3D viewer By Michael Migliore and Mathieu Westphal. F3D (pronounced /fɛd/) is a VTK-based 3D viewer following the KISS princ

null 515 Jun 25, 2022
PHP Encoder, protect PHP scripts in PHP 8 and PHP 7, High Performance, Compitable with X86_64, MIPS, ARM platform and Ubuntu/Centos/OpenWRT system.

What's FRICC2? FRICC2 is a PHP Script encryption tool. When you are developing a commercial software using PHP, the script can be distributed as encry

Hoowa Sun 31 Jun 30, 2022
Turing-ring is a simple Turing Machine using just a Nano, a NeoPixel ring and a rotary encoder+push-button The ring is the tape and the UI.

Turing-ring Turing-ring is a simple Turing Machine using just a Nano, a NeoPixel ring and a rotary encoder+push-button The ring is the tape and the UI

Mark Wilson 2 Dec 26, 2021
isabel - Simple, minimalist note manager.

isabel isabel - Simple, minimalist note manager. Usage Type name and body of note and press Ctrl+s for save note. Press Tab to open notes list. Press

null 1 Oct 2, 2021
Legacy stepper motor analyzer - A DYI minimalist hardware stepper motor analyzer with graphical touch screen.

Simple Stepper Motor Analyzer NOTE: This is the legacy STM32 based design which was replaced by the single board, Raspberry Pi Pico design at https://

Zapta 158 Mar 21, 2022
A Sol-inspired minimalist Lua binding for Zig.

zoltan A Sol-inspired minimalist Lua binding for Zig. Features Supports Zig 0.9.0 Lua tables table creation from Zig get/set/create methods possible k

null 66 Jun 20, 2022
Browser and NodeJS Web Assembly audio decoder libraries that are highly optimized for size and performance.

WASM Audio Decoders WASM Audio Decoders is a collection of Web Assembly audio decoder libraries that are highly optimized for browser use. Each module

Ethan Halsall 55 Jul 5, 2022
Arduino code to interface with quadrature-encoder mice, specifically the Depraz mouse

Depraz Mice on USB via Arduino This code lets you connect a Depraz mouse to a modern computer via USB. The Depraz mouse has a male DE-9 connector but

John Floren 4 Jan 7, 2022
Cheap 3D Printed Absolute Encoder Knob

A cheap, 3D printed absolute position encoder knob based on a low-cost AS5600 breakout module. Demo firmware using a TTGO T-Display ESP32 board is pro

Scott Bezek 142 Jun 30, 2022
QuadratureDecoder - PIO based Encoder Library for the RP2040

QuadratureDecoder - PIO based Encoder Library for the RP2040 Overview The QuadratureDecoder C++ class can be used to count quadrature encoder signal t

Adam Green 10 Jun 2, 2022
Fast single source file BC7/BPTC texture encoder with perceptual metric support

Note: Since this repo was created, we've released two new codecs with better BC7 encoders: https://github.com/richgel999/bc7enc_rdo https://github.com

Rich Geldreich 142 Apr 11, 2022
SleighCraft is a decoder based on ghidra's decompiler implementation.

SleighCraft is a decoder (or, linear disassembler) based on ghidra's decompiler implementation. Sleighcraft can be used in Rust or Python, with both high-level and low-level API.

PortalLab 230 Jul 2, 2022
Multichannel HFDL decoder

dumphfdl dumphfdl is a multichannel HFDL (High Frequency Data Link) decoder. HFDL (High Frequency Data Link) is a protocol used for radio communicatio

Tomasz Lemiech 72 May 25, 2022