A C++ header-only ZLib wrapper

Related tags

Compression zstr
Overview

A C++ ZLib wrapper

http://travis-ci.org/mateidavid/zstr.svg?branch=master http://img.shields.io/:license-mit-blue.svg

This C++ header-only library enables the use of C++ standard iostreams to access ZLib-compressed streams.

For input access (decompression), the compression format is auto-detected, and multiple concatenated compressed streams are decompressed seamlessly.

For output access (compression), the only parameter exposed by this API is the compression level.

Alternatives to this library include:

  • The original ZLib, through its C API. This does not interact nicely with C++ iostreams.
  • The GZStream library. This library does not auto-detect input compression, and it cannot wrap streams (only files).
  • The Boost IOStreams library. The library does not auto-detect input compression (by default, though that can be easily implemented with filters), and more importantly, it is not a header-only Boost library.
  • The bxzstr library, if you want support for BZ2 and/or LZMA as well.

For an example usage, see examples/ztxtpipe.cpp and examples/zc.cpp.

It is compatible with miniz in case you don’t want to get frustrated with zlib e. g. on Windows.

Input Auto-detection

For input access, the library seamlessly auto-detects whether the source stream is compressed or not. The following compressed streams are detected:

  • GZip header, when stream starts with 1F 8B. See GZip format.
  • ZLib header, when stream starts with 78 01, 78 9C, and 78 DA. See answer here.

If none of these formats are detected, the library assumes the input is not compressed, and it produces a plain copy of the source stream.

Classes

The package provides 6 classes for accessing ZLib streams:

  • zstr::istreambuf is the core decompression class. This is constructed from an existing std::streambuf that contains source data. The zstr::istreambuf constructor accepts explicit settings for the internal buffer size (default: 1 MB) and the auto-detection option (default: on). ZLib errors cause exceptions to be thrown.
  • zstr::ostreambuf is the core compression class. This is constructed from an existing std::streambuf that contains sink data. The zstr::ostreambuf constructor accepts explicit settings for the internal buffer size (default: 1 MB) and the compression option (default: ZLib default). ZLib errors cause exceptions to be thrown.
  • zstr::istream is a wrapper for a zstr::istreambuf that accesses an external std::streambuf. It can be constructed from an existing std::istream (such as std::cin) or std::streambuf.
  • zstr::ostream is a wrapper for a zstr::ostreambuf that accesses an external std::streambuf. It can be constructed from an existing std::ostream (such as std::cout) or std::streambuf.
  • zstr::ifstream is a wrapper for a zstr::istreambuf that accesses an internal std::ifstream. This can be used to open a file and read decompressed data from it.
  • zstr::ofstream is a wrapper for a zstr::ostreambuf that accesses an internal std::ofstream. This can be used to open a file and write compressed data to it.

For all stream objects, the badbit of their expection mask is turned on in order to propagate exceptions.

License

Released under the MIT license.

Comments
  • make it possible to specify window bits, and possible to specify parameters from zstr::istream/ostream

    make it possible to specify window bits, and possible to specify parameters from zstr::istream/ostream

    fwiw, added it because it's necessary for decompressing the age of empires (2) game files

    the changes have been tested a lot in https://github.com/sandsmark/freeaoe/ and https://github.com/sandsmark/AGE via https://github.com/sandsmark/genieutils

    bug enhancement 
    opened by sandsmark 14
  • strerror problem on a Mac

    strerror problem on a Mac

    The various ifdefs at the beginning of strict_fstream.hpp that determine how to interact with strerror_r break for me on a Mac. Specifically, it drops into the third, GNU-specific case, but it should be using the second one. I quickly fixed my local copy by adding defined(__APPLE__) to the second case, but I'm not sure if it's the correct universal fix.

    bug 
    opened by mrzv 11
  • crashing in strict_fstream::detail::static_method_holder::check_open

    crashing in strict_fstream::detail::static_method_holder::check_open

    I am using msys2 gcc and I getting crashes from this and I use Dr. memory to get some info about that error and it showed me this before crashing

    Error #20: UNADDRESSABLE ACCESS beyond top of stack: reading 0x00000097d2df3700-0x00000097d2df3708 8 byte(s)
    # 0 .text                                                                  [../../../gcc-10.3.0/libgcc/config/i386/cygwin.S:132]
    # 1 d_demangle_callback.constprop.0                                        [C:\_\M\mingw-w64-gcc\src\build-x86_64-w64-mingw32\x86_64-w64-mingw32\libstdc++-v3\libsupc++/cp-demangle.c:6314]
    # 2 __cxa_demangle                                                         [C:\_\M\mingw-w64-gcc\src\build-x86_64-w64-mingw32\x86_64-w64-mingw32\libstdc++-v3\libsupc++/cp-demangle.c:6380]
    # 3 __gnu_cxx::__verbose_terminate_handler                                 [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/vterminate.cc:61]
    # 4 __cxxabiv1::__terminate                                                [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:48]
    # 5 std::terminate                                                         [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:58]
    # 6 __cxa_throw                                                            [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_throw.cc:95]
    # 7 strict_fstream::detail::static_method_holder::check_open               [C:/enough_words_c_tcod/include/zstr/strict_fstream.hpp:153]
    # 8 zstr::ofstream::ofstream                                               [C:/enough_words_c_tcod/include/zstr/strict_fstream.hpp:212]
    # 9 deload_tile                                                            [C:/enough_words_c_tcod/src/systems/saving/dynimic_tile_loader.hpp:111]
    #10 fill_roads                                                             [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:133]
    #11 fill_inered_roads                                                      [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:186]
    #12 puploate_huge_tiles                                                    [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:566]
    #13 Create_huge_map::init                                                  [C:/enough_words_c_tcod/src/huge_map.cpp:69]
    #14 Engine::handle_events                                                  [C:/enough_words_c_tcod/src/engine.cpp:14]
    #15 SDL_main                                                               [C:/enough_words_c_tcod/src/main.cpp:45]
    #16 main_getcmdline                                                        [../src/main/windows/SDL_windows_main.c:126]
    #17 __tmainCRTStartup                                                      [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:321]
    #18 .l_start                                                               [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:202]
    #19 KERNEL32.dll!BaseThreadInitThunk
    Note: @0:11:20.399 in thread 26304
    Note: 0x00000097d2df3700 refers to 200 byte(s) beyond the top of the stack 0x00000097d2df37c8
    Note: instruction: or     $0x0000000000000000 (%rcx) -> (%rcx)
    

    my code works correctly in the debug mode and without any optimization but when I try it with the O3 that happen I know that isn't that helpful and I willing to share any more info if that needed, also any idea of what the cause of it is helpful

    question 
    opened by hero2002 10
  • Changes to enable compilation under Visual Studio (2015)

    Changes to enable compilation under Visual Studio (2015)

    Hi,

    If you're interested, I am making available some changes I had to make to your very useful zstr library so that it would compile under Microsoft Visual Studio. Most notably, Microsoft's compiler doesn't recognize and/or/not keywords.

    Thanks, -Mark Wood

    opened by markdwood 10
  • strerror_r variant compatibility

    strerror_r variant compatibility

    This pull request fixes issues with the different variants of strerror_r.

    Originally encountered this in Alpine Linux, which uses musl as their libc implementation. I was unable to compile due to a non-compliant implementation of strerror_r.

    The fix wraps the call to strerror_r with an overloaded function that deals with the non-compliant return type. This eliminates the preprocessor macros for XSI/GNU detection and should fix the related pull requests too.

    Closes: #5 Related: #6, #7

    opened by graemenail 7
  • Incompatible with GCC 4.8.2

    Incompatible with GCC 4.8.2

    Hi,

    I recently started playing with zstr in one of my projects and I noticed the library does not compile with GCC 4.8.2. I know that GCC 4.8.2 is reaaaaally old now, I think almost 10 years old, but unfortunately my users also use very old compilers...

    Anyway, here is the issue (one issue at 2 places):

    zstr.hpp: In member function ‘void zstr::ifstream::open(std::string, std::ios_base::openmode)’:
    zstr.hpp:429:31: error: use of deleted function ‘std::basic_istream<char>& std::basic_istream<char>::operator=(const std::basic_istream<char>&)’
             std::istream::operator=(std::istream(new istreambuf(_fs.rdbuf())));
    
    zstr.hpp: In member function ‘void zstr::ofstream::open(std::string, std::ios_base::openmode, int)’:
    zstr.hpp:467:31: error: use of deleted function ‘std::basic_ostream<char>& std::basic_ostream<char>::operator=(const std::basic_ostream<char>&)’
             std::ostream::operator=(std::ostream(new ostreambuf(_fs.rdbuf(), default_buff_size, level)));
    

    If I compile with GCC 5.4.0 everything is fine. Would there be any way to work around that?

    Thank you, Guillaume

    enhancement 
    opened by GuillaumeHolley 5
  • the proper way of getting decompressed file size

    the proper way of getting decompressed file size

    A buffer is needed to hold the whole file after decompressing, how could I get the final file size? It seems "seekg" and "tellg" could only get the original compressed size.

    question 
    opened by passerbyd 5
  • what language are the released zip/tgz files in ?

    what language are the released zip/tgz files in ?

    Am I missing something? zstr.zip/tgz doesn't seem to be cpp since some if not most the logical operators have been translated to their lexical counterparts, thus rendering the lib useless for use in e.g. vcpkg: Example from near the beginning of strict_fstream.hpp ..... line 63 res += (not res.empty()? "|" : ""); line 72 if ((mode & std::ios_base::trunc) and not (mode & std::ios_base::out)) ... and lot's more

    question 
    opened by yamcenutzer 5
  • eliminate use of std::make_unique, remove zstr_make_unique_polyfill.h

    eliminate use of std::make_unique, remove zstr_make_unique_polyfill.h

    While trying to use zstr in another project, there was some concern about the license of zstr_make_unique_polyfill.h: https://github.com/ERGO-Code/HiGHS/issues/786#issuecomment-1085062067

    This PR replaces the use of std::make_unique<>() in zstr by equivalent code that preserves C++11 compatibility. As a consequence, zstr_make_unique_polyfill.h can be removed.

    enhancement 
    opened by svigerske 3
  • can't compile in clang

    can't compile in clang

    so I am trying to build for android and android ndk use clang and I am getting this error

    C:\enough_words_c_tcod\include\zstr/strict_fstream.hpp:77:12: error: cannot initialize a variable of type 'char *' with
          an rvalue of type 'int'
        char * p = strerror_r(errno, &buff[0], buff.size());
               ^   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    help wanted 
    opened by hero2002 3
  • Flush close on destroy, CMake and buff_size

    Flush close on destroy, CMake and buff_size

    Added CMakeLists.txt and included buff_size in the constructor of the i/ofstream classes Including a close on the underlying I/ofstream on destruction

    opened by robegan21 3
  • call to implicitly-deleted copy constructor of 'zstr::ifstream'

    call to implicitly-deleted copy constructor of 'zstr::ifstream'

    I'm trying to use zstr::ifstream with a CSV parser but I'm getting:

    //from the IDE (clion)
    In template: call to implicitly-deleted copy constructor of 'zstr::ifstream' error occurred here 
    in instantiation of member function 'csv::internals::StreamParser<zstr::ifstream>::StreamParser' requested here 
    in instantiation of function template specialization 'csv::CSVReader::CSVReader<zstr::ifstream, 0>' requested here 
    copy constructor of 'ifstream' is implicitly deleted because base 
    class 'detail::strict_fstream_holder<strict_fstream::ifstream>' has a deleted copy constructor 
    copy constructor of 'strict_fstream_holder<strict_fstream::ifstream>' is implicitly deleted because 
    field '_fs' has a deleted copy constructor copy constructor of 'ifstream' is implicitly deleted because 
    base class 'std::ifstream' (aka 'basic_ifstream<char>') has a deleted copy constructor 'basic_ifstream' has 
    been explicitly marked deleted here
    
    //and from the compiler
    lib/zstr/src/zstr.hpp:411:7: error: use of deleted function ‘zstr::detail::strict_fstream_holder<strict_fstream::ifstream>::strict_fstream_holder(const zstr::detail::strict_fstream_holder<strict_fstream::ifstream>&)’
    lib/zstr/src/zstr.hpp:400:8: note: ‘zstr::detail::strict_fstream_holder<strict_fstream::ifstream>::strict_fstream_holder(const zstr::detail::strict_fstream_holder<strict_fstream::ifstream>&)’ is implicitly deleted because the default definition would be ill-formed:
      400 | struct strict_fstream_holder
          |        ^~~~~~~~~~~~~~~~~~~~~
    lib/zstr/src/zstr.hpp:400:8: error: use of deleted function ‘strict_fstream::ifstream::ifstream(const strict_fstream::ifstream&)’
    lib/zstr/src/strict_fstream.hpp:177:7: error: use of deleted function ‘std::basic_ifstream<_CharT, _Traits>::basic_ifstream(const std::basic_ifstream<_CharT, _Traits>&) [with _CharT = char; _Traits = std::char_traits<char>]’
    lib/zstr/src/strict_fstream.hpp:177:7: error: use of deleted function ‘std::basic_ios<_CharT, _Traits>::basic_ios(const std::basic_ios<_CharT, _Traits>&) [with _CharT = char; _Traits = std::char_traits<char>]’
      177 | class ifstream
          |       ^~~~~~~~
    
    //this works
    std::ifstream file(fp, std::ios::binary);
    csv::CSVReader reader(file, csv_format);
    
    //doesn't compile - getting the error above
    zstr::ifstream file(fp, std::ios::binary);
    csv::CSVReader reader(file, csv_format);
    

    Any help appreciated.

    enhancement help wanted question 
    opened by jmakov 3
  • Error with defined(__APPLE__) && !_GNU_SOURCE

    Error with defined(__APPLE__) && !_GNU_SOURCE

    Hi,

    A friend has been trying to compile a project of mine including zstr on his Macbook (not the ARM ones but an older x86 version). Apparently, the compiler issues an error for the following line in strict_fstream.hpp

    #elif ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 || defined(__APPLE__)) && ! _GNU_SOURCE) || defined(__MUSL__)
    

    The error is error: expected value in expression and points to _GNU_SOURCE. My friend solved the issue (temporarily) by removing the ! _GNU_SOURCE. My short understanding of macros and preprocessor is that an undefined macro value initializes automatically to 0. So not sure what is the problem. Unfortunately, I do not have access to a Macbook so I cannot really test for it.

    Thanks, Guillaume

    bug 
    opened by GuillaumeHolley 5
  • Request: constants instead of magic numbers for windowBits parameter

    Request: constants instead of magic numbers for windowBits parameter

    zstr.hpp:99 and zstr.hpp:103 use 15+32 and 12+16 as default window size. The semantics of this behavior are documented in https://zlib.net/manual.html, but I find this interface not very intuitive. It's not clear that gzip is used by default, which lead to subtle bugs in some project code.

    I would appreciate if there were constants or separate parameters to make the used window size and header explicit. It would be more clear that the default value 15 + 16 refers to a 64KiB window with gzip header. Also, changing the compression header would not involve setting arbitrary numbers to the windowBits parameter.

    I would propose adding an enum class type for the compression header:

    enum class header {auto_detect, zlib, gzip, raw_deflate};
    

    The streambuf classes get additional constructors which take a compression header argument and calculate the windowBits value based on the compression header and the desired window size (+32, +16 or negate window size).

    enhancement 
    opened by stertingen 1
  • zlib header detection is too restrictive.

    zlib header detection is too restrictive.

    This statement looks only for three specific possible zlib headers. There are many more valid zlib headers that would be missed. 61 to be precise. Look at RFC 1950 for the definition of a zlib header.

    bug help wanted 
    opened by madler 3
  • gcount, tellg, seekg functions

    gcount, tellg, seekg functions

    tellg(), gcount() and seekg() functions might be helpful for estimating file size.

    See https://stackoverflow.com/questions/2409504/using-c-filestreams-fstream-how-can-you-determine-the-size-of-a-file

    enhancement help wanted 
    opened by ferdymercury 3
Releases(v1.0.6)
Owner
Matei David
Matei David
Fork of the popular zip manipulation library found in the zlib distribution.

minizip-ng 3.0.0 minizip-ng is a zip manipulation library written in C that is supported on Windows, macOS, and Linux. Developed and maintained by Nat

zlib-ng 961 Nov 16, 2022
zlib replacement with optimizations for "next generation" systems.

zlib-ng zlib data compression library for the next generation systems Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org) C

zlib-ng 12 Nov 19, 2022
Fork of the popular zip manipulation library found in the zlib distribution.

minizip-ng 3.0.1 minizip-ng is a zip manipulation library written in C that is supported on Windows, macOS, and Linux. Developed and maintained by Nat

zlib-ng 962 Nov 25, 2022
zlib replacement with optimizations for "next generation" systems.

zlib-ng zlib data compression library for the next generation systems Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org) C

zlib-ng 1.1k Nov 16, 2022
PNGFuse is a cross-platform application that allows you to embed and extract full zlib-compressed files within PNG metadata.

PNGFuse PNGFuse is a portable, lightweight, and cross-platform application written in C++ that allows you to embed and extract full zlib-compressed fi

Eta 3 Dec 29, 2021
Gzip header-only C++ library

Gzip C++ lib for gzip compression and decompression. Extracted from mapnik-vector-tile for light-weight modularity. Usage // Include the specific gzip

Mapbox 241 Nov 13, 2022
PhysFS++ is a C++ wrapper for the PhysicsFS library.

PhysFS++ PhysFS++ is a C++ wrapper for the excellent PhysicsFS library by Ryan C. Gordon and others. It is licensed under the zlib license - same as P

Kevin Howell 80 Oct 25, 2022
Single header lib for JPEG encoding. Public domain. C99. stb style.

tiny_jpeg.h A header-only public domain implementation of Baseline JPEG compression. Features: stb-style header only library. Does not do dynamic allo

Sergio Gonzalez 208 Nov 18, 2022
miniz: Single C source file zlib-replacement library, originally from code.google.com/p/miniz

Miniz Miniz is a lossless, high performance data compression library in a single source file that implements the zlib (RFC 1950) and Deflate (RFC 1951

Rich Geldreich 1.6k Nov 23, 2022
Fork of the popular zip manipulation library found in the zlib distribution.

minizip-ng 3.0.0 minizip-ng is a zip manipulation library written in C that is supported on Windows, macOS, and Linux. Developed and maintained by Nat

zlib-ng 961 Nov 16, 2022
zlib replacement with optimizations for "next generation" systems.

zlib-ng zlib data compression library for the next generation systems Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org) C

zlib-ng 12 Nov 19, 2022
Heavily optimized zlib compression algorithm

Optimized version of longest_match for zlib Summary Fast zlib longest_match function. Produces slightly smaller compressed files for significantly fas

Konstantin Nosov 123 Oct 29, 2022
Przemyslaw Skibinski 575 Nov 22, 2022
zlib replacement with optimizations for "next generation" systems.

zlib-ng zlib data compression library for the next generation systems Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org) C

zlib-ng 11 Oct 9, 2022
Fork of the popular zip manipulation library found in the zlib distribution.

minizip-ng 3.0.1 minizip-ng is a zip manipulation library written in C that is supported on Windows, macOS, and Linux. Developed and maintained by Nat

zlib-ng 962 Nov 25, 2022
zlib replacement with optimizations for "next generation" systems.

zlib-ng zlib data compression library for the next generation systems Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org) C

zlib-ng 1.1k Nov 16, 2022
PNGFuse is a cross-platform application that allows you to embed and extract full zlib-compressed files within PNG metadata.

PNGFuse PNGFuse is a portable, lightweight, and cross-platform application written in C++ that allows you to embed and extract full zlib-compressed fi

Eta 3 Dec 29, 2021
Header-only, event based, tiny and easy to use libuv wrapper in modern C++ - now available as also shared/static library!

Do you have a question that doesn't require you to open an issue? Join the gitter channel. If you use uvw and you want to say thanks or support the pr

Michele Caini 1.5k Nov 16, 2022
Header-only, event based, tiny and easy to use libuv wrapper in modern C++ - now available as also shared/static library!

Do you have a question that doesn't require you to open an issue? Join the gitter channel. If you use uvw and you want to say thanks or support the pr

Michele Caini 1.5k Nov 28, 2022
Header-only C++20 wrapper for MPI 4.0.

MPI Modern C++20 message passing interface wrapper. Examples Initialization: mpi::environment environment; const auto& communicator = mpi::world_c

Ali Can Demiralp 29 Apr 8, 2022