A C++ header-only ZLib wrapper

fwiw, added it because it's necessary for decompressing the age of empires (2) game files

the changes have been tested a lot in https://github.com/sandsmark/freeaoe/ and https://github.com/sandsmark/AGE via https://github.com/sandsmark/genieutils

The various ifdefs at the beginning of strict_fstream.hpp that determine how to interact with strerror_r break for me on a Mac. Specifically, it drops into the third, GNU-specific case, but it should be using the second one. I quickly fixed my local copy by adding defined(__APPLE__) to the second case, but I'm not sure if it's the correct universal fix.

I am using msys2 gcc and I getting crashes from this and I use Dr. memory to get some info about that error and it showed me this before crashing

Error #20: UNADDRESSABLE ACCESS beyond top of stack: reading 0x00000097d2df3700-0x00000097d2df3708 8 byte(s)
# 0 .text                                                                  [../../../gcc-10.3.0/libgcc/config/i386/cygwin.S:132]
# 1 d_demangle_callback.constprop.0                                        [C:\_\M\mingw-w64-gcc\src\build-x86_64-w64-mingw32\x86_64-w64-mingw32\libstdc++-v3\libsupc++/cp-demangle.c:6314]
# 2 __cxa_demangle                                                         [C:\_\M\mingw-w64-gcc\src\build-x86_64-w64-mingw32\x86_64-w64-mingw32\libstdc++-v3\libsupc++/cp-demangle.c:6380]
# 3 __gnu_cxx::__verbose_terminate_handler                                 [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/vterminate.cc:61]
# 4 __cxxabiv1::__terminate                                                [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:48]
# 5 std::terminate                                                         [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:58]
# 6 __cxa_throw                                                            [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_throw.cc:95]
# 7 strict_fstream::detail::static_method_holder::check_open               [C:/enough_words_c_tcod/include/zstr/strict_fstream.hpp:153]
# 8 zstr::ofstream::ofstream                                               [C:/enough_words_c_tcod/include/zstr/strict_fstream.hpp:212]
# 9 deload_tile                                                            [C:/enough_words_c_tcod/src/systems/saving/dynimic_tile_loader.hpp:111]
#10 fill_roads                                                             [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:133]
#11 fill_inered_roads                                                      [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:186]
#12 puploate_huge_tiles                                                    [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:566]
#13 Create_huge_map::init                                                  [C:/enough_words_c_tcod/src/huge_map.cpp:69]
#14 Engine::handle_events                                                  [C:/enough_words_c_tcod/src/engine.cpp:14]
#15 SDL_main                                                               [C:/enough_words_c_tcod/src/main.cpp:45]
#16 main_getcmdline                                                        [../src/main/windows/SDL_windows_main.c:126]
#17 __tmainCRTStartup                                                      [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:321]
#18 .l_start                                                               [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:202]
#19 KERNEL32.dll!BaseThreadInitThunk
Note: @0:11:20.399 in thread 26304
Note: 0x00000097d2df3700 refers to 200 byte(s) beyond the top of the stack 0x00000097d2df37c8
Note: instruction: or     $0x0000000000000000 (%rcx) -> (%rcx)

my code works correctly in the debug mode and without any optimization but when I try it with the O3 that happen I know that isn't that helpful and I willing to share any more info if that needed, also any idea of what the cause of it is helpful

Hi,

If you're interested, I am making available some changes I had to make to your very useful zstr library so that it would compile under Microsoft Visual Studio. Most notably, Microsoft's compiler doesn't recognize and/or/not keywords.

Thanks, -Mark Wood

This pull request fixes issues with the different variants of strerror_r.

Originally encountered this in Alpine Linux, which uses musl as their libc implementation. I was unable to compile due to a non-compliant implementation of strerror_r.

The fix wraps the call to strerror_r with an overloaded function that deals with the non-compliant return type. This eliminates the preprocessor macros for XSI/GNU detection and should fix the related pull requests too.

Closes: #5 Related: #6, #7

The LBANN project that uses zstr fails to compile:

In file included from /usr/local/include/zstr.hpp:15,
                 from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/include/lbann/data_readers/sample_list_impl.hpp:57,
                 from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/include/lbann/data_readers/data_reader_sample_list_impl.hpp:32,
                 from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/src/data_readers/data_reader_smiles.cpp:29:
/usr/local/include/strict_fstream.hpp: In function 'std::string strict_fstream::strerror()':
/usr/local/include/strict_fstream.hpp:77:26: error: invalid conversion from 'int' to 'char*' [-fpermissive]
   77 |     char * p = strerror_r(errno, &buff[0], buff.size());
      |                ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                          |
      |                          int

gcc-11 FreeBSD 13.1

Hi,

A friend has been trying to compile a project of mine including zstr on his Macbook (not the ARM ones but an older x86 version). Apparently, the compiler issues an error for the following line in strict_fstream.hpp

#elif ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 || defined(__APPLE__)) && ! _GNU_SOURCE) || defined(__MUSL__)

The error is error: expected value in expression and points to _GNU_SOURCE. My friend solved the issue (temporarily) by removing the ! _GNU_SOURCE. My short understanding of macros and preprocessor is that an undefined macro value initializes automatically to 0. So not sure what is the problem. Unfortunately, I do not have access to a Macbook so I cannot really test for it.

Thanks, Guillaume

Hi,

I recently started playing with zstr in one of my projects and I noticed the library does not compile with GCC 4.8.2. I know that GCC 4.8.2 is reaaaaally old now, I think almost 10 years old, but unfortunately my users also use very old compilers...

Anyway, here is the issue (one issue at 2 places):

zstr.hpp: In member function ‘void zstr::ifstream::open(std::string, std::ios_base::openmode)’:
zstr.hpp:429:31: error: use of deleted function ‘std::basic_istream<char>& std::basic_istream<char>::operator=(const std::basic_istream<char>&)’
         std::istream::operator=(std::istream(new istreambuf(_fs.rdbuf())));

zstr.hpp: In member function ‘void zstr::ofstream::open(std::string, std::ios_base::openmode, int)’:
zstr.hpp:467:31: error: use of deleted function ‘std::basic_ostream<char>& std::basic_ostream<char>::operator=(const std::basic_ostream<char>&)’
         std::ostream::operator=(std::ostream(new ostreambuf(_fs.rdbuf(), default_buff_size, level)));

If I compile with GCC 5.4.0 everything is fine. Would there be any way to work around that?

Thank you, Guillaume

A buffer is needed to hold the whole file after decompressing, how could I get the final file size? It seems "seekg" and "tellg" could only get the original compressed size.

Am I missing something? zstr.zip/tgz doesn't seem to be cpp since some if not most the logical operators have been translated to their lexical counterparts, thus rendering the lib useless for use in e.g. vcpkg: Example from near the beginning of strict_fstream.hpp ..... line 63 res += (not res.empty()? "|" : ""); line 72 if ((mode & std::ios_base::trunc) and not (mode & std::ios_base::out)) ... and lot's more

I'm trying to use zstr::ifstream with a CSV parser but I'm getting:

//from the IDE (clion)
In template: call to implicitly-deleted copy constructor of 'zstr::ifstream' error occurred here 
in instantiation of member function 'csv::internals::StreamParser<zstr::ifstream>::StreamParser' requested here 
in instantiation of function template specialization 'csv::CSVReader::CSVReader<zstr::ifstream, 0>' requested here 
copy constructor of 'ifstream' is implicitly deleted because base 
class 'detail::strict_fstream_holder<strict_fstream::ifstream>' has a deleted copy constructor 
copy constructor of 'strict_fstream_holder<strict_fstream::ifstream>' is implicitly deleted because 
field '_fs' has a deleted copy constructor copy constructor of 'ifstream' is implicitly deleted because 
base class 'std::ifstream' (aka 'basic_ifstream<char>') has a deleted copy constructor 'basic_ifstream' has 
been explicitly marked deleted here

//and from the compiler
lib/zstr/src/zstr.hpp:411:7: error: use of deleted function ‘zstr::detail::strict_fstream_holder<strict_fstream::ifstream>::strict_fstream_holder(const zstr::detail::strict_fstream_holder<strict_fstream::ifstream>&)’
lib/zstr/src/zstr.hpp:400:8: note: ‘zstr::detail::strict_fstream_holder<strict_fstream::ifstream>::strict_fstream_holder(const zstr::detail::strict_fstream_holder<strict_fstream::ifstream>&)’ is implicitly deleted because the default definition would be ill-formed:
  400 | struct strict_fstream_holder
      |        ^~~~~~~~~~~~~~~~~~~~~
lib/zstr/src/zstr.hpp:400:8: error: use of deleted function ‘strict_fstream::ifstream::ifstream(const strict_fstream::ifstream&)’
lib/zstr/src/strict_fstream.hpp:177:7: error: use of deleted function ‘std::basic_ifstream<_CharT, _Traits>::basic_ifstream(const std::basic_ifstream<_CharT, _Traits>&) [with _CharT = char; _Traits = std::char_traits<char>]’
lib/zstr/src/strict_fstream.hpp:177:7: error: use of deleted function ‘std::basic_ios<_CharT, _Traits>::basic_ios(const std::basic_ios<_CharT, _Traits>&) [with _CharT = char; _Traits = std::char_traits<char>]’
  177 | class ifstream
      |       ^~~~~~~~

//this works
std::ifstream file(fp, std::ios::binary);
csv::CSVReader reader(file, csv_format);

//doesn't compile - getting the error above
zstr::ifstream file(fp, std::ios::binary);
csv::CSVReader reader(file, csv_format);

Any help appreciated.

zstr.hpp:99 and zstr.hpp:103 use 15+32 and 12+16 as default window size. The semantics of this behavior are documented in https://zlib.net/manual.html, but I find this interface not very intuitive. It's not clear that gzip is used by default, which lead to subtle bugs in some project code.

I would appreciate if there were constants or separate parameters to make the used window size and header explicit. It would be more clear that the default value 15 + 16 refers to a 64KiB window with gzip header. Also, changing the compression header would not involve setting arbitrary numbers to the windowBits parameter.

I would propose adding an enum class type for the compression header:

enum class header {auto_detect, zlib, gzip, raw_deflate};

The streambuf classes get additional constructors which take a compression header argument and calculate the windowBits value based on the compression header and the desired window size (+32, +16 or negate window size).

This statement looks only for three specific possible zlib headers. There are many more valid zlib headers that would be missed. 61 to be precise. Look at RFC 1950 for the definition of a zlib header.

tellg(), gcount() and seekg() functions might be helpful for estimating file size.

See https://stackoverflow.com/questions/2409504/using-c-filestreams-fstream-how-can-you-determine-the-size-of-a-file

It appears that zstr does not open input streams in binary mode (std::ios_base::binary):

explicit ifstream(const std::string& filename, std::ios_base::openmode mode = std::ios_base::in)
    : detail::strict_fstream_holder< strict_fstream::ifstream >(filename, mode),
      std::istream(new istreambuf(_fs.rdbuf()))

Although it does for the output stream:

explicit ofstream(const std::string& filename, std::ios_base::openmode mode = std::ios_base::out)
    : detail::strict_fstream_holder< strict_fstream::ofstream >(filename, mode | std::ios_base::binary),
      std::ostream(new ostreambuf(_fs.rdbuf()))

This might be a bug? When running on Linux, I haven't found any issues, but when compiling and running on Windows, non-binary mode appears to corrupt the bytes read in, presumably because in text mode it attempts to do some Windows-specific end-of-line character conversions. Adding std::ios_base::binary to the ifstream mode fixes this.