miniz: Single C source file zlib-replacement library, originally from code.google.com/p/miniz

Related tags

Compression miniz
Overview

Miniz

Miniz is a lossless, high performance data compression library in a single source file that implements the zlib (RFC 1950) and Deflate (RFC 1951) compressed data format specification standards. It supports the most commonly used functions exported by the zlib library, but is a completely independent implementation so zlib's licensing requirements do not apply. Miniz also contains simple to use functions for writing .PNG format image files and reading/writing/appending .ZIP format archives. Miniz's compression speed has been tuned to be comparable to zlib's, and it also has a specialized real-time compressor function designed to compare well against fastlz/minilzo.

Usage

Please use the files from the releases page in your projects. Do not use the git checkout directly! The different source and header files are amalgamated into one miniz.c/miniz.h pair in a build step (amalgamate.sh). Include miniz.c and miniz.h in your project to use Miniz.

Features

  • MIT licensed
  • A portable, single source and header file library written in plain C. Tested with GCC, clang and Visual Studio.
  • Easily tuned and trimmed down by defines
  • A drop-in replacement for zlib's most used API's (tested in several open source projects that use zlib, such as libpng and libzip).
  • Fills a single threaded performance vs. compression ratio gap between several popular real-time compressors and zlib. For example, at level 1, miniz.c compresses around 5-9% better than minilzo, but is approx. 35% slower. At levels 2-9, miniz.c is designed to compare favorably against zlib's ratio and speed. See the miniz performance comparison page for example timings.
  • Not a block based compressor: miniz.c fully supports stream based processing using a coroutine-style implementation. The zlib-style API functions can be called a single byte at a time if that's all you've got.
  • Easy to use. The low-level compressor (tdefl) and decompressor (tinfl) have simple state structs which can be saved/restored as needed with simple memcpy's. The low-level codec API's don't use the heap in any way.
  • Entire inflater (including optional zlib header parsing and Adler-32 checking) is implemented in a single function as a coroutine, which is separately available in a small (~550 line) source file: miniz_tinfl.c
  • A fairly complete (but totally optional) set of .ZIP archive manipulation and extraction API's. The archive functionality is intended to solve common problems encountered in embedded, mobile, or game development situations. (The archive API's are purposely just powerful enough to write an entire archiver given a bit of additional higher-level logic.)

Known Problems

  • No support for encrypted archives. Not sure how useful this stuff is in practice.
  • Minimal documentation. The assumption is that the user is already familiar with the basic zlib API. I need to write an API wiki - for now I've tried to place key comments before each enum/API, and I've included 6 examples that demonstrate how to use the module's major features.

Special Thanks

Thanks to Alex Evans for the PNG writer function. Also, thanks to Paul Holden and Thorsten Scheuermann for feedback and testing, Matt Pritchard for all his encouragement, and Sean Barrett's various public domain libraries for inspiration (and encouraging me to write miniz.c in C, which was much more enjoyable and less painful than I thought it would be considering I've been programming in C++ for so long).

Thanks to Bruce Dawson for reporting a problem with the level_and_flags archive API parameter (which is fixed in v1.12) and general feedback, and Janez Zemva for indirectly encouraging me into writing more examples.

Patents

I was recently asked if miniz avoids patent issues. miniz purposely uses the same core algorithms as the ones used by zlib. The compressor uses vanilla hash chaining as described here. Also see the gzip FAQ. In my opinion, if miniz falls prey to a patent attack then zlib/gzip are likely to be at serious risk too.

Issues
  • Decompress consumes an extra byte when extra data are provided

    Decompress consumes an extra byte when extra data are provided

    What steps will reproduce the problem?
    
    1. Provide extra data in the input buffer for tinfl_decompress().
    2. An extra byte is consumed beyond the deflated data.
    3. The output parameter pIn_buf_size of tinfl_decompress() has an extra byte.
    
    This is a problem for gzip stream data since it has 8 extra bytes after the 
    deflated data (4 bytes CRC and 4 bytes original length).  When reading gzip 
    stream data, it's often not known when the end of stream is.  One has to read 
    as much as data there are and pass them to tinfl_decompress() to let it decode 
    the deflated data.  Once tinfl_decompress() stops, the remaining data in the 
    buffer are interpreted separately by the caller.
    
    What is the expected output? What do you see instead?
    
    Expect decompress() to consume exactly the number of deflated data, or if an 
    extra byte is consumed, let the caller knows so that it can unread the byte in 
    the buffer.
    
    What version of the product are you using? On what operating system?
    version 1.15.  On Win32.
    
    Please provide any additional information below.
    
    Sample gzip deflated data causing the problem.
    
    [0x73, 0x74, 0x72, 0x76, 0x71, 0x75, 0x73, 0xF7, 0xE0, 0xE5, 0x02, 0x00,
      0x94, 0xA6, 0xD7, 0xD0,  0x0A, 0x00, 0x00, 0x00]
    
    The first 12 bytes are the deflated data.  The next 4 bytes are the CRC.  The 
    last 4 bytes are the original length.  tinfl_decompress() should stop after 12 
    bytes.
    
    The original data is "ABCDEFGH\r\n"
    
    --------
    
    I found the problem where it's too late to call TINFL_HUFF_BITBUF_FILL().  It 
    is only called when only 1 byte are left in the input buffer, which is never 
    triggered for the above case where there are 8 extra bytes.
    
    I have changed the number of bytes to check from ((pIn_buf_end - pIn_buf_cur) < 
    2) to ((pIn_buf_end - pIn_buf_cur) < 16) and that works.  See attached file for 
    the diffs.
    
    It's not an airtight fix since it only helps the cases with input buffer upto 
    15 extra bytes.  It makes miniz work on gzip files.
    
    The best fix would be to consume the extra byte as needed but fix up the 
    pointer by counting how many bits left in bit_buf.  Since the data are in a 
    buffer, it's easy to move the pointer back as needed.
    
    

    Original issue reported on code.google.com by [email protected] on 18 Oct 2013 at 10:01

    Attachments:

    Priority-Medium Type-Defect auto-migrated 
    opened by GoogleCodeExporter 18
  • Correct license?

    Correct license?

    The original Google Code project this Git repository is based on, listed miniz as being under Unlicense.

    Completely free: Public domain in jurisdictions that recognize copyright laws, with a license patterned after the public domain SQLite project, see unlicense.org.

    But with commit 224d207ce8fffb908e156d27478be3afb5d83e6a @uroni marked the project as MIT licensed with the vague commit comment.

    MiniZ ZIP64 was part of valve vogl which is MIT licensed

    Is there a source to that Valve vogl claim?

    opened by eXpl0it3r 12
  • Reduce memory usage for inflate

    Reduce memory usage for inflate

    The Huffman tables are dimensioned after the biggest one (288) but that wastes a lot of memory for the other two which are considerably smaller (32 and 19). This PR creates separate arrays with proper lengths.

    opened by christiansandberg 11
  • valgrind Conditional jump or move depends on uninitialised value(s)

    valgrind Conditional jump or move depends on uninitialised value(s)

    valgrind error : ==23553== Conditional jump or move depends on uninitialised value(s) ==23553== at 0x4032AC: tdefl_find_match (in miniz/bin_linux/example1) ==23553== by 0x407B33: tdefl_compress_normal (in miniz/bin_linux/example1) ==23553== by 0x408DCB: mz_deflate (in miniz/bin_linux/example1) ==23553== by 0x4024CC: main (in miniz/bin_linux/example1) ==23553==

    opened by yag00 7
  • Has anyone got it working on android devices (arm based platforms)

    Has anyone got it working on android devices (arm based platforms)

    Just curious has anyone got miniz working on arm-based Android devices. 
    
    I had a stripped down version of miniz (v115_r4), which contains only 
    mz_compress() and mz_uncompress() functions and their dependenices. It is 
    working well on a Ubuntu desktop, however, after I ported it to Android, I 
    found that if the compress block size is 16K bytes or bigger, the compression 
    can easily fail. Most time it failed silently, I only noticed the problem when 
    trying to uncompress the compressed the content. 
    
    Confirmed the issue is on the compression side by trying to uncompress the same 
    compressed block using miniz on Ubuntu PC, it failed with the same error: 
    MZ_DATA_ERROR. Also tried using miniz on Ubuntu PC to compress and uncompress 
    the same original data block (16Kbytes), and it works fine.
    
    I will for sure debug further and report back to this thread, just in case 
    someone already done the simliar porting work and knows what need to be tweaked 
    ...
    
    Btw, Rich, very nice tool, thanks a lot for sharing. 
    
    

    Original issue reported on code.google.com by [email protected] on 24 Oct 2014 at 3:36

    Priority-Medium Type-Defect auto-migrated 
    opened by GoogleCodeExporter 7
  • warnings and static analysis

    warnings and static analysis

    I am using miniz vendored in a project. I am using strict warning settings in clang and gcc. I am getting warnings due to miniz "conversion", "sign-conversion", "cast-qual", "cast-align", "unused-parameter", "switch-enum", "c++-compat". It would be nice if some of those could be fixed in miniz. Would you accept patches?

    Furthermore by using clang static analyis via the "scan-build" tool a few more issues are uncovered. Unused assignments etc.

    opened by minad 6
  • Fixes for you, in a different repo

    Fixes for you, in a different repo

    Hi,
    
    I've been doing some fixes and pushed them up to this git repository:
    https://github.com/paulharris/miniz
    
    I imported the SVN repo, so all the SVN patches are contained within too.
    
    Are you still active in this project?  There are quite a few open bugs that 
    seem to be already fixed, or easily fixed elsewhere.
    
    Also note that I'm not the only alternative miniz repo out there,
    eg, this one has changes for working on Android:
    https://github.com/codefireXperiment/external_libminiz
    
    cheers,
    Paul
    
    

    Original issue reported on code.google.com by [email protected] on 8 Oct 2013 at 3:23

    Priority-Medium Type-Defect auto-migrated 
    opened by GoogleCodeExporter 6
  • Using miniz.c in multiple projects can cause symbol clashes

    Using miniz.c in multiple projects can cause symbol clashes

    miniz is designed as a single file to be included by another source file via 
    #include "miniz.c" as stated in the example applications. This works fine when 
    miniz is used within one project. When two projects using miniz unexpected 
    results arise.
    
    Situation:
    
    Shared library A includes miniz.c. Application B also includes miniz.c. 
    Application A will load B.
    
    
    Problem:
    
    When B loads library A there are two copies of the symbols related to miniz. 
    Depending on the OS, B might fail to load A. Also, depending on the OS the 
    miniz that is part of A might be used by B or vice versa. This can be a major 
    issue if A and B have different versions (especially if they are not ABI or API 
    compatible) of miniz included.
    
    
    Solutions:
    
    1) Make all parts within miniz.c static. This will make everything in miniz.c 
    only visible to what is including it. This will prevent the miniz symbols from 
    being exposed publicly.
    
    This could be facilitated by setting a define similar to MINIZ_NO_ARCHIVE_APIS. 
    Something like:
    
    #ifdef MINIZ_STATIC
    # define MINIZ_STATIC static
    #else
    # define MINIZ_STATIC
    #endif
    
    Then any any function that would be considered public would be prefixed by 
    MINIZ_STATIC:
    
    MINIZ_STATIC int mz_uncompress(...
    
    2) Break miniz.c appart into miniz.h and miniz.c to facilitate building and 
    installing miniz as a shared library.
    

    Original issue reported on code.google.com by [email protected] on 8 Aug 2012 at 3:14

    Priority-Medium Type-Defect auto-migrated 
    opened by GoogleCodeExporter 6
  • mz_crc32 fails on x86_64

    mz_crc32 fails on x86_64

    using a 32bit unsigned int instead of a 64bit unsigned long fixes the problem.
    
    -mz_ulong mz_crc32(mz_ulong crc, const unsigned char *ptr, size_t buf_len);
    +mz_uint32 mz_crc32(mz_uint32 crc, const unsigned char *ptr, size_t buf_len);
    
    also the typedef for mz_uint32 must be moved to preceed this.
    

    Original issue reported on code.google.com by [email protected] on 24 Feb 2012 at 2:28

    Priority-Medium Type-Defect auto-migrated 
    opened by GoogleCodeExporter 6
  • miniz fails to decompress valid ZIP archive

    miniz fails to decompress valid ZIP archive

    What steps will reproduce the problem?
    1. Complie attached test_crash.c with latest miniz.
    2. Run it against attached test.zip.
    
    What is the expected output? What do you see instead?
    Expected is MANIFEST.MF contents since test.zip is a valid archive.
    However it fails to decompress file contents.
    
    What version of the product are you using? On what operating system?
    Miniz 1.11b, Win7 and GNU/Linux (gcc 4.1.2)
    
    Please provide any additional information below.
    
    I've discovered that failure occurs in tinfl_decompress() routine:
    
    ...
            TINFL_HUFF_DECODE(26, dist, &r->m_tables[1]);
            num_extra = s_dist_extra[dist]; dist = s_dist_base[dist];
            if (num_extra) { mz_uint extra_bits; TINFL_GET_BITS(27, extra_bits, num_extra); dist += extra_bits; }
    
            dist_from_out_buf_start = pOut_buf_cur - pOut_buf_start;
            if ((dist > dist_from_out_buf_start) && (decomp_flags & TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF))
            {
              TINFL_CR_RETURN_FOREVER(37, TINFL_STATUS_FAILED);
            }
    ...
    
    
    dist occures to be 2049 which is much greater than dist_from_out_buf_start thus 
    decompression fails.
    

    Original issue reported on code.google.com by [email protected] on 22 Nov 2011 at 6:59

    Attachments:

    Priority-Medium Type-Defect auto-migrated 
    opened by GoogleCodeExporter 6
  • mz_uncompress with MZ_NO_FLUSH flag

    mz_uncompress with MZ_NO_FLUSH flag

    I wanted to implement these 2 simple functions:

    string mCompress(string content);
    string mUncompress(string content);
    

    by simply using the mz_compress/mz_uncompress functions. For mCompress, I came up with this working solution:

    string mCompress(string content){
        int src_len = content.size();
        unsigned long cmp_len = mz_compressBound(src_len);
        unsigned char *pCmp = (unsigned char *)malloc((size_t)cmp_len);
        string ret;
        int cmp_status = mz_compress(pCmp, &cmp_len, (const unsigned char *)content.c_str(), src_len);
        if(cmp_status == Z_OK){
            ret = string((char*)pCmp, cmp_len);
        }
        free(pCmp);
        return ret;
    }
    

    But for mUncompress, things got more complicated because the current implementation of mz_uncompress uses the MZ_FINISH flag which, according to the documentation, implies that the input and output buffers are large enough to hold the entire compressed/decompressed file. The problem with this approach is that sometimes you don't know the buffer size that you need to allocate when uncompressing an unknown compressed string. So I came up with this solution:

    int xuncompress(unsigned char **pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong source_len){
        mz_stream stream;
        int status, bufLen = *pDest_len;
        memset(&stream, 0, sizeof(stream));
    
        // In case mz_ulong is 64-bits (argh I hate longs).
        if ((source_len | *pDest_len) > 0xFFFFFFFFU) return MZ_PARAM_ERROR;
    
        stream.next_in = pSource;
        stream.avail_in = (mz_uint32)source_len;
        stream.next_out = *pDest;
        stream.avail_out = (mz_uint32)bufLen;
    
        status = mz_inflateInit(&stream);
        if (status != MZ_OK)
            return status;
    
        while(status == MZ_OK){
            status = mz_inflate(&stream, MZ_NO_FLUSH);
            if(status == MZ_OK){   // if status == MZ_STREAM_END, we don't need to realloc anymore
                bufLen += *pDest_len;
                *pDest = (unsigned char *)realloc(*pDest, bufLen);
                stream.next_out = *pDest + stream.total_out;
                stream.avail_out = bufLen - stream.total_out;
            }
        }
        *pDest_len = stream.total_out;
    
        return mz_inflateEnd(&stream);
    }
    
    string mUncompress(string content){
        int src_len = content.size();
        unsigned long cmp_len = src_len * 2;
        unsigned char *pCmp = (unsigned char *)malloc((size_t)cmp_len);
        string ret;
        int cmp_status = xuncompress(&pCmp, &cmp_len, (const unsigned char *)content.c_str(), src_len);
        if(cmp_status == Z_OK){
            ret = string((char*)pCmp, cmp_len);
        }
        free(pCmp);
        return ret;
    }
    

    xuncompress is similar to mz_uncompress, but it calls mz_inflate with the MZ_NO_FLUSH flag and realloc's the buffer by incrementing with the initial *pDest_len that was passed into the function.

    This works great and I'm able to handle compression/uncompression properly when my application communicates with my Node.js server (where I use the zlib module to compress/uncompress). However, I'm a bit skeptical about my implementation. Anyways, I'd appreciate if anyone could give me any inputs on how to improve this or even if there's a better solution that I haven't found yet.

    Thanks!

    opened by reliasn 4
  • stack usage?

    stack usage?

    Trying to use compress on a ADI blackfin, which works with a 32kBytes stack but doesn't for a 4k. Stack is located in on-chip sram which is not big :( Is there any data on how much stack is acutally used? Any way to reduce stack consumption? I found tdefl_optimize_huffman_table consumes about 5700 bytes which seems to be the bad guy for me

    opened by al-65 0
  • New release?

    New release?

    Looking on https://github.com/richgel999/miniz/compare/2.2.0...master I think that it would be good to make the new release. Any chance to to do that soon? 🤔

    opened by kloczek 0
  • Please make an example that zips a directory into an output zip file.

    Please make an example that zips a directory into an output zip file.

    I have been searching everywhere for a zip library that zips a directory and writes to an output zip file. But every library talks about these things called entries and archives. All I want to do is to zip a directory and I am having so much trouble doing it.

    It's like when you compress a directory and it gives you a zip file. I am using windows 11

    opened by beemibrahim 0
  • pkg-config tweaks

    pkg-config tweaks

    /usr/share/pkgconfig should be used for architecture independent libraries (e.g. data or scripts), while an architecture dependent directory like /usr/lib64/pkgconfig should be used for native binaries.

    Also set the include path to allow users to include <miniz.h> instead of <miniz/miniz.h> which seems to be the intended behaviour.

    opened by MatthewGentoo 0
Owner
Rich Geldreich
Rich Geldreich
zlib replacement with optimizations for "next generation" systems.

zlib-ng zlib data compression library for the next generation systems Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org) C

zlib-ng 1.1k Jul 25, 2022
Fork of the popular zip manipulation library found in the zlib distribution.

minizip-ng 3.0.0 minizip-ng is a zip manipulation library written in C that is supported on Windows, macOS, and Linux. Developed and maintained by Nat

zlib-ng 926 Jul 26, 2022
Fork of the popular zip manipulation library found in the zlib distribution.

minizip-ng 3.0.1 minizip-ng is a zip manipulation library written in C that is supported on Windows, macOS, and Linux. Developed and maintained by Nat

zlib-ng 929 Aug 5, 2022
PNGFuse is a cross-platform application that allows you to embed and extract full zlib-compressed files within PNG metadata.

PNGFuse PNGFuse is a portable, lightweight, and cross-platform application written in C++ that allows you to embed and extract full zlib-compressed fi

Eta 3 Dec 29, 2021
A C++ header-only ZLib wrapper

A C++ ZLib wrapper This C++ header-only library enables the use of C++ standard iostreams to access ZLib-compressed streams. For input access (decompr

Matei David 222 Aug 3, 2022
gzip (GNU zip) is a compression utility designed to be a replacement for 'compress'

gzip (GNU zip) is a compression utility designed to be a replacement for 'compress'

ACM at UCLA 7 Apr 27, 2022
NanaZip is an open source file archiver intended for the modern Windows experience

NanaZip is an open source file archiver intended for the modern Windows experience, forked from the source code of well-known open source file archiver 7-Zip 21.03.

M2-Team 3k Aug 5, 2022
7zip source code slightly modified for the Surface RT

7zip-rt 7zip source code slightly modified for running in the Surface RT. Tested only on Windows RT 10. Requirements Building Visual Studio 2012 Expre

sader 4 Jul 28, 2022
Single header lib for JPEG encoding. Public domain. C99. stb style.

tiny_jpeg.h A header-only public domain implementation of Baseline JPEG compression. Features: stb-style header only library. Does not do dynamic allo

Sergio Gonzalez 196 Jul 20, 2022
A C++ static library offering a clean and simple interface to the 7-zip DLLs.

bit7z A C++ static library offering a clean and simple interface to the 7-zip DLLs Supported Features • Getting Started • Download • Requirements • Bu

Riccardo 279 Jul 29, 2022
Multi-format archive and compression library

Welcome to libarchive! The libarchive project develops a portable, efficient C library that can read and write streaming archives in a variety of form

null 1.8k Aug 1, 2022
LZFSE compression library and command line tool

LZFSE This is a reference C implementation of the LZFSE compressor introduced in the Compression library with OS X 10.11 and iOS 9. LZFSE is a Lempel-

null 1.7k Jul 30, 2022
Small strings compression library

SMAZ - compression for very small strings ----------------------------------------- Smaz is a simple compression library suitable for compressing ver

Salvatore Sanfilippo 990 Jul 27, 2022
A massively spiffy yet delicately unobtrusive compression library.

ZLIB DATA COMPRESSION LIBRARY zlib 1.2.11 is a general purpose data compression library. All the code is thread safe. The data format used by the z

Mark Adler 3.7k Jul 31, 2022
PhysFS++ is a C++ wrapper for the PhysicsFS library.

PhysFS++ PhysFS++ is a C++ wrapper for the excellent PhysicsFS library by Ryan C. Gordon and others. It is licensed under the zlib license - same as P

Kevin Howell 78 May 17, 2022
An embedded-friendly library for decompressing files from zip archives

An 'embedded-friendly' (aka Arduino) library to extract and decompress files from ZIP archives

Larry Bank 25 Jul 17, 2022
Gzip header-only C++ library

Gzip C++ lib for gzip compression and decompression. Extracted from mapnik-vector-tile for light-weight modularity. Usage // Include the specific gzip

Mapbox 226 Aug 5, 2022
Simple data packing library (written in C99)

Features Compressed file pack creation Runtime file pack reading Supported operating systems Ubuntu MacOS Windows Build requirements C99 compiler CMak

Nikita Fediuchin 3 Feb 25, 2022
A simple C library implementing the compression algorithm for isosceles triangles.

orvaenting Summary A simple C library implementing the compression algorithm for isosceles triangles. License This project's license is GPL 2 (as of J

Kevin Matthes 0 Apr 1, 2022