miniz: Single C source file zlib-replacement library, originally from code.google.com/p/miniz

What steps will reproduce the problem?

1. Provide extra data in the input buffer for tinfl_decompress().
2. An extra byte is consumed beyond the deflated data.
3. The output parameter pIn_buf_size of tinfl_decompress() has an extra byte.

This is a problem for gzip stream data since it has 8 extra bytes after the 
deflated data (4 bytes CRC and 4 bytes original length).  When reading gzip 
stream data, it's often not known when the end of stream is.  One has to read 
as much as data there are and pass them to tinfl_decompress() to let it decode 
the deflated data.  Once tinfl_decompress() stops, the remaining data in the 
buffer are interpreted separately by the caller.

What is the expected output? What do you see instead?

Expect decompress() to consume exactly the number of deflated data, or if an 
extra byte is consumed, let the caller knows so that it can unread the byte in 
the buffer.

What version of the product are you using? On what operating system?
version 1.15.  On Win32.

Please provide any additional information below.

Sample gzip deflated data causing the problem.

[0x73, 0x74, 0x72, 0x76, 0x71, 0x75, 0x73, 0xF7, 0xE0, 0xE5, 0x02, 0x00,
  0x94, 0xA6, 0xD7, 0xD0,  0x0A, 0x00, 0x00, 0x00]

The first 12 bytes are the deflated data.  The next 4 bytes are the CRC.  The 
last 4 bytes are the original length.  tinfl_decompress() should stop after 12 
bytes.

The original data is "ABCDEFGH\r\n"

--------

I found the problem where it's too late to call TINFL_HUFF_BITBUF_FILL().  It 
is only called when only 1 byte are left in the input buffer, which is never 
triggered for the above case where there are 8 extra bytes.

I have changed the number of bytes to check from ((pIn_buf_end - pIn_buf_cur) < 
2) to ((pIn_buf_end - pIn_buf_cur) < 16) and that works.  See attached file for 
the diffs.

It's not an airtight fix since it only helps the cases with input buffer upto 
15 extra bytes.  It makes miniz work on gzip files.

The best fix would be to consume the extra byte as needed but fix up the 
pointer by counting how many bits left in bit_buf.  Since the data are in a 
buffer, it's easy to move the pointer back as needed.

Original issue reported on code.google.com by [email protected] on 18 Oct 2013 at 10:01

Attachments:

• miniz_fix.txt

The original Google Code project this Git repository is based on, listed miniz as being under Unlicense.

Completely free: Public domain in jurisdictions that recognize copyright laws, with a license patterned after the public domain SQLite project, see unlicense.org.

But with commit 224d207ce8fffb908e156d27478be3afb5d83e6a @uroni marked the project as MIT licensed with the vague commit comment.

MiniZ ZIP64 was part of valve vogl which is MIT licensed

Is there a source to that Valve vogl claim?

The Huffman tables are dimensioned after the biggest one (288) but that wastes a lot of memory for the other two which are considerably smaller (32 and 19). This PR creates separate arrays with proper lengths.

valgrind error : ==23553== Conditional jump or move depends on uninitialised value(s) ==23553== at 0x4032AC: tdefl_find_match (in miniz/bin_linux/example1) ==23553== by 0x407B33: tdefl_compress_normal (in miniz/bin_linux/example1) ==23553== by 0x408DCB: mz_deflate (in miniz/bin_linux/example1) ==23553== by 0x4024CC: main (in miniz/bin_linux/example1) ==23553==

Just curious has anyone got miniz working on arm-based Android devices. 

I had a stripped down version of miniz (v115_r4), which contains only 
mz_compress() and mz_uncompress() functions and their dependenices. It is 
working well on a Ubuntu desktop, however, after I ported it to Android, I 
found that if the compress block size is 16K bytes or bigger, the compression 
can easily fail. Most time it failed silently, I only noticed the problem when 
trying to uncompress the compressed the content. 

Confirmed the issue is on the compression side by trying to uncompress the same 
compressed block using miniz on Ubuntu PC, it failed with the same error: 
MZ_DATA_ERROR. Also tried using miniz on Ubuntu PC to compress and uncompress 
the same original data block (16Kbytes), and it works fine.

I will for sure debug further and report back to this thread, just in case 
someone already done the simliar porting work and knows what need to be tweaked 
...

Btw, Rich, very nice tool, thanks a lot for sharing. 

Original issue reported on code.google.com by [email protected] on 24 Oct 2014 at 3:36

I am using miniz vendored in a project. I am using strict warning settings in clang and gcc. I am getting warnings due to miniz "conversion", "sign-conversion", "cast-qual", "cast-align", "unused-parameter", "switch-enum", "c++-compat". It would be nice if some of those could be fixed in miniz. Would you accept patches?

Furthermore by using clang static analyis via the "scan-build" tool a few more issues are uncovered. Unused assignments etc.

Hi,

I've been doing some fixes and pushed them up to this git repository:
https://github.com/paulharris/miniz

I imported the SVN repo, so all the SVN patches are contained within too.

Are you still active in this project?  There are quite a few open bugs that 
seem to be already fixed, or easily fixed elsewhere.

Also note that I'm not the only alternative miniz repo out there,
eg, this one has changes for working on Android:
https://github.com/codefireXperiment/external_libminiz

cheers,
Paul

Original issue reported on code.google.com by [email protected] on 8 Oct 2013 at 3:23

miniz is designed as a single file to be included by another source file via 
#include "miniz.c" as stated in the example applications. This works fine when 
miniz is used within one project. When two projects using miniz unexpected 
results arise.

Situation:

Shared library A includes miniz.c. Application B also includes miniz.c. 
Application A will load B.


Problem:

When B loads library A there are two copies of the symbols related to miniz. 
Depending on the OS, B might fail to load A. Also, depending on the OS the 
miniz that is part of A might be used by B or vice versa. This can be a major 
issue if A and B have different versions (especially if they are not ABI or API 
compatible) of miniz included.


Solutions:

1) Make all parts within miniz.c static. This will make everything in miniz.c 
only visible to what is including it. This will prevent the miniz symbols from 
being exposed publicly.

This could be facilitated by setting a define similar to MINIZ_NO_ARCHIVE_APIS. 
Something like:

#ifdef MINIZ_STATIC
# define MINIZ_STATIC static
#else
# define MINIZ_STATIC
#endif

Then any any function that would be considered public would be prefixed by 
MINIZ_STATIC:

MINIZ_STATIC int mz_uncompress(...

2) Break miniz.c appart into miniz.h and miniz.c to facilitate building and 
installing miniz as a shared library.

Original issue reported on code.google.com by [email protected] on 8 Aug 2012 at 3:14

using a 32bit unsigned int instead of a 64bit unsigned long fixes the problem.

-mz_ulong mz_crc32(mz_ulong crc, const unsigned char *ptr, size_t buf_len);
+mz_uint32 mz_crc32(mz_uint32 crc, const unsigned char *ptr, size_t buf_len);

also the typedef for mz_uint32 must be moved to preceed this.

Original issue reported on code.google.com by [email protected] on 24 Feb 2012 at 2:28

What steps will reproduce the problem?
1. Complie attached test_crash.c with latest miniz.
2. Run it against attached test.zip.

What is the expected output? What do you see instead?
Expected is MANIFEST.MF contents since test.zip is a valid archive.
However it fails to decompress file contents.

What version of the product are you using? On what operating system?
Miniz 1.11b, Win7 and GNU/Linux (gcc 4.1.2)

Please provide any additional information below.

I've discovered that failure occurs in tinfl_decompress() routine:

...
        TINFL_HUFF_DECODE(26, dist, &r->m_tables[1]);
        num_extra = s_dist_extra[dist]; dist = s_dist_base[dist];
        if (num_extra) { mz_uint extra_bits; TINFL_GET_BITS(27, extra_bits, num_extra); dist += extra_bits; }

        dist_from_out_buf_start = pOut_buf_cur - pOut_buf_start;
        if ((dist > dist_from_out_buf_start) && (decomp_flags & TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF))
        {
          TINFL_CR_RETURN_FOREVER(37, TINFL_STATUS_FAILED);
        }
...


dist occures to be 2049 which is much greater than dist_from_out_buf_start thus 
decompression fails.

Original issue reported on code.google.com by [email protected] on 22 Nov 2011 at 6:59

Attachments:

• test.zip
• test_crash.c

Hello, firstly thanks for your work and share. Currently I'm trying to figure out how to compress/zip data/file on a ESP32 microcontroller and wandering in google for it. Whenever I do search for esp32 compressing algorithms and libraries I see this work. I wasn't sure if this library will work with Arduino IDE but I tried to copy .c,.h files and test the example but wasn't able to success, so I wonder if this library works with Arduino IDE and if it is, how?

The following program:

#include <stdio.h>
#include "miniz.c"
int
main(void) {
  FILE* f = fopen("files.zip", "rb");
  if (NULL == f) {
    printf("error: 'fopen'\n");
    exit(1);
  }
  mz_zip_archive ar = {0};
  if (!mz_zip_reader_init_cfile(&ar, f, 0, 0)) {
    printf("error: 'mz_zip_reader_init_cfile'\n");
    exit(1);
  }
  int nfiles = (int)mz_zip_reader_get_num_files(&ar);
  for (int a = 0; a < nfiles; ++a) {
    char fname[MZ_ZIP_MAX_ARCHIVE_FILENAME_SIZE];
    int nfname = -1 + (int)mz_zip_reader_get_filename(&ar, (unsigned int)a, fname, sizeof(fname));
    fname[nfname] = '\0';
    printf("%s\n", fname);
  }
  return 0;
}

prints

files/
files/aaa.txt
files/bcd.txt
files/b_d.txt
files/ccc.txt

instead of

files/
files/aaa.txt
files/b_d.txt
files/bcd.txt
files/ccc.txt

files/b_d.txt should come before files/bcd.txt because '_'(95) < 'c'(99)

Pragmatic approach: Treat Linux on x86_64 the same as Mac OS X and FreeBSD, i.e. as a Unix-like 64-bit operating system with known large file support.

Fixes #257

tinfl_decompress returns TINFL_STATUS_BAD_PARAM if output buffer is not power of 2. I'm not sure if this is a bug or not, but according to the comment below this should be ignored if output buffer is large enough.

https://github.com/richgel999/miniz/blob/master/miniz_tinfl.c#L205

/* Ensure the output buffer's size is a power of 2, unless the output buffer is large enough to hold the entire output file (in which case it doesn't matter). */
if (((out_buf_size_mask + 1) & out_buf_size_mask) || (pOut_buf_next < pOut_buf_start))
{
    *pIn_buf_size = *pOut_buf_size = 0;
    return TINFL_STATUS_BAD_PARAM;
}

The if statement in the code above is always true when length is not a power of 2.

Hi, since v.3.0 I cannot use miniz anymore. I'm including miniz.c/miniz.h from the ZIP archive from the release page in my project, but compilation fails:

../../master/application/./3rdparty/miniz/miniz.c:1254:22: error: default initialization of an object of const type 'const mz_uint[11]' (aka 'const unsigned int[11]')
static const mz_uint s_tdefl_num_probes[11];
                     ^
../../master/application/./3rdparty/miniz/miniz.c:2113:22: error: redefinition of 's_tdefl_num_probes'
static const mz_uint s_tdefl_num_probes[11] = { 0, 1, 6, 32, 16, 32, 128, 256, 512, 768, 1500 };
                     ^
../../master/application/./3rdparty/miniz/miniz.c:1254:22: note: previous definition is here
static const mz_uint s_tdefl_num_probes[11];
                     ^

With v2.2.0 everything is fine.

Edit: Just tested again with v3.0.1 -> still the same error.

Hi, I want to encrypt zip file, I use zip_stream_open instead of zip_open. And use zip_stream_copy to get the buffer, encrypt it and write it to the file. Is there anything wrong with me doing this?