Lossless data compression codec with LZMA-like ratios but 1.5x-8x faster decompression speed, C/C++

Last update: Dec 22, 2022

Related tags

Overview

LZHAM - Lossless Data Compression Codec

Public Domain (see LICENSE)

LZHAM is a lossless data compression codec written in C/C++ (specifically C++03), with a compression ratio similar to LZMA but with 1.5x-8x faster decompression speed. It officially supports Linux x86/x64, Windows x86/x64, OSX, and iOS, with Android support on the way.

The old alpha version of LZHAM (bitstream incompatible with the v1.x release) is here: https://github.com/richgel999/lzham_alpha

Introduction

LZHAM is a lossless (LZ based) data compression codec optimized for particularly fast decompression at very high compression ratios with a zlib compatible API. It's been developed over a period of 3 years and alpha versions have already shipped in many products. (The alpha is here: https://code.google.com/p/lzham/) LZHAM's decompressor is slower than zlib's, but generally much faster than LZMA's, with a compression ratio that is typically within a few percent of LZMA's and sometimes better.

LZHAM's compressor is intended for offline use, but it is tested alongside the decompressor on mobile devices and is usable on the faster settings.

LZHAM's decompressor currently has a higher cost to initialize than LZMA, so the threshold where LZHAM is typically faster vs. LZMA decompression is between 1000-13,000 of *compressed* output bytes, depending on the platform. It is not a good small block compressor: it likes large (10KB-15KB minimum) blocks.

LZHAM has simple support for patch files (delta compression), but this is a side benefit of its design, not its primary use case. Internally it supports LZ matches up to ~64KB and very large dictionaries (up to .5 GB).

LZHAM may be valuable to you if you compress data offline and distribute it to many customers, care about read/download times, and decompression speed/low CPU+power use are important to you.

I've been profiling LZHAM vs. LZMA and publishing the results on my blog: http://richg42.blogspot.com

Some independent benchmarks of the previous alpha versions: http://heartofcomp.altervista.org/MOC/MOCADE.htm, http://mattmahoney.net/dc/text.html

LZHAM has been integrated into the 7zip archiver (command line and GUI) as a custom codec plugin: http://richg42.blogspot.com/2015/02/lzham-10-integrated-into-7zip-command.html

10GB Benchmark Results

Results with 7zip-LZHAM 9.38 32-bit (64MB dictionary) on Matt Mahoney's 10GB benchmark:

LZHAM (-mx=8): 3,577,047,629 Archive Test Time: 70.652 secs
LZHAM (-mx=9): 3,573,782,721 Archive Test Time: 71.292 secs
LZMA  (-mx=9): 3,560,052,414 Archive Test Time: 223.050 secs
7z .ZIP      : 4,681,291,655 Archive Test Time: 73.304 secs (unzip v6 x64 test time: 61.074 secs)

Most Common Question: So how does it compare to other libs like LZ4?

There is no single compression algorithm that perfectly suites all use cases and practical constraints. LZ4 and LZHAM are tools which lie at completely opposite ends of the spectrum:

LZ4: A symmetrical codec with very fast compression and decompression but very low ratios. Its compression ratio is typically less than even zlib's (which uses a 21+ year old algorithm). LZ4 does a good job of trading off a large amount of compression ratio for very fast overall throughput. Usage example: Reading LZMA/LZHAM/etc. compressed data from the network and decompressing it, then caching this data locally on disk using LZ4 to reduce disk usage and decrease future loading times.
LZHAM: A very asymmetrical codec with slow compression speed, but with a very competitive (LZMA-like) compression ratio and reasonably fast decompression speeds (slower than zlib, but faster than LZMA). LZHAM trades off a lot of compression throughput for very high ratios and higher decompression throughput relative to other codecs in its ratio class (which is LZMA, which runs circles around LZ4's ratio). Usage example: Compress your product's data once on a build server, distribute it to end users over a slow media like the internet, then decompress it on the end user's device.

How Much Memory Does It Need?

For decompression it's easy to compute:

Buffered mode: decomp_mem = dict_size + ~34KB for work tables
Unbuffered mode: decomp_mem = ~34KB

I'll be honest here, the compressor is currently an angry beast when it comes to memory. The amount needed depends mostly on the compression level and dict. size. It's approximately (max_probes=128 at level -m4): comp_mem = min(512 * 1024, dict_size / 8) * max_probes * 6 + dict_size * 9 + 22020096

Compression mem usage examples from Windows lzhamtest_x64 (note the equation is pretty off for small dictionary sizes):

32KB: 11MB
128KB: 21MB
512KB: 63MB
1MB: 118MB
8MB: 478MB
64MB: 982MB
128MB: 1558MB
256MB: 2710MB
512MB: 5014MB

Compressed Bitstream Compatibility

v1.0's bitstream format is now locked in place, so any future v1.x releases will be backwards/forward compatible with compressed files written with v1.0. The only thing that could change this are critical bugfixes.

Note LZHAM v1.x bitstreams are NOT backwards compatible with any of the previous alpha versions on Google Code.

Platforms/Compiler Support

LZHAM currently officially supports x86/x64 Linux, iOS, OSX, FreeBSD, and Windows x86/x64. At one time the codec compiled and ran fine on Xbox 360 (PPC, big endian). Android support is coming next. It should be easy to retarget by modifying the macros in lzham_core.h.

LZHAM has optional support for multithreaded compression. It supports gcc built-ins or MSVC intrinsics for atomic ops. For threading, it supports OSX specific Pthreads, generic Pthreads, or Windows API's.

For compilers, I've tested with gcc, clang, and MSVC 2008, 2010, and 2013. In previous alphas I also compiled with TDM-GCC x64.

API

LZHAM supports streaming or memory to memory compression/decompression. See include/lzham.h. LZHAM can be linked statically or dynamically, just study the headers and the lzhamtest project. On Linux/OSX, it's only been tested with static linking so far.

LZHAM also supports a usable subset of the zlib API with extensions, either include/zlib.h or #define LZHAM_DEFINE_ZLIB_API and use include/lzham.h.

Usage Tips

Always try to use the smallest dictionary size that makes sense for the file or block you are compressing, i.e. don't use a 128MB dictionary for a 15KB file. The codec doesn't automatically choose for you because in streaming scenarios it has no idea how large the file or block will be.
The larger the dictionary, the more RAM is required during compression and decompression. I would avoid using more than 8-16MB dictionaries on iOS.
For faster decompression, prefer "unbuffered" decompression mode vs. buffered decompression (avoids a dictionary alloc and extra memcpy()'s), and disable adler-32 checking. Also, use the built-in LZHAM API's, not the zlib-style API's for fastest decompression.
Experiment with the "m_table_update_rate" compression/decompression parameter. This setting trades off a small amount of ratio for faster decompression. Note the m_table_update_rate decompression parameter MUST match the setting used during compression (same for the dictionary size). It's up to you to store this info somehow.
Avoid using LZHAM on small compressed blocks, where small is 1KB-10KB compressed bytes depending on the platform. LZHAM's decompressor is only faster than LZMA's beyond the small block threshold. Optimizing LZHAM's decompressor to reduce its startup time relative to LZMA is a high priority.
For best compression (I've seen up to ~4% better), enable the compressor's "extreme" parser, which is much slower but finds cheaper paths through a much denser parse graph. Note the extreme parser can greatly slow down on files containing large amounts of repeated data/strings, but it is guaranteed to finish.
The compressor's m_level parameter can make a big impact on compression speed. Level 0 (LZHAM_COMP_LEVEL_FASTEST) uses a much simpler greedy parser, and the other levels use near-optimal parsing with different heuristic settings.
Check out the compressor/decompressor reinit() API's, which are useful if you'll be compressing or decompressing many times. Using the reinit() API's is a lot cheaper than fully initializing/deinitializing the entire codec every time.
LZHAM's compressor is no speed demon. It's usually slower than LZMA's, sometimes by a wide (~2x slower or so) margin. In "extreme" parsing mode, it can be many times slower. This codec was designed with offline compression in mind.
One significant difference between LZMA and LZHAM is how uncompressible files are handled. LZMA usually expands uncompressible files, and its decompressor can bog down and run extremely slowly on uncompressible data. LZHAM internally detects when each 512KB block is uncompressible and stores these blocks as uncompressed bytes instead. LZHAM's literal decoding is significantly faster than LZMA's, so the more plain literals in the output stream, the faster LZHAM's decompressor runs vs. LZMA's.
General advice (applies to LZMA and other codecs too): If you are compressing large amounts of serialized game assets, sort the serialized data by asset type and compress the whole thing as a single large "solid" block of data. Don't compress each individual asset, this will kill your ratio and have a higher decompression startup cost. If you need random access, consider compressing the assets lumped together into groups of a few hundred kilobytes (or whatever) each.
LZHAM is a raw codec. It doesn't include any sort of preprocessing: EXE rel to abs jump transformation, audio predictors, etc. That's up to you to do, before compression.

Codec Test App

lzhamtest_x86/x64 is a simple command line test program that uses the LZHAM codec to compress/decompress single files. lzhamtest is not intended as a file archiver or end user tool, it's just a simple testbed.

-- Usage examples:

Compress single file "source_filename" to "compressed_filename": lzhamtest_x64 c source_filename compressed_filename
Decompress single file "compressed_filename" to "decompressed_filename": lzhamtest_x64 d compressed_filename decompressed_filename
Compress single file "source_filename" to "compressed_filename", then verify the compressed file decompresses properly to the source file: lzhamtest_x64 -v c source_filename compressed_filename
Recursively compress all files under specified directory and verify that each file decompresses properly: lzhamtest_x64 -v a c:\source_path

-- Options

Set dictionary size used during compressed to 1MB (2^20): lzhamtest_x64 -d20 c source_filename compressed_filename

Valid dictionary sizes are [15,26] for x86, and [15,29] for x64. (See LZHAM_MIN_DICT_SIZE_LOG2, etc. defines in include/lzham.h.) The x86 version defaults to 64MB (26), and the x64 version defaults to 256MB (28). I wouldn't recommend setting the dictionary size to 512MB unless your machine has more than 4GB of physical memory.

Set compression level to fastest: lzhamtest_x64 -m0 c source_filename compressed_filename
Set compression level to uber (the default): lzhamtest_x64 -m4 c source_filename compressed_filename
For best possible compression, use -d29 to enable the largest dictionary size (512MB) and the -x option which enables more rigorous (but ~4X slower!) parsing: lzhamtest_x64 -d29 -x -m4 c source_filename compressed_filename

See lzhamtest_x86/x64.exe's help text for more command line parameters.

Compiling LZHAM

Linux: Use "cmake ." then "make". The cmake script only supports Linux at the moment. (Sorry, working on build systems is a drag.)
OSX/iOS: Use the included XCode project. (NOTE: I haven't merged this over yet. It's coming!)
Windows: Use the included VS 2010 project

IMPORTANT: With clang or gcc compile LZHAM with "No strict aliasing" ENABLED: -fno-strict-aliasing

I DO NOT test or develop the codec with strict aliasing:

It might work fine, I don't know yet. This is usually not a problem with MSVC, which defaults to strict aliasing being off.

ANSI C/C++

LZHAM supports compiling as plain vanilla ANSI C/C++. To see how the codec configures itself check out lzham_core.h and search for "LZHAM_ANSI_CPLUSPLUS". All platform specific stuff (unaligned loads, threading, atomic ops, etc.) should be disabled when this macro is defined. Note, the compressor doesn't use threads or atomic operations when built this way so it's going to be pretty slow. (The compressor was built from the ground up to be threaded.)

Known Problems

LZHAM's decompressor is like a drag racer that needs time to get up to speed. LZHAM is not intended or optimized to be used on "small" blocks of data (less than ~10,000 bytes of *compressed* data on desktops, or around 1,000-5,000 on iOS). If your usage case involves calling the codec over and over with tiny blocks then LZMA, LZ4, Deflate, etc. are probably better choices.

The decompressor still takes too long to init vs. LZMA. On iOS the cost is not that bad, but on desktop the cost is high. I have reduced the startup cost vs. the alpha but there's still work to do.

The compressor is slower than I would like, and doesn't scale as well as it could. I added a reinit() method to make it initialize faster, but it's not a speed demon. My focus has been on ratio and decompression speed.

I use tabs=3 spaces, but I think some actual tabs got in the code. I need to run the sources through ClangFormat or whatever.

Special Thanks

Thanks to everyone at the http://encode.ru forums. I read these forums as a lurker before working on LZHAM, and I studied every LZ related post I could get my hands on. Especially anything related to LZ optimal parsing, which still seems like a black art. LZHAM was my way of learning how to implement optimal parsing (and you can see this if you study the progress I made in the early alphas on Google Code).

Also, thanks to Igor Pavlov, the original creator of LZMA and 7zip, for advancing the start of the art in LZ compression.

Comments

decompress_memory fails on data compressed with compress_memory

Maybe I'm doing something wrong, but I feel like this should get me LZHAM_DECOMP_STATUS_SUCCESS not LZHAM_DECOMP_STATUS_FAILED_BAD_CODE:

#define LOREM_IPSUM                                                     \
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vulputate " \
  "lectus nisl, vitae ultricies justo dictum nec. Vestibulum ante ipsum " \
  "primis in faucibus orci luctus et ultrices posuere cubilia Curae; "  \
  "Suspendisse suscipit quam a lectus adipiscing, sed tempor purus "    \
  "cursus. Vivamus id nulla eget elit eleifend molestie. Integer "      \
  "sollicitudin lorem enim, eu eleifend orci facilisis sed. Pellentesque " \
  "sodales luctus enim vel viverra. Cras interdum vel nisl in "         \
  "facilisis. Curabitur sollicitudin tortor vel congue "                \
  "auctor. Suspendisse egestas orci vitae neque placerat blandit.\n"    \
  "\n"                                                                  \
  "Aenean sed nisl ultricies, vulputate lorem a, suscipit nulla. Donec " \
  "egestas volutpat neque a eleifend. Nullam porta semper "             \
  "nunc. Pellentesque adipiscing molestie magna, quis pulvinar metus "  \
  "gravida sit amet. Vestibulum mollis et sapien eu posuere. Quisque "  \
  "tristique dignissim ante et aliquet. Phasellus vulputate condimentum " \
  "nulla in vulputate.\n"                                               \
  "\n"                                                                  \
  "Nullam volutpat tellus at nisi auctor, vitae mattis nibh viverra. Nunc " \
  "vitae lectus tristique, ultrices nibh quis, lobortis elit. Curabitur " \
  "at vestibulum nisi, nec facilisis ante. Nulla pharetra blandit lacus, " \
  "at sodales nulla placerat eget. Nulla congue varius tortor, sit amet " \
  "tempor est mattis nec. Praesent vitae tristique ipsum, rhoncus "     \
  "tristique lorem. Sed et erat tristique ligula accumsan fringilla eu in " \
  "urna. Donec dapibus hendrerit neque nec venenatis. In euismod sapien " \
  "ipsum, auctor consectetur mi dapibus hendrerit.\n"                   \
  "\n"                                                                  \
  "Phasellus sagittis rutrum velit, in sodales nibh imperdiet a. Integer " \
  "vitae arcu blandit nibh laoreet scelerisque eu sit amet eros. Aenean " \
  "odio felis, aliquam in eros at, ornare luctus magna. In semper "     \
  "tincidunt nunc, sollicitudin gravida nunc laoreet eu. Cras eu tempor " \
  "sapien, ut dignissim elit. Proin eleifend arcu tempus, semper erat et, " \
  "accumsan erat. Praesent vulputate diam mi, eget mollis leo "         \
  "pellentesque eget. Aliquam eu tortor posuere, posuere velit sed, "   \
  "suscipit eros. Nam eu leo vitae mauris condimentum lobortis non quis " \
  "mauris. Nulla venenatis fringilla urna nec venenatis. Nam eget velit " \
  "nulla. Proin ut malesuada felis. Suspendisse vitae nunc neque. Donec " \
  "faucibus tempor lacinia. Vivamus ac vulputate sapien, eget lacinia " \
  "nisl.\n"                                                             \
  "\n"                                                                  \
  "Curabitur eu dolor molestie, ullamcorper lorem quis, egestas "       \
  "urna. Suspendisse in arcu sed justo blandit condimentum. Ut auctor, " \
  "sem quis condimentum mattis, est purus pulvinar elit, quis viverra " \
  "nibh metus ac diam. Etiam aliquet est eu dui fermentum consequat. Cras " \
  "auctor diam eget bibendum sagittis. Aenean elementum purus sit amet " \
  "sem euismod, non varius felis dictum. Aliquam tempus pharetra ante a " \
  "sagittis. Curabitur ut urna felis. Etiam sed vulputate nisi. Praesent " \
  "at libero eleifend, sagittis quam a, varius sapien."
#define LOREM_IPSUM_LENGTH (sizeof(LOREM_IPSUM) - 1)

#include <stdio.h>
#include <lzham.h>
#include <stdint.h>
#include <assert.h>

int main (int argc, char** argv) {
  uint8_t compressed[4096];
  size_t compressed_length;
  uint8_t decompressed[1024 * 1024];
  size_t decompressed_length = sizeof(decompressed);

  lzham_compress_status_t comp_res;
  lzham_compress_params comp_params = {
    sizeof(lzham_compress_params),
    LZHAM_MAX_DICT_SIZE_LOG2_X86,
    LZHAM_COMP_LEVEL_DEFAULT,
    LZHAM_DEFAULT_TABLE_UPDATE_RATE,
    -1,
    0,
    0,
    NULL,
    64,
    64
  };

  lzham_decompress_status_t decomp_res;
  lzham_decompress_params decomp_params = {
    sizeof(lzham_decompress_params),
    LZHAM_MAX_DICT_SIZE_LOG2_X86,
    LZHAM_DEFAULT_TABLE_UPDATE_RATE,
    0,
    0,
    NULL,
    64,
    16
  };

  comp_res = lzham_compress_memory (&comp_params,
                    compressed, &compressed_length,
                    (const lzham_uint8*) LOREM_IPSUM, LOREM_IPSUM_LENGTH,
                    NULL);
  assert (comp_res == LZHAM_COMP_STATUS_SUCCESS);

  fprintf (stdout, "Compressed from %zu bytes to %zu bytes.\n", LOREM_IPSUM_LENGTH, compressed_length);

  decomp_res = lzham_decompress_memory (&decomp_params,
                    decompressed, &decompressed_length,
                    compressed, compressed_length,
                    NULL);
  fprintf (stdout, "Decompressed status = %d (LZHAM_DECOMP_STATUS_FAILED_BAD_CODE = %d)\n",
       decomp_res, LZHAM_DECOMP_STATUS_FAILED_BAD_CODE);

  return 0;
}

opened by nemequ 7

decompress does not immediately write data after a flush

When I flush a compression stream and send the output to a decompression stream the decompression stream just keeps the data in its buffer instead of immediately making it available (like zlib does), which pretty much destroys the usefulness of flushing.

Test case:

#define LOREM_IPSUM                                                     \
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vulputate " \
  "lectus nisl, vitae ultricies justo dictum nec. Vestibulum ante ipsum " \
  "primis in faucibus orci luctus et ultrices posuere cubilia Curae; "  \
  "Suspendisse suscipit quam a lectus adipiscing, sed tempor purus "    \
  "cursus. Vivamus id nulla eget elit eleifend molestie. Integer "      \
  "sollicitudin lorem enim, eu eleifend orci facilisis sed. Pellentesque " \
  "sodales luctus enim vel viverra. Cras interdum vel nisl in "         \
  "facilisis. Curabitur sollicitudin tortor vel congue "                \
  "auctor. Suspendisse egestas orci vitae neque placerat blandit.\n"    \
  "\n"                                                                  \
  "Aenean sed nisl ultricies, vulputate lorem a, suscipit nulla. Donec " \
  "egestas volutpat neque a eleifend. Nullam porta semper "             \
  "nunc. Pellentesque adipiscing molestie magna, quis pulvinar metus "  \
  "gravida sit amet. Vestibulum mollis et sapien eu posuere. Quisque "  \
  "tristique dignissim ante et aliquet. Phasellus vulputate condimentum " \
  "nulla in vulputate.\n"                                               \
  "\n"                                                                  \
  "Nullam volutpat tellus at nisi auctor, vitae mattis nibh viverra. Nunc " \
  "vitae lectus tristique, ultrices nibh quis, lobortis elit. Curabitur " \
  "at vestibulum nisi, nec facilisis ante. Nulla pharetra blandit lacus, " \
  "at sodales nulla placerat eget. Nulla congue varius tortor, sit amet " \
  "tempor est mattis nec. Praesent vitae tristique ipsum, rhoncus "     \
  "tristique lorem. Sed et erat tristique ligula accumsan fringilla eu in " \
  "urna. Donec dapibus hendrerit neque nec venenatis. In euismod sapien " \
  "ipsum, auctor consectetur mi dapibus hendrerit.\n"                   \
  "\n"                                                                  \
  "Phasellus sagittis rutrum velit, in sodales nibh imperdiet a. Integer " \
  "vitae arcu blandit nibh laoreet scelerisque eu sit amet eros. Aenean " \
  "odio felis, aliquam in eros at, ornare luctus magna. In semper "     \
  "tincidunt nunc, sollicitudin gravida nunc laoreet eu. Cras eu tempor " \
  "sapien, ut dignissim elit. Proin eleifend arcu tempus, semper erat et, " \
  "accumsan erat. Praesent vulputate diam mi, eget mollis leo "         \
  "pellentesque eget. Aliquam eu tortor posuere, posuere velit sed, "   \
  "suscipit eros. Nam eu leo vitae mauris condimentum lobortis non quis " \
  "mauris. Nulla venenatis fringilla urna nec venenatis. Nam eget velit " \
  "nulla. Proin ut malesuada felis. Suspendisse vitae nunc neque. Donec " \
  "faucibus tempor lacinia. Vivamus ac vulputate sapien, eget lacinia " \
  "nisl.\n"                                                             \
  "\n"                                                                  \
  "Curabitur eu dolor molestie, ullamcorper lorem quis, egestas "       \
  "urna. Suspendisse in arcu sed justo blandit condimentum. Ut auctor, " \
  "sem quis condimentum mattis, est purus pulvinar elit, quis viverra " \
  "nibh metus ac diam. Etiam aliquet est eu dui fermentum consequat. Cras " \
  "auctor diam eget bibendum sagittis. Aenean elementum purus sit amet " \
  "sem euismod, non varius felis dictum. Aliquam tempus pharetra ante a " \
  "sagittis. Curabitur ut urna felis. Etiam sed vulputate nisi. Praesent " \
  "at libero eleifend, sagittis quam a, varius sapien."
#define LOREM_IPSUM_LENGTH 2725

#include <stdio.h>
#include <lzham.h>
#include <stdint.h>
#include <assert.h>

int main (int argc, char** argv) {
  size_t uncompressed_length = LOREM_IPSUM_LENGTH;
  uint8_t compressed[4096];
  size_t compressed_length = sizeof (compressed);
  uint8_t decompressed[LOREM_IPSUM_LENGTH * 2];
  size_t decompressed_length = sizeof (decompressed);
  size_t pos1, pos2;

  lzham_compress_state_ptr comp;
  lzham_compress_status_t comp_res;
  lzham_compress_params comp_params = {
    sizeof(lzham_compress_params),
    LZHAM_MAX_DICT_SIZE_LOG2_X86,
    LZHAM_COMP_LEVEL_DEFAULT,
    LZHAM_DEFAULT_TABLE_UPDATE_RATE,
    -1,
    0,
    0,
    NULL,
    0,
    0
  };

  lzham_decompress_state_ptr decomp;
  lzham_decompress_status_t decomp_res;
  lzham_decompress_params decomp_params = {
    sizeof(lzham_decompress_params),
    LZHAM_MAX_DICT_SIZE_LOG2_X86,
    LZHAM_DEFAULT_TABLE_UPDATE_RATE,
    0,
    0,
    NULL,
    0,
    0
  };

  const lzham_flush_t flush_or_finish = LZHAM_SYNC_FLUSH;
  // const lzham_flush_t flush_or_finish = LZHAM_FINISH;

  comp = lzham_compress_init (&comp_params);
  comp_res = lzham_compress2 (comp,
                  (const lzham_uint8*) LOREM_IPSUM, &uncompressed_length,
                  compressed, &compressed_length,
                  LZHAM_NO_FLUSH);

  assert (comp_res == LZHAM_COMP_STATUS_NEEDS_MORE_INPUT);
  assert (uncompressed_length == LOREM_IPSUM_LENGTH);
  assert (compressed_length == 0);

  pos1 = 0;
  pos2 = sizeof (compressed) - compressed_length;
  comp_res = lzham_compress2 (comp, (const lzham_uint8*) LOREM_IPSUM, &pos1,
                  compressed + compressed_length, &pos2,
                  flush_or_finish);

  compressed_length += pos2;

  assert ((comp_res == LZHAM_COMP_STATUS_NOT_FINISHED) || // For LZHAM_SYNC_FLUSH
      (comp_res == LZHAM_COMP_STATUS_SUCCESS));       // For LZHAM_FINISH
  assert (compressed_length > 0);

  decomp = lzham_decompress_init (&decomp_params);

  pos1 = compressed_length;
  decomp_res = lzham_decompress (decomp,
                 compressed, &pos1,
                 decompressed, &decompressed_length,
                 flush_or_finish == LZHAM_FINISH);

  assert (pos1 == compressed_length);

  // THIS FAILS
  assert (decompressed_length != 0);
  assert (decompressed_length == LOREM_IPSUM_LENGTH);

  lzham_compress_deinit (comp);
  lzham_decompress_deinit (decomp);

  return 0;
}

opened by nemequ 6

Visual Studio 2013 assert

VS2010 runs fine, but VS2013 asserts during generate_codes.

The reason seems to be that in total_bits(uint v) the call to _BitScanReverse behaves differently in Visual Studio 2013 to Visual Studio 2010. When the variable v is zero, _BitScanReverse(&l, v) will set l to 0xcccccccc and return zero. In VS2010 l does not get changed and so is still zero.

As a workaround fix I changed: if (_BitScanReverse(&l, v)) { l++; }

to: if (_BitScanReverse(&l, v)) { l++; } else { l = 0; }

This gets past this error, although I've yet to fully test the rest of the lib so I don't know if there are any other issues.

opened by Muzza 6

Compiling for Mac OSX 10.10 Yosemite failed: "error: expected unqualified-id"

I'm trying to compile it, revision 4ed83a3, on Mac OSX, and I got the error "error: expected unqualified-id"

Here are the steps I done:

$ mkdir build
$ cd build
$ cmake ..
$ make

I got an error when the first file is being compiled. Compilation message following:

[  5%] Building CXX object lzhamdecomp/CMakeFiles/lzhamdecomp.dir/lzham_assert.cpp.o
In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:260:
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:132:15: error: expected unqualified-id
         if ( LZHAM_HAS_DESTRUCTOR(T) )
              ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:82:64: note: expanded from macro 'LZHAM_HAS_DESTRUCTOR'
#define LZHAM_HAS_DESTRUCTOR(T) ((!scalar_type<T>::cFlag) && (!LZHAM_IS_POD(T)))
                                                               ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
   #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                ^
In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:56:14: error: expected unqualified-id
         if (LZHAM_IS_BITWISE_COPYABLE(T))
             ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
#define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                 ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
   #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                ^
In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:101:14: error: expected unqualified-id
         if (LZHAM_IS_BITWISE_COPYABLE(T))
             ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
#define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                 ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
   #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                ^
In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:257:17: error: expected unqualified-id
            if (LZHAM_IS_BITWISE_COPYABLE(T))
                ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
#define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                 ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
   #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                ^
In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:274:14: error: expected unqualified-id
         if (LZHAM_IS_BITWISE_COPYABLE(T))
             ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
#define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                 ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
   #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                ^
In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:323:14: error: expected unqualified-id
         if (LZHAM_IS_BITWISE_COPYABLE(T))
             ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
#define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                 ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
   #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                ^
In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:573:14: error: expected unqualified-id
            (LZHAM_IS_BITWISE_MOVABLE(T) || (is_vector<T>::cFlag)) ? NULL : object_mover, nofail);
             ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:80:38: note: expanded from macro 'LZHAM_IS_BITWISE_MOVABLE'
#define LZHAM_IS_BITWISE_MOVABLE(T) (LZHAM_IS_BITWISE_COPYABLE(T) || (bitwise_movable<T>::cFlag))
                                     ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
#define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                 ^
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
   #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                ^
In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
/lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:569:81: warning: unused parameter 'nofail' [-Wunused-parameter]
      inline bool increase_capacity(uint min_new_capacity, bool grow_hint, bool nofail = false)
                                                                                ^
1 warning and 7 errors generated.
make[2]: *** [lzhamdecomp/CMakeFiles/lzhamdecomp.dir/lzham_assert.cpp.o] Error 1
make[1]: *** [lzhamdecomp/CMakeFiles/lzhamdecomp.dir/all] Error 2
make: *** [all] Error 2

Here is my CMake cache, in advanced mode (all default or auto detected):

BUILD_SHARED_LIBS:BOOL=ON
BUILD_X64:BOOL=ON
CMAKE_AR:FILEPATH=/usr/bin/ar
CMAKE_BUILD_TYPE:STRING=
CMAKE_COLOR_MAKEFILE:BOOL=ON
CMAKE_CXX_COMPILER:FILEPATH=/usr/bin/c++
CMAKE_CXX_FLAGS:STRING=
CMAKE_CXX_FLAGS_DEBUG:STRING=-g
CMAKE_CXX_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_CXX_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG
CMAKE_C_COMPILER:FILEPATH=/usr/bin/cc
CMAKE_C_FLAGS:STRING=
CMAKE_C_FLAGS_DEBUG:STRING=-g
CMAKE_C_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_C_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG
CMAKE_EXE_LINKER_FLAGS:STRING= 
CMAKE_EXE_LINKER_FLAGS_DEBUG:STRING=
CMAKE_EXE_LINKER_FLAGS_MINSIZEREL:STRING=
CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING=
CMAKE_EXE_LINKER_FLAGS_RELWITHDEBINFO:STRING=
CMAKE_EXPORT_COMPILE_COMMANDS:BOOL=OFF
CMAKE_INSTALL_NAME_TOOL:FILEPATH=/usr/bin/install_name_tool
CMAKE_INSTALL_PREFIX:PATH=/usr/local
CMAKE_LINKER:FILEPATH=/usr/bin/ld
CMAKE_MAKE_PROGRAM:FILEPATH=/usr/bin/make
CMAKE_MODULE_LINKER_FLAGS:STRING= 
CMAKE_MODULE_LINKER_FLAGS_DEBUG:STRING=
CMAKE_MODULE_LINKER_FLAGS_MINSIZEREL:STRING=
CMAKE_MODULE_LINKER_FLAGS_RELEASE:STRING=
CMAKE_MODULE_LINKER_FLAGS_RELWITHDEBINFO:STRING=
CMAKE_NM:FILEPATH=/usr/bin/nm
CMAKE_OBJCOPY:FILEPATH=CMAKE_OBJCOPY-NOTFOUND
CMAKE_OBJDUMP:FILEPATH=CMAKE_OBJDUMP-NOTFOUND
CMAKE_OSX_ARCHITECTURES:STRING=
CMAKE_OSX_DEPLOYMENT_TARGET:STRING=
CMAKE_OSX_SYSROOT:STRING=
CMAKE_RANLIB:FILEPATH=/usr/bin/ranlib
CMAKE_SHARED_LINKER_FLAGS:STRING= 
CMAKE_SHARED_LINKER_FLAGS_DEBUG:STRING=
CMAKE_SHARED_LINKER_FLAGS_MINSIZEREL:STRING=
CMAKE_SHARED_LINKER_FLAGS_RELEASE:STRING=
CMAKE_SHARED_LINKER_FLAGS_RELWITHDEBINFO:STRING=
CMAKE_SKIP_INSTALL_RPATH:BOOL=OFF
CMAKE_SKIP_RPATH:BOOL=OFF
CMAKE_STATIC_LINKER_FLAGS:STRING=
CMAKE_STATIC_LINKER_FLAGS_DEBUG:STRING=
CMAKE_STATIC_LINKER_FLAGS_MINSIZEREL:STRING=
CMAKE_STATIC_LINKER_FLAGS_RELEASE:STRING=
CMAKE_STATIC_LINKER_FLAGS_RELWITHDEBINFO:STRING=
CMAKE_STRIP:FILEPATH=/usr/bin/strip
CMAKE_USE_RELATIVE_PATHS:BOOL=OFF
CMAKE_VERBOSE_MAKEFILE:BOOL=OFF

Additional informations:

cc version:

Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.1.0
Thread model: posix

c++ version:

Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.1.0
Thread model: posix

opened by lolo32 3

fix for FreeBSD 10.1

I was tried building on FreeBSD 10.1 (amd64). But failed to compile.

So I fixed problems.

System and compiler information:

$ uname -mrs
FreeBSD 10.1-RELEASE-p16 amd64
$ cc -v
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Target: x86_64-unknown-freebsd10.1
Thread model: posix

Modified patch is following (based revision is 2e5a0bf4c0f311d9c6f39ca2518c990df3f36f17):

diff --git a/lzhamcomp/lzham_pthreads_threading.cpp b/lzhamcomp/lzham_pthreads_threading.cpp
index 4dd6c5c..bb2c565 100644
--- a/lzhamcomp/lzham_pthreads_threading.cpp
+++ b/lzhamcomp/lzham_pthreads_threading.cpp
@@ -27,7 +27,7 @@
 #include <process.h>
 #endif

-#if defined(__GNUC__) && !defined(__APPLE__) && !defined(__MINGW32__)
+#if defined(__GNUC__) && !defined(__APPLE__) && !defined(__MINGW32__) && !defined(__FreeBSD__)
 #include <sys/sysinfo.h>
 #endif

@@ -211,7 +211,7 @@ namespace lzham

    uint lzham_get_max_helper_threads()
    {
-#if defined(__APPLE__)
+#if defined(__APPLE__) || defined(__FreeBSD__)
       int num_procs = static_cast<int>(sysconf(_SC_NPROCESSORS_ONLN));
       return (num_procs >= 1) ? (num_procs - 1) : 0;
 #elif (1)
diff --git a/lzhamdecomp/lzham_core.h b/lzhamdecomp/lzham_core.h
index fc6e16f..f4e2d4f 100644
--- a/lzhamdecomp/lzham_core.h
+++ b/lzhamdecomp/lzham_core.h
@@ -241,7 +241,7 @@ const bool c_lzham_big_endian_platform = !c_lzham_little_endian_platform;
 #include <stdlib.h>
 #include <stdio.h>
 #include <math.h>
-#ifndef __APPLE__
+#if !defined(__APPLE__) && !defined(__FreeBSD__)
 #include <malloc.h>
 #endif
 #include <stdarg.h>
diff --git a/lzhamdecomp/lzham_mem.cpp b/lzhamdecomp/lzham_mem.cpp
index 84d8bcd..02f2324 100644
--- a/lzhamdecomp/lzham_mem.cpp
+++ b/lzhamdecomp/lzham_mem.cpp
@@ -6,6 +6,9 @@
    #include <malloc/malloc.h>
 #elif defined(__FreeBSD__) || defined(__NetBSD__)
    #include <malloc_np.h>
+   #if defined(__FreeBSD__)
+      #define malloc(size) aligned_alloc((LZHAM_MIN_ALLOC_ALIGNMENT), (size))
+   #endif
 #else
    #include <malloc.h>
 #endif
diff --git a/lzhamdecomp/lzham_traits.h b/lzhamdecomp/lzham_traits.h
index cb0076c..950c3fd 100644
--- a/lzhamdecomp/lzham_traits.h
+++ b/lzhamdecomp/lzham_traits.h
@@ -67,7 +67,7 @@ namespace lzham
    // Defines type Q as bitwise copyable.
 #define LZHAM_DEFINE_BITWISE_COPYABLE(Q) template<> struct bitwise_copyable<Q> { enum { cFlag = true }; };

-#if defined(__APPLE__) || defined(__FreeBSD__) || defined(__NetBSD__)
+#if defined(__APPLE__) || defined(__NetBSD__)
    #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
 #else
    #define LZHAM_IS_POD(T) __is_pod(T)
diff --git a/lzhamtest/lzhamtest.cpp b/lzhamtest/lzhamtest.cpp
index 3af955a..14bb3d1 100644
--- a/lzhamtest/lzhamtest.cpp
+++ b/lzhamtest/lzhamtest.cpp
@@ -14,7 +14,7 @@
 #include <assert.h>
 #include <memory.h>
 #include <stdarg.h>
-#ifndef __APPLE__
+#if !defined(__APPLE__) && !defined(__FreeBSD__)
 #include <malloc.h>
 #endif
 #include <vector>
@@ -46,6 +46,14 @@
    #define fopen fopen
    #define _fseeki64 fseek
    #define _ftelli64 ftell
+#elif defined(__FreeBSD__)
+   #include <unistd.h>
+   #define Sleep(ms) usleep(ms*1000)
+   #define _aligned_malloc(size, alignment) aligned_alloc(alignment, size)
+   #define _aligned_free free
+   #define fopen fopen
+   #define _fseeki64 fseeko
+   #define _ftelli64 ftello
 #else
    #include <unistd.h>
    #define Sleep(ms) usleep(ms*1000)
diff --git a/lzhamtest/timer.cpp b/lzhamtest/timer.cpp
index 648d87e..affd260 100644
--- a/lzhamtest/timer.cpp
+++ b/lzhamtest/timer.cpp
@@ -27,7 +27,7 @@ inline void query_counter_frequency(timer_ticks *pTicks)
    QueryPerformanceFrequency(reinterpret_cast<LARGE_INTEGER*>(pTicks));
 }
 #elif defined(__GNUC__)
-#ifdef __APPLE__
+#if defined(__APPLE__) || defined(__FreeBSD__)
 #include <sys/time.h>
 #else
 #include <sys/timex.h>

opened by dearblue 2

What is the main purpose of zlib interface?
Hello, could you please tell me what the purpose if zlib interface in lzham is? I can see two possible reasons:

- So that the code using zlib could switch to lzham easily - To allow support zlib binary format (rfc1950)

In my view the first of the two is most practical, as I can't see much advantage from supporting zlib format with a different compression method. Could you please confirm/expand?

The practical reason for this question is that I'm thinking of a .net managed wrapper around lzham and I think that zlib support reason in lzham can affect how the wrapper should function.
opened by AndrewSav 1
Create New Release

As it stands, the latest release is over 7 years old, and there have been many commits since then, including several portability fixes. Any possibility a new release may be created?

opened by partiallyderived 0
Use lzham prefixed max int values

This PR incorporates the changes in #28 which address Linux build failures, namely the use of UINTXX_MAX rather than LZHAM_UINTXX_MAX, where XX is the number of bits, which leads to i.e. error: ‘UINT16_MAX’ was not declared in this scope

opened by partiallyderived 0
Put cmake_minimum_required before project

cmake_minimum_required should appear before PROJECT (see docs) in CMakeLists, so this PR changes that. Having them in the wrong order can cause issues at least for MSVC, similar to what is talked about in this issue.

This PR also uncomments the PROJECT statement in the top-level CMakeLists.

opened by partiallyderived 0
Mac "error: expected unqualified-id"

https://github.com/richgel999/lzham_codec/issues/6 This issue is still happening, because the main branch source wasn't updated.

I can confirm that with latest Xcode:

https://github.com/richgel999/lzham_codec/blob/master/lzhamdecomp/lzham_traits.h#L70 #if defined(APPLE) || defined(NetBSD) #define LZHAM_IS_POD(T) std::__is_pod::__value #else #define LZHAM_IS_POD(T) __is_pod(T) #endif

can just be replaced with #define LZHAM_IS_POD(T) __is_pod(T)

and compiles fine for Mac/iOS/iOS simulator

opened by GregSlazinski 0