Lossless data compression codec with LZMA-like ratios but 1.5x-8x faster decompression speed, C/C++

Overview

LZHAM - Lossless Data Compression Codec

Public Domain (see LICENSE)

LZHAM is a lossless data compression codec written in C/C++ (specifically C++03), with a compression ratio similar to LZMA but with 1.5x-8x faster decompression speed. It officially supports Linux x86/x64, Windows x86/x64, OSX, and iOS, with Android support on the way.

The old alpha version of LZHAM (bitstream incompatible with the v1.x release) is here: https://github.com/richgel999/lzham_alpha

Introduction

LZHAM is a lossless (LZ based) data compression codec optimized for particularly fast decompression at very high compression ratios with a zlib compatible API. It's been developed over a period of 3 years and alpha versions have already shipped in many products. (The alpha is here: https://code.google.com/p/lzham/) LZHAM's decompressor is slower than zlib's, but generally much faster than LZMA's, with a compression ratio that is typically within a few percent of LZMA's and sometimes better.

LZHAM's compressor is intended for offline use, but it is tested alongside the decompressor on mobile devices and is usable on the faster settings.

LZHAM's decompressor currently has a higher cost to initialize than LZMA, so the threshold where LZHAM is typically faster vs. LZMA decompression is between 1000-13,000 of *compressed* output bytes, depending on the platform. It is not a good small block compressor: it likes large (10KB-15KB minimum) blocks.

LZHAM has simple support for patch files (delta compression), but this is a side benefit of its design, not its primary use case. Internally it supports LZ matches up to ~64KB and very large dictionaries (up to .5 GB).

LZHAM may be valuable to you if you compress data offline and distribute it to many customers, care about read/download times, and decompression speed/low CPU+power use are important to you.

I've been profiling LZHAM vs. LZMA and publishing the results on my blog: http://richg42.blogspot.com

Some independent benchmarks of the previous alpha versions: http://heartofcomp.altervista.org/MOC/MOCADE.htm, http://mattmahoney.net/dc/text.html

LZHAM has been integrated into the 7zip archiver (command line and GUI) as a custom codec plugin: http://richg42.blogspot.com/2015/02/lzham-10-integrated-into-7zip-command.html

10GB Benchmark Results

Results with 7zip-LZHAM 9.38 32-bit (64MB dictionary) on Matt Mahoney's 10GB benchmark:

LZHAM (-mx=8): 3,577,047,629 Archive Test Time: 70.652 secs
LZHAM (-mx=9): 3,573,782,721 Archive Test Time: 71.292 secs
LZMA  (-mx=9): 3,560,052,414 Archive Test Time: 223.050 secs
7z .ZIP      : 4,681,291,655 Archive Test Time: 73.304 secs (unzip v6 x64 test time: 61.074 secs)

Most Common Question: So how does it compare to other libs like LZ4?

There is no single compression algorithm that perfectly suites all use cases and practical constraints. LZ4 and LZHAM are tools which lie at completely opposite ends of the spectrum:

  • LZ4: A symmetrical codec with very fast compression and decompression but very low ratios. Its compression ratio is typically less than even zlib's (which uses a 21+ year old algorithm). LZ4 does a good job of trading off a large amount of compression ratio for very fast overall throughput. Usage example: Reading LZMA/LZHAM/etc. compressed data from the network and decompressing it, then caching this data locally on disk using LZ4 to reduce disk usage and decrease future loading times.

  • LZHAM: A very asymmetrical codec with slow compression speed, but with a very competitive (LZMA-like) compression ratio and reasonably fast decompression speeds (slower than zlib, but faster than LZMA). LZHAM trades off a lot of compression throughput for very high ratios and higher decompression throughput relative to other codecs in its ratio class (which is LZMA, which runs circles around LZ4's ratio). Usage example: Compress your product's data once on a build server, distribute it to end users over a slow media like the internet, then decompress it on the end user's device.

How Much Memory Does It Need?

For decompression it's easy to compute:

  • Buffered mode: decomp_mem = dict_size + ~34KB for work tables
  • Unbuffered mode: decomp_mem = ~34KB

I'll be honest here, the compressor is currently an angry beast when it comes to memory. The amount needed depends mostly on the compression level and dict. size. It's approximately (max_probes=128 at level -m4): comp_mem = min(512 * 1024, dict_size / 8) * max_probes * 6 + dict_size * 9 + 22020096

Compression mem usage examples from Windows lzhamtest_x64 (note the equation is pretty off for small dictionary sizes):

  • 32KB: 11MB
  • 128KB: 21MB
  • 512KB: 63MB
  • 1MB: 118MB
  • 8MB: 478MB
  • 64MB: 982MB
  • 128MB: 1558MB
  • 256MB: 2710MB
  • 512MB: 5014MB

Compressed Bitstream Compatibility

v1.0's bitstream format is now locked in place, so any future v1.x releases will be backwards/forward compatible with compressed files written with v1.0. The only thing that could change this are critical bugfixes.

Note LZHAM v1.x bitstreams are NOT backwards compatible with any of the previous alpha versions on Google Code.

Platforms/Compiler Support

LZHAM currently officially supports x86/x64 Linux, iOS, OSX, FreeBSD, and Windows x86/x64. At one time the codec compiled and ran fine on Xbox 360 (PPC, big endian). Android support is coming next. It should be easy to retarget by modifying the macros in lzham_core.h.

LZHAM has optional support for multithreaded compression. It supports gcc built-ins or MSVC intrinsics for atomic ops. For threading, it supports OSX specific Pthreads, generic Pthreads, or Windows API's.

For compilers, I've tested with gcc, clang, and MSVC 2008, 2010, and 2013. In previous alphas I also compiled with TDM-GCC x64.

API

LZHAM supports streaming or memory to memory compression/decompression. See include/lzham.h. LZHAM can be linked statically or dynamically, just study the headers and the lzhamtest project. On Linux/OSX, it's only been tested with static linking so far.

LZHAM also supports a usable subset of the zlib API with extensions, either include/zlib.h or #define LZHAM_DEFINE_ZLIB_API and use include/lzham.h.

Usage Tips

  • Always try to use the smallest dictionary size that makes sense for the file or block you are compressing, i.e. don't use a 128MB dictionary for a 15KB file. The codec doesn't automatically choose for you because in streaming scenarios it has no idea how large the file or block will be.
  • The larger the dictionary, the more RAM is required during compression and decompression. I would avoid using more than 8-16MB dictionaries on iOS.
  • For faster decompression, prefer "unbuffered" decompression mode vs. buffered decompression (avoids a dictionary alloc and extra memcpy()'s), and disable adler-32 checking. Also, use the built-in LZHAM API's, not the zlib-style API's for fastest decompression.
  • Experiment with the "m_table_update_rate" compression/decompression parameter. This setting trades off a small amount of ratio for faster decompression. Note the m_table_update_rate decompression parameter MUST match the setting used during compression (same for the dictionary size). It's up to you to store this info somehow.
  • Avoid using LZHAM on small compressed blocks, where small is 1KB-10KB compressed bytes depending on the platform. LZHAM's decompressor is only faster than LZMA's beyond the small block threshold. Optimizing LZHAM's decompressor to reduce its startup time relative to LZMA is a high priority.
  • For best compression (I've seen up to ~4% better), enable the compressor's "extreme" parser, which is much slower but finds cheaper paths through a much denser parse graph. Note the extreme parser can greatly slow down on files containing large amounts of repeated data/strings, but it is guaranteed to finish.
  • The compressor's m_level parameter can make a big impact on compression speed. Level 0 (LZHAM_COMP_LEVEL_FASTEST) uses a much simpler greedy parser, and the other levels use near-optimal parsing with different heuristic settings.
  • Check out the compressor/decompressor reinit() API's, which are useful if you'll be compressing or decompressing many times. Using the reinit() API's is a lot cheaper than fully initializing/deinitializing the entire codec every time.
  • LZHAM's compressor is no speed demon. It's usually slower than LZMA's, sometimes by a wide (~2x slower or so) margin. In "extreme" parsing mode, it can be many times slower. This codec was designed with offline compression in mind.
  • One significant difference between LZMA and LZHAM is how uncompressible files are handled. LZMA usually expands uncompressible files, and its decompressor can bog down and run extremely slowly on uncompressible data. LZHAM internally detects when each 512KB block is uncompressible and stores these blocks as uncompressed bytes instead. LZHAM's literal decoding is significantly faster than LZMA's, so the more plain literals in the output stream, the faster LZHAM's decompressor runs vs. LZMA's.
  • General advice (applies to LZMA and other codecs too): If you are compressing large amounts of serialized game assets, sort the serialized data by asset type and compress the whole thing as a single large "solid" block of data. Don't compress each individual asset, this will kill your ratio and have a higher decompression startup cost. If you need random access, consider compressing the assets lumped together into groups of a few hundred kilobytes (or whatever) each.
  • LZHAM is a raw codec. It doesn't include any sort of preprocessing: EXE rel to abs jump transformation, audio predictors, etc. That's up to you to do, before compression.

Codec Test App

lzhamtest_x86/x64 is a simple command line test program that uses the LZHAM codec to compress/decompress single files. lzhamtest is not intended as a file archiver or end user tool, it's just a simple testbed.

-- Usage examples:

  • Compress single file "source_filename" to "compressed_filename": lzhamtest_x64 c source_filename compressed_filename

  • Decompress single file "compressed_filename" to "decompressed_filename": lzhamtest_x64 d compressed_filename decompressed_filename

  • Compress single file "source_filename" to "compressed_filename", then verify the compressed file decompresses properly to the source file: lzhamtest_x64 -v c source_filename compressed_filename

  • Recursively compress all files under specified directory and verify that each file decompresses properly: lzhamtest_x64 -v a c:\source_path

-- Options

  • Set dictionary size used during compressed to 1MB (2^20): lzhamtest_x64 -d20 c source_filename compressed_filename

Valid dictionary sizes are [15,26] for x86, and [15,29] for x64. (See LZHAM_MIN_DICT_SIZE_LOG2, etc. defines in include/lzham.h.) The x86 version defaults to 64MB (26), and the x64 version defaults to 256MB (28). I wouldn't recommend setting the dictionary size to 512MB unless your machine has more than 4GB of physical memory.

  • Set compression level to fastest: lzhamtest_x64 -m0 c source_filename compressed_filename

  • Set compression level to uber (the default): lzhamtest_x64 -m4 c source_filename compressed_filename

  • For best possible compression, use -d29 to enable the largest dictionary size (512MB) and the -x option which enables more rigorous (but ~4X slower!) parsing: lzhamtest_x64 -d29 -x -m4 c source_filename compressed_filename

See lzhamtest_x86/x64.exe's help text for more command line parameters.

Compiling LZHAM

  • Linux: Use "cmake ." then "make". The cmake script only supports Linux at the moment. (Sorry, working on build systems is a drag.)
  • OSX/iOS: Use the included XCode project. (NOTE: I haven't merged this over yet. It's coming!)
  • Windows: Use the included VS 2010 project

IMPORTANT: With clang or gcc compile LZHAM with "No strict aliasing" ENABLED: -fno-strict-aliasing

I DO NOT test or develop the codec with strict aliasing:

It might work fine, I don't know yet. This is usually not a problem with MSVC, which defaults to strict aliasing being off.

ANSI C/C++

LZHAM supports compiling as plain vanilla ANSI C/C++. To see how the codec configures itself check out lzham_core.h and search for "LZHAM_ANSI_CPLUSPLUS". All platform specific stuff (unaligned loads, threading, atomic ops, etc.) should be disabled when this macro is defined. Note, the compressor doesn't use threads or atomic operations when built this way so it's going to be pretty slow. (The compressor was built from the ground up to be threaded.)

Known Problems

LZHAM's decompressor is like a drag racer that needs time to get up to speed. LZHAM is not intended or optimized to be used on "small" blocks of data (less than ~10,000 bytes of *compressed* data on desktops, or around 1,000-5,000 on iOS). If your usage case involves calling the codec over and over with tiny blocks then LZMA, LZ4, Deflate, etc. are probably better choices.

The decompressor still takes too long to init vs. LZMA. On iOS the cost is not that bad, but on desktop the cost is high. I have reduced the startup cost vs. the alpha but there's still work to do.

The compressor is slower than I would like, and doesn't scale as well as it could. I added a reinit() method to make it initialize faster, but it's not a speed demon. My focus has been on ratio and decompression speed.

I use tabs=3 spaces, but I think some actual tabs got in the code. I need to run the sources through ClangFormat or whatever.

Special Thanks

Thanks to everyone at the http://encode.ru forums. I read these forums as a lurker before working on LZHAM, and I studied every LZ related post I could get my hands on. Especially anything related to LZ optimal parsing, which still seems like a black art. LZHAM was my way of learning how to implement optimal parsing (and you can see this if you study the progress I made in the early alphas on Google Code).

Also, thanks to Igor Pavlov, the original creator of LZMA and 7zip, for advancing the start of the art in LZ compression.

Comments
  • decompress_memory fails on data compressed with compress_memory

    decompress_memory fails on data compressed with compress_memory

    Maybe I'm doing something wrong, but I feel like this should get me LZHAM_DECOMP_STATUS_SUCCESS not LZHAM_DECOMP_STATUS_FAILED_BAD_CODE:

    #define LOREM_IPSUM                                                     \
      "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vulputate " \
      "lectus nisl, vitae ultricies justo dictum nec. Vestibulum ante ipsum " \
      "primis in faucibus orci luctus et ultrices posuere cubilia Curae; "  \
      "Suspendisse suscipit quam a lectus adipiscing, sed tempor purus "    \
      "cursus. Vivamus id nulla eget elit eleifend molestie. Integer "      \
      "sollicitudin lorem enim, eu eleifend orci facilisis sed. Pellentesque " \
      "sodales luctus enim vel viverra. Cras interdum vel nisl in "         \
      "facilisis. Curabitur sollicitudin tortor vel congue "                \
      "auctor. Suspendisse egestas orci vitae neque placerat blandit.\n"    \
      "\n"                                                                  \
      "Aenean sed nisl ultricies, vulputate lorem a, suscipit nulla. Donec " \
      "egestas volutpat neque a eleifend. Nullam porta semper "             \
      "nunc. Pellentesque adipiscing molestie magna, quis pulvinar metus "  \
      "gravida sit amet. Vestibulum mollis et sapien eu posuere. Quisque "  \
      "tristique dignissim ante et aliquet. Phasellus vulputate condimentum " \
      "nulla in vulputate.\n"                                               \
      "\n"                                                                  \
      "Nullam volutpat tellus at nisi auctor, vitae mattis nibh viverra. Nunc " \
      "vitae lectus tristique, ultrices nibh quis, lobortis elit. Curabitur " \
      "at vestibulum nisi, nec facilisis ante. Nulla pharetra blandit lacus, " \
      "at sodales nulla placerat eget. Nulla congue varius tortor, sit amet " \
      "tempor est mattis nec. Praesent vitae tristique ipsum, rhoncus "     \
      "tristique lorem. Sed et erat tristique ligula accumsan fringilla eu in " \
      "urna. Donec dapibus hendrerit neque nec venenatis. In euismod sapien " \
      "ipsum, auctor consectetur mi dapibus hendrerit.\n"                   \
      "\n"                                                                  \
      "Phasellus sagittis rutrum velit, in sodales nibh imperdiet a. Integer " \
      "vitae arcu blandit nibh laoreet scelerisque eu sit amet eros. Aenean " \
      "odio felis, aliquam in eros at, ornare luctus magna. In semper "     \
      "tincidunt nunc, sollicitudin gravida nunc laoreet eu. Cras eu tempor " \
      "sapien, ut dignissim elit. Proin eleifend arcu tempus, semper erat et, " \
      "accumsan erat. Praesent vulputate diam mi, eget mollis leo "         \
      "pellentesque eget. Aliquam eu tortor posuere, posuere velit sed, "   \
      "suscipit eros. Nam eu leo vitae mauris condimentum lobortis non quis " \
      "mauris. Nulla venenatis fringilla urna nec venenatis. Nam eget velit " \
      "nulla. Proin ut malesuada felis. Suspendisse vitae nunc neque. Donec " \
      "faucibus tempor lacinia. Vivamus ac vulputate sapien, eget lacinia " \
      "nisl.\n"                                                             \
      "\n"                                                                  \
      "Curabitur eu dolor molestie, ullamcorper lorem quis, egestas "       \
      "urna. Suspendisse in arcu sed justo blandit condimentum. Ut auctor, " \
      "sem quis condimentum mattis, est purus pulvinar elit, quis viverra " \
      "nibh metus ac diam. Etiam aliquet est eu dui fermentum consequat. Cras " \
      "auctor diam eget bibendum sagittis. Aenean elementum purus sit amet " \
      "sem euismod, non varius felis dictum. Aliquam tempus pharetra ante a " \
      "sagittis. Curabitur ut urna felis. Etiam sed vulputate nisi. Praesent " \
      "at libero eleifend, sagittis quam a, varius sapien."
    #define LOREM_IPSUM_LENGTH (sizeof(LOREM_IPSUM) - 1)
    
    #include <stdio.h>
    #include <lzham.h>
    #include <stdint.h>
    #include <assert.h>
    
    int main (int argc, char** argv) {
      uint8_t compressed[4096];
      size_t compressed_length;
      uint8_t decompressed[1024 * 1024];
      size_t decompressed_length = sizeof(decompressed);
    
      lzham_compress_status_t comp_res;
      lzham_compress_params comp_params = {
        sizeof(lzham_compress_params),
        LZHAM_MAX_DICT_SIZE_LOG2_X86,
        LZHAM_COMP_LEVEL_DEFAULT,
        LZHAM_DEFAULT_TABLE_UPDATE_RATE,
        -1,
        0,
        0,
        NULL,
        64,
        64
      };
    
      lzham_decompress_status_t decomp_res;
      lzham_decompress_params decomp_params = {
        sizeof(lzham_decompress_params),
        LZHAM_MAX_DICT_SIZE_LOG2_X86,
        LZHAM_DEFAULT_TABLE_UPDATE_RATE,
        0,
        0,
        NULL,
        64,
        16
      };
    
      comp_res = lzham_compress_memory (&comp_params,
                        compressed, &compressed_length,
                        (const lzham_uint8*) LOREM_IPSUM, LOREM_IPSUM_LENGTH,
                        NULL);
      assert (comp_res == LZHAM_COMP_STATUS_SUCCESS);
    
      fprintf (stdout, "Compressed from %zu bytes to %zu bytes.\n", LOREM_IPSUM_LENGTH, compressed_length);
    
      decomp_res = lzham_decompress_memory (&decomp_params,
                        decompressed, &decompressed_length,
                        compressed, compressed_length,
                        NULL);
      fprintf (stdout, "Decompressed status = %d (LZHAM_DECOMP_STATUS_FAILED_BAD_CODE = %d)\n",
           decomp_res, LZHAM_DECOMP_STATUS_FAILED_BAD_CODE);
    
      return 0;
    }
    
    
    opened by nemequ 7
  • decompress does not immediately write data after a flush

    decompress does not immediately write data after a flush

    When I flush a compression stream and send the output to a decompression stream the decompression stream just keeps the data in its buffer instead of immediately making it available (like zlib does), which pretty much destroys the usefulness of flushing.

    Test case:

    #define LOREM_IPSUM                                                     \
      "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vulputate " \
      "lectus nisl, vitae ultricies justo dictum nec. Vestibulum ante ipsum " \
      "primis in faucibus orci luctus et ultrices posuere cubilia Curae; "  \
      "Suspendisse suscipit quam a lectus adipiscing, sed tempor purus "    \
      "cursus. Vivamus id nulla eget elit eleifend molestie. Integer "      \
      "sollicitudin lorem enim, eu eleifend orci facilisis sed. Pellentesque " \
      "sodales luctus enim vel viverra. Cras interdum vel nisl in "         \
      "facilisis. Curabitur sollicitudin tortor vel congue "                \
      "auctor. Suspendisse egestas orci vitae neque placerat blandit.\n"    \
      "\n"                                                                  \
      "Aenean sed nisl ultricies, vulputate lorem a, suscipit nulla. Donec " \
      "egestas volutpat neque a eleifend. Nullam porta semper "             \
      "nunc. Pellentesque adipiscing molestie magna, quis pulvinar metus "  \
      "gravida sit amet. Vestibulum mollis et sapien eu posuere. Quisque "  \
      "tristique dignissim ante et aliquet. Phasellus vulputate condimentum " \
      "nulla in vulputate.\n"                                               \
      "\n"                                                                  \
      "Nullam volutpat tellus at nisi auctor, vitae mattis nibh viverra. Nunc " \
      "vitae lectus tristique, ultrices nibh quis, lobortis elit. Curabitur " \
      "at vestibulum nisi, nec facilisis ante. Nulla pharetra blandit lacus, " \
      "at sodales nulla placerat eget. Nulla congue varius tortor, sit amet " \
      "tempor est mattis nec. Praesent vitae tristique ipsum, rhoncus "     \
      "tristique lorem. Sed et erat tristique ligula accumsan fringilla eu in " \
      "urna. Donec dapibus hendrerit neque nec venenatis. In euismod sapien " \
      "ipsum, auctor consectetur mi dapibus hendrerit.\n"                   \
      "\n"                                                                  \
      "Phasellus sagittis rutrum velit, in sodales nibh imperdiet a. Integer " \
      "vitae arcu blandit nibh laoreet scelerisque eu sit amet eros. Aenean " \
      "odio felis, aliquam in eros at, ornare luctus magna. In semper "     \
      "tincidunt nunc, sollicitudin gravida nunc laoreet eu. Cras eu tempor " \
      "sapien, ut dignissim elit. Proin eleifend arcu tempus, semper erat et, " \
      "accumsan erat. Praesent vulputate diam mi, eget mollis leo "         \
      "pellentesque eget. Aliquam eu tortor posuere, posuere velit sed, "   \
      "suscipit eros. Nam eu leo vitae mauris condimentum lobortis non quis " \
      "mauris. Nulla venenatis fringilla urna nec venenatis. Nam eget velit " \
      "nulla. Proin ut malesuada felis. Suspendisse vitae nunc neque. Donec " \
      "faucibus tempor lacinia. Vivamus ac vulputate sapien, eget lacinia " \
      "nisl.\n"                                                             \
      "\n"                                                                  \
      "Curabitur eu dolor molestie, ullamcorper lorem quis, egestas "       \
      "urna. Suspendisse in arcu sed justo blandit condimentum. Ut auctor, " \
      "sem quis condimentum mattis, est purus pulvinar elit, quis viverra " \
      "nibh metus ac diam. Etiam aliquet est eu dui fermentum consequat. Cras " \
      "auctor diam eget bibendum sagittis. Aenean elementum purus sit amet " \
      "sem euismod, non varius felis dictum. Aliquam tempus pharetra ante a " \
      "sagittis. Curabitur ut urna felis. Etiam sed vulputate nisi. Praesent " \
      "at libero eleifend, sagittis quam a, varius sapien."
    #define LOREM_IPSUM_LENGTH 2725
    
    #include <stdio.h>
    #include <lzham.h>
    #include <stdint.h>
    #include <assert.h>
    
    int main (int argc, char** argv) {
      size_t uncompressed_length = LOREM_IPSUM_LENGTH;
      uint8_t compressed[4096];
      size_t compressed_length = sizeof (compressed);
      uint8_t decompressed[LOREM_IPSUM_LENGTH * 2];
      size_t decompressed_length = sizeof (decompressed);
      size_t pos1, pos2;
    
      lzham_compress_state_ptr comp;
      lzham_compress_status_t comp_res;
      lzham_compress_params comp_params = {
        sizeof(lzham_compress_params),
        LZHAM_MAX_DICT_SIZE_LOG2_X86,
        LZHAM_COMP_LEVEL_DEFAULT,
        LZHAM_DEFAULT_TABLE_UPDATE_RATE,
        -1,
        0,
        0,
        NULL,
        0,
        0
      };
    
      lzham_decompress_state_ptr decomp;
      lzham_decompress_status_t decomp_res;
      lzham_decompress_params decomp_params = {
        sizeof(lzham_decompress_params),
        LZHAM_MAX_DICT_SIZE_LOG2_X86,
        LZHAM_DEFAULT_TABLE_UPDATE_RATE,
        0,
        0,
        NULL,
        0,
        0
      };
    
      const lzham_flush_t flush_or_finish = LZHAM_SYNC_FLUSH;
      // const lzham_flush_t flush_or_finish = LZHAM_FINISH;
    
      comp = lzham_compress_init (&comp_params);
      comp_res = lzham_compress2 (comp,
                      (const lzham_uint8*) LOREM_IPSUM, &uncompressed_length,
                      compressed, &compressed_length,
                      LZHAM_NO_FLUSH);
    
      assert (comp_res == LZHAM_COMP_STATUS_NEEDS_MORE_INPUT);
      assert (uncompressed_length == LOREM_IPSUM_LENGTH);
      assert (compressed_length == 0);
    
      pos1 = 0;
      pos2 = sizeof (compressed) - compressed_length;
      comp_res = lzham_compress2 (comp, (const lzham_uint8*) LOREM_IPSUM, &pos1,
                      compressed + compressed_length, &pos2,
                      flush_or_finish);
    
      compressed_length += pos2;
    
      assert ((comp_res == LZHAM_COMP_STATUS_NOT_FINISHED) || // For LZHAM_SYNC_FLUSH
          (comp_res == LZHAM_COMP_STATUS_SUCCESS));       // For LZHAM_FINISH
      assert (compressed_length > 0);
    
      decomp = lzham_decompress_init (&decomp_params);
    
      pos1 = compressed_length;
      decomp_res = lzham_decompress (decomp,
                     compressed, &pos1,
                     decompressed, &decompressed_length,
                     flush_or_finish == LZHAM_FINISH);
    
      assert (pos1 == compressed_length);
    
      // THIS FAILS
      assert (decompressed_length != 0);
      assert (decompressed_length == LOREM_IPSUM_LENGTH);
    
      lzham_compress_deinit (comp);
      lzham_decompress_deinit (decomp);
    
      return 0;
    }
    
    opened by nemequ 6
  • Visual Studio 2013 assert

    Visual Studio 2013 assert

    VS2010 runs fine, but VS2013 asserts during generate_codes.

    The reason seems to be that in total_bits(uint v) the call to _BitScanReverse behaves differently in Visual Studio 2013 to Visual Studio 2010. When the variable v is zero, _BitScanReverse(&l, v) will set l to 0xcccccccc and return zero. In VS2010 l does not get changed and so is still zero.

    As a workaround fix I changed: if (_BitScanReverse(&l, v)) { l++; }

    to: if (_BitScanReverse(&l, v)) { l++; } else { l = 0; }

    This gets past this error, although I've yet to fully test the rest of the lib so I don't know if there are any other issues.

    opened by Muzza 6
  • Compiling for Mac OSX 10.10 Yosemite failed:

    Compiling for Mac OSX 10.10 Yosemite failed: "error: expected unqualified-id"

    I'm trying to compile it, revision 4ed83a3, on Mac OSX, and I got the error "error: expected unqualified-id"

    Here are the steps I done:

    $ mkdir build
    $ cd build
    $ cmake ..
    $ make
    

    I got an error when the first file is being compiled. Compilation message following:

    [  5%] Building CXX object lzhamdecomp/CMakeFiles/lzhamdecomp.dir/lzham_assert.cpp.o
    In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
    In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:260:
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:132:15: error: expected unqualified-id
             if ( LZHAM_HAS_DESTRUCTOR(T) )
                  ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:82:64: note: expanded from macro 'LZHAM_HAS_DESTRUCTOR'
    #define LZHAM_HAS_DESTRUCTOR(T) ((!scalar_type<T>::cFlag) && (!LZHAM_IS_POD(T)))
                                                                   ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
       #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                    ^
    In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
    In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:56:14: error: expected unqualified-id
             if (LZHAM_IS_BITWISE_COPYABLE(T))
                 ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
    #define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                     ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
       #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                    ^
    In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
    In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:101:14: error: expected unqualified-id
             if (LZHAM_IS_BITWISE_COPYABLE(T))
                 ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
    #define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                     ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
       #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                    ^
    In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
    In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:257:17: error: expected unqualified-id
                if (LZHAM_IS_BITWISE_COPYABLE(T))
                    ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
    #define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                     ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
       #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                    ^
    In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
    In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:274:14: error: expected unqualified-id
             if (LZHAM_IS_BITWISE_COPYABLE(T))
                 ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
    #define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                     ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
       #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                    ^
    In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
    In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:323:14: error: expected unqualified-id
             if (LZHAM_IS_BITWISE_COPYABLE(T))
                 ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
    #define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                     ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
       #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                    ^
    In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
    In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:573:14: error: expected unqualified-id
                (LZHAM_IS_BITWISE_MOVABLE(T) || (is_vector<T>::cFlag)) ? NULL : object_mover, nofail);
                 ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:80:38: note: expanded from macro 'LZHAM_IS_BITWISE_MOVABLE'
    #define LZHAM_IS_BITWISE_MOVABLE(T) (LZHAM_IS_BITWISE_COPYABLE(T) || (bitwise_movable<T>::cFlag))
                                         ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:78:98: note: expanded from macro 'LZHAM_IS_BITWISE_COPYABLE'
    #define LZHAM_IS_BITWISE_COPYABLE(T) ((scalar_type<T>::cFlag) || (bitwise_copyable<T>::cFlag) || LZHAM_IS_POD(T))
                                                                                                     ^
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_traits.h:71:33: note: expanded from macro 'LZHAM_IS_POD'
       #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
                                    ^
    In file included from /lzham-git/lzhamdecomp/lzham_assert.cpp:3:
    In file included from /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_core.h:264:
    /lzham-git/lzhamdecomp/../lzhamdecomp/lzham_vector.h:569:81: warning: unused parameter 'nofail' [-Wunused-parameter]
          inline bool increase_capacity(uint min_new_capacity, bool grow_hint, bool nofail = false)
                                                                                    ^
    1 warning and 7 errors generated.
    make[2]: *** [lzhamdecomp/CMakeFiles/lzhamdecomp.dir/lzham_assert.cpp.o] Error 1
    make[1]: *** [lzhamdecomp/CMakeFiles/lzhamdecomp.dir/all] Error 2
    make: *** [all] Error 2
    

    Here is my CMake cache, in advanced mode (all default or auto detected):

    BUILD_SHARED_LIBS:BOOL=ON
    BUILD_X64:BOOL=ON
    CMAKE_AR:FILEPATH=/usr/bin/ar
    CMAKE_BUILD_TYPE:STRING=
    CMAKE_COLOR_MAKEFILE:BOOL=ON
    CMAKE_CXX_COMPILER:FILEPATH=/usr/bin/c++
    CMAKE_CXX_FLAGS:STRING=
    CMAKE_CXX_FLAGS_DEBUG:STRING=-g
    CMAKE_CXX_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
    CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
    CMAKE_CXX_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG
    CMAKE_C_COMPILER:FILEPATH=/usr/bin/cc
    CMAKE_C_FLAGS:STRING=
    CMAKE_C_FLAGS_DEBUG:STRING=-g
    CMAKE_C_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
    CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
    CMAKE_C_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG
    CMAKE_EXE_LINKER_FLAGS:STRING= 
    CMAKE_EXE_LINKER_FLAGS_DEBUG:STRING=
    CMAKE_EXE_LINKER_FLAGS_MINSIZEREL:STRING=
    CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING=
    CMAKE_EXE_LINKER_FLAGS_RELWITHDEBINFO:STRING=
    CMAKE_EXPORT_COMPILE_COMMANDS:BOOL=OFF
    CMAKE_INSTALL_NAME_TOOL:FILEPATH=/usr/bin/install_name_tool
    CMAKE_INSTALL_PREFIX:PATH=/usr/local
    CMAKE_LINKER:FILEPATH=/usr/bin/ld
    CMAKE_MAKE_PROGRAM:FILEPATH=/usr/bin/make
    CMAKE_MODULE_LINKER_FLAGS:STRING= 
    CMAKE_MODULE_LINKER_FLAGS_DEBUG:STRING=
    CMAKE_MODULE_LINKER_FLAGS_MINSIZEREL:STRING=
    CMAKE_MODULE_LINKER_FLAGS_RELEASE:STRING=
    CMAKE_MODULE_LINKER_FLAGS_RELWITHDEBINFO:STRING=
    CMAKE_NM:FILEPATH=/usr/bin/nm
    CMAKE_OBJCOPY:FILEPATH=CMAKE_OBJCOPY-NOTFOUND
    CMAKE_OBJDUMP:FILEPATH=CMAKE_OBJDUMP-NOTFOUND
    CMAKE_OSX_ARCHITECTURES:STRING=
    CMAKE_OSX_DEPLOYMENT_TARGET:STRING=
    CMAKE_OSX_SYSROOT:STRING=
    CMAKE_RANLIB:FILEPATH=/usr/bin/ranlib
    CMAKE_SHARED_LINKER_FLAGS:STRING= 
    CMAKE_SHARED_LINKER_FLAGS_DEBUG:STRING=
    CMAKE_SHARED_LINKER_FLAGS_MINSIZEREL:STRING=
    CMAKE_SHARED_LINKER_FLAGS_RELEASE:STRING=
    CMAKE_SHARED_LINKER_FLAGS_RELWITHDEBINFO:STRING=
    CMAKE_SKIP_INSTALL_RPATH:BOOL=OFF
    CMAKE_SKIP_RPATH:BOOL=OFF
    CMAKE_STATIC_LINKER_FLAGS:STRING=
    CMAKE_STATIC_LINKER_FLAGS_DEBUG:STRING=
    CMAKE_STATIC_LINKER_FLAGS_MINSIZEREL:STRING=
    CMAKE_STATIC_LINKER_FLAGS_RELEASE:STRING=
    CMAKE_STATIC_LINKER_FLAGS_RELWITHDEBINFO:STRING=
    CMAKE_STRIP:FILEPATH=/usr/bin/strip
    CMAKE_USE_RELATIVE_PATHS:BOOL=OFF
    CMAKE_VERBOSE_MAKEFILE:BOOL=OFF
    

    Additional informations:

    • cc version:
    Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
    Target: x86_64-apple-darwin14.1.0
    Thread model: posix
    
    • c++ version:
    Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
    Target: x86_64-apple-darwin14.1.0
    Thread model: posix
    
    opened by lolo32 3
  • fix for FreeBSD 10.1

    fix for FreeBSD 10.1

    I was tried building on FreeBSD 10.1 (amd64). But failed to compile.

    So I fixed problems.

    System and compiler information:

    $ uname -mrs
    FreeBSD 10.1-RELEASE-p16 amd64
    $ cc -v
    FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
    Target: x86_64-unknown-freebsd10.1
    Thread model: posix
    

    Modified patch is following (based revision is 2e5a0bf4c0f311d9c6f39ca2518c990df3f36f17):

    diff --git a/lzhamcomp/lzham_pthreads_threading.cpp b/lzhamcomp/lzham_pthreads_threading.cpp
    index 4dd6c5c..bb2c565 100644
    --- a/lzhamcomp/lzham_pthreads_threading.cpp
    +++ b/lzhamcomp/lzham_pthreads_threading.cpp
    @@ -27,7 +27,7 @@
     #include <process.h>
     #endif
    
    -#if defined(__GNUC__) && !defined(__APPLE__) && !defined(__MINGW32__)
    +#if defined(__GNUC__) && !defined(__APPLE__) && !defined(__MINGW32__) && !defined(__FreeBSD__)
     #include <sys/sysinfo.h>
     #endif
    
    @@ -211,7 +211,7 @@ namespace lzham
    
        uint lzham_get_max_helper_threads()
        {
    -#if defined(__APPLE__)
    +#if defined(__APPLE__) || defined(__FreeBSD__)
           int num_procs = static_cast<int>(sysconf(_SC_NPROCESSORS_ONLN));
           return (num_procs >= 1) ? (num_procs - 1) : 0;
     #elif (1)
    diff --git a/lzhamdecomp/lzham_core.h b/lzhamdecomp/lzham_core.h
    index fc6e16f..f4e2d4f 100644
    --- a/lzhamdecomp/lzham_core.h
    +++ b/lzhamdecomp/lzham_core.h
    @@ -241,7 +241,7 @@ const bool c_lzham_big_endian_platform = !c_lzham_little_endian_platform;
     #include <stdlib.h>
     #include <stdio.h>
     #include <math.h>
    -#ifndef __APPLE__
    +#if !defined(__APPLE__) && !defined(__FreeBSD__)
     #include <malloc.h>
     #endif
     #include <stdarg.h>
    diff --git a/lzhamdecomp/lzham_mem.cpp b/lzhamdecomp/lzham_mem.cpp
    index 84d8bcd..02f2324 100644
    --- a/lzhamdecomp/lzham_mem.cpp
    +++ b/lzhamdecomp/lzham_mem.cpp
    @@ -6,6 +6,9 @@
        #include <malloc/malloc.h>
     #elif defined(__FreeBSD__) || defined(__NetBSD__)
        #include <malloc_np.h>
    +   #if defined(__FreeBSD__)
    +      #define malloc(size) aligned_alloc((LZHAM_MIN_ALLOC_ALIGNMENT), (size))
    +   #endif
     #else
        #include <malloc.h>
     #endif
    diff --git a/lzhamdecomp/lzham_traits.h b/lzhamdecomp/lzham_traits.h
    index cb0076c..950c3fd 100644
    --- a/lzhamdecomp/lzham_traits.h
    +++ b/lzhamdecomp/lzham_traits.h
    @@ -67,7 +67,7 @@ namespace lzham
        // Defines type Q as bitwise copyable.
     #define LZHAM_DEFINE_BITWISE_COPYABLE(Q) template<> struct bitwise_copyable<Q> { enum { cFlag = true }; };
    
    -#if defined(__APPLE__) || defined(__FreeBSD__) || defined(__NetBSD__)
    +#if defined(__APPLE__) || defined(__NetBSD__)
        #define LZHAM_IS_POD(T) std::__is_pod<T>::__value
     #else
        #define LZHAM_IS_POD(T) __is_pod(T)
    diff --git a/lzhamtest/lzhamtest.cpp b/lzhamtest/lzhamtest.cpp
    index 3af955a..14bb3d1 100644
    --- a/lzhamtest/lzhamtest.cpp
    +++ b/lzhamtest/lzhamtest.cpp
    @@ -14,7 +14,7 @@
     #include <assert.h>
     #include <memory.h>
     #include <stdarg.h>
    -#ifndef __APPLE__
    +#if !defined(__APPLE__) && !defined(__FreeBSD__)
     #include <malloc.h>
     #endif
     #include <vector>
    @@ -46,6 +46,14 @@
        #define fopen fopen
        #define _fseeki64 fseek
        #define _ftelli64 ftell
    +#elif defined(__FreeBSD__)
    +   #include <unistd.h>
    +   #define Sleep(ms) usleep(ms*1000)
    +   #define _aligned_malloc(size, alignment) aligned_alloc(alignment, size)
    +   #define _aligned_free free
    +   #define fopen fopen
    +   #define _fseeki64 fseeko
    +   #define _ftelli64 ftello
     #else
        #include <unistd.h>
        #define Sleep(ms) usleep(ms*1000)
    diff --git a/lzhamtest/timer.cpp b/lzhamtest/timer.cpp
    index 648d87e..affd260 100644
    --- a/lzhamtest/timer.cpp
    +++ b/lzhamtest/timer.cpp
    @@ -27,7 +27,7 @@ inline void query_counter_frequency(timer_ticks *pTicks)
        QueryPerformanceFrequency(reinterpret_cast<LARGE_INTEGER*>(pTicks));
     }
     #elif defined(__GNUC__)
    -#ifdef __APPLE__
    +#if defined(__APPLE__) || defined(__FreeBSD__)
     #include <sys/time.h>
     #else
     #include <sys/timex.h>
    
    opened by dearblue 2
  • What is the main purpose of zlib interface?

    What is the main purpose of zlib interface?

    Hello, could you please tell me what the purpose if zlib interface in lzham is? I can see two possible reasons:

    - So that the code using zlib could switch to lzham easily 
    - To allow support zlib binary format (rfc1950)
    

    In my view the first of the two is most practical, as I can't see much advantage from supporting zlib format with a different compression method. Could you please confirm/expand?

    The practical reason for this question is that I'm thinking of a .net managed wrapper around lzham and I think that zlib support reason in lzham can affect how the wrapper should function.

    opened by AndrewSav 1
  • Create New Release

    Create New Release

    As it stands, the latest release is over 7 years old, and there have been many commits since then, including several portability fixes. Any possibility a new release may be created?

    opened by partiallyderived 0
  • Use lzham prefixed max int values

    Use lzham prefixed max int values

    This PR incorporates the changes in #28 which address Linux build failures, namely the use of UINTXX_MAX rather than LZHAM_UINTXX_MAX, where XX is the number of bits, which leads to i.e. error: ‘UINT16_MAX’ was not declared in this scope

    opened by partiallyderived 0
  • Put cmake_minimum_required before project

    Put cmake_minimum_required before project

    cmake_minimum_required should appear before PROJECT (see docs) in CMakeLists, so this PR changes that. Having them in the wrong order can cause issues at least for MSVC, similar to what is talked about in this issue.

    This PR also uncomments the PROJECT statement in the top-level CMakeLists.

    opened by partiallyderived 0
  • Mac

    Mac "error: expected unqualified-id"

    https://github.com/richgel999/lzham_codec/issues/6 This issue is still happening, because the main branch source wasn't updated.

    I can confirm that with latest Xcode:

    https://github.com/richgel999/lzham_codec/blob/master/lzhamdecomp/lzham_traits.h#L70 #if defined(APPLE) || defined(NetBSD) #define LZHAM_IS_POD(T) std::__is_pod::__value #else #define LZHAM_IS_POD(T) __is_pod(T) #endif

    can just be replaced with #define LZHAM_IS_POD(T) __is_pod(T)

    and compiles fine for Mac/iOS/iOS simulator

    opened by GregSlazinski 0
Owner
Rich Geldreich
Rich Geldreich
A bespoke sample compression codec for 64k intros

pulsejet A bespoke sample compression codec for 64K intros codec pulsejet lifts a lot of ideas from Opus, and more specifically, its CELT layer, which

logicoma 34 Jul 25, 2022
A variation CredBandit that uses compression to reduce the size of the data that must be trasnmitted.

compressedCredBandit compressedCredBandit is a modified version of anthemtotheego's proof of concept Beacon Object File (BOF). This version does all t

Conor Richard 18 Sep 22, 2022
Data compression utility for minimalist demoscene programs.

bzpack Bzpack is a data compression utility which targets retrocomputing and demoscene enthusiasts. Given the artificially imposed size limits on prog

Milos Bazelides 20 Jul 27, 2022
Brotli compression format

SECURITY NOTE Please consider updating brotli to version 1.0.9 (latest). Version 1.0.9 contains a fix to "integer overflow" problem. This happens when

Google 11.7k Dec 29, 2022
Multi-format archive and compression library

Welcome to libarchive! The libarchive project develops a portable, efficient C library that can read and write streaming archives in a variety of form

null 1.9k Jan 8, 2023
Extremely Fast Compression algorithm

LZ4 - Extremely fast compression LZ4 is lossless compression algorithm, providing compression speed > 500 MB/s per core, scalable with multi-cores CPU

lz4 7.9k Dec 31, 2022
LZFSE compression library and command line tool

LZFSE This is a reference C implementation of the LZFSE compressor introduced in the Compression library with OS X 10.11 and iOS 9. LZFSE is a Lempel-

null 1.7k Jan 4, 2023
Small strings compression library

SMAZ - compression for very small strings ----------------------------------------- Smaz is a simple compression library suitable for compressing ver

Salvatore Sanfilippo 1k Dec 28, 2022
Zstandard - Fast real-time compression algorithm

Zstandard, or zstd as short version, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better comp

Facebook 19.2k Jan 1, 2023
A massively spiffy yet delicately unobtrusive compression library.

ZLIB DATA COMPRESSION LIBRARY zlib 1.2.11 is a general purpose data compression library. All the code is thread safe. The data format used by the z

Mark Adler 4.1k Dec 30, 2022
A simple C library implementing the compression algorithm for isosceles triangles.

orvaenting Summary A simple C library implementing the compression algorithm for isosceles triangles. License This project's license is GPL 2 (as of J

Kevin Matthes 0 Apr 1, 2022
gzip (GNU zip) is a compression utility designed to be a replacement for 'compress'

gzip (GNU zip) is a compression utility designed to be a replacement for 'compress'

ACM at UCLA 8 Nov 6, 2022
Advanced DXTc texture compression and transcoding library

crunch/crnlib v1.04 - Advanced DXTn texture compression library Public Domain - Please see license.txt. Portions of this software make use of public d

null 775 Dec 26, 2022
Slow5tools is a toolkit for converting (FAST5 <-> SLOW5), compressing, viewing, indexing and manipulating data in SLOW5 format.

slow5tools Slow5tools is a simple toolkit for converting (FAST5 <-> SLOW5), compressing, viewing, indexing and manipulating data in SLOW5 format. Abou

Hasindu Gamaarachchi 57 Dec 2, 2022
Simple data packing library (written in C99)

Features Compressed file pack creation Runtime file pack reading Supported operating systems Ubuntu MacOS Windows Build requirements C99 compiler CMak

Nikita Fediuchin 3 Feb 25, 2022
Experimental data compressor for 8-bit computers and low-end platforms

ZX5 (experimental) ZX5 is an experimental data compressor derived from ZX0, similarly targeted for low-end platforms, including 8-bit computers like t

Einar Saukas 9 Apr 14, 2022
Przemyslaw Skibinski 579 Jan 8, 2023
Analysing and implementation of lossless data compression techniques like Huffman encoding and LZW was conducted along with JPEG lossy compression technique based on discrete cosine transform (DCT) for Image compression.

PROJECT FILE COMPRESSION ALGORITHMS - Huffman compression LZW compression DCT Aim of the project - Implement above mentioned compression algorithms an

null 1 Dec 14, 2021
A drop-in replacement for std::list with 293% faster insertion, 57% faster erasure, 17% faster iteration and 77% faster sorting on average. 20-24% speed increase in use-case testing.

plf_list A drop-in replacement for std::list with (on average): 293% faster insertion 57% faster erasure 17% faster iteration 77% faster sorting 70% f

Matt Bentley 117 Nov 8, 2022
A drop-in replacement for std::list with 293% faster insertion, 57% faster erasure, 17% faster iteration and 77% faster sorting on average. 20-24% speed increase in use-case testing.

plf::list A drop-in replacement for std::list with (on average): 293% faster insertion 57% faster erasure 17% faster iteration 77% faster sorting 70%

Matt Bentley 117 Nov 8, 2022