Compression abstraction library and utilities

Related tags

Compilers squash
Overview
Squash - Compresion Abstraction Library
<https://quixdb.github.io/squash>
======================================================================

Squash is an abstraction library which provides a single API to access
many compression libraries, allowing applications a great deal of
flexibility when choosing a compression algorithm, or allowing a
choice between several of them.

Because Squash provides a single API, any algorithms which Squash has
a plugin to support will be useable from any language for which Squash
provides bindings.

The actual integration with individual compression libraries is done
through plugins which can be installed separately from Squash itself
and are not loaded until they are required. This allows Squash
consumers to utilize a great many compression algorithms without
rewriting code or unnecessary bloat.

For documentation, support, and other details, please see
<http://quixdb.github.io/squash/>.
Comments
  • Consider adding support for libdeflate

    Consider adding support for libdeflate

    These alternative implementations of DEFLATE should help the DEFLATE format reach into different areas of the compression spectrum.

    SLZ (http://git.1wt.eu/git/libslz.git/) claims to be a much faster compressor zopfli (https://github.com/google/zopfli) tries to push deflate to it's compression ratio limit libdeflate (https://github.com/ebiggers/libdeflate) claims faster compression and decompression

    enhancement plugins help-wanted bite-sized plugin-request 
    opened by jrmuizel 18
  • Strange brotli errors

    Strange brotli errors

    I am seeing some very strange errors when compressing buffers with the brotli plugin.

    Initially I had an "Operation failed" error with one input that was exactly 32768 bytes long (I can provide the exact bytes if needed). Then I updated to the latest version and now it fails immediately for any input.

    I tried pinning the error down to one specific change either in squash or brotli but it got more complicated:

    I am not sure if this is at all a squash bug or a bug in Brotli.

    opened by dswd 16
  • Provide wide character versions of functions which use strings

    Provide wide character versions of functions which use strings

    Squash doesn't actually use strings too heavily, but there are still a lot of places where we apparently need to expose wchar_t versions to be nice to people writing internationalized Windows programs.

    • Option parsing
    • squash_file_open*
    • squash_file_printf
    • Every convenience wrapper which has a char* codec argument

    Obviously that last one is the major issue. These are just supposed to be convenience wrappers for programmers, but if we have to provide wide-char versions it is a major headache… It may be better to just remove them and move the SquashCodec* codec versions to the short ones. It's not like it's particularly difficult to call squash_get_codec once, it encourages people to do the right thing instead of wasting cycles on implicit lookups from passing around codecs as strings. Need to make a determination soon, though, since it would be a pretty big API/ABI break.

    For functions which will have wchar_t versions, we should provide text-generic mappings.

    core windows 
    opened by nemequ 10
  • Drop glib dependency for unit tests

    Drop glib dependency for unit tests

    This would make Windows, where things are more complicated than just apt-get install libglib2.0-dev or dnf install glib2-devel, much easier to deal with.

    tinycthread has a very rudimentary test runner we could steal (unless I can find something better). Some modifications would probably be necessary, but it's doable.

    enhancement core unit-testing windows 
    opened by nemequ 9
  • Run unit tests in AppVeyor

    Run unit tests in AppVeyor

    There is a nuget package for glib, so it shouldn't be too hard to get the unit tests running. Getting them to run successfully will, of course, be more work.

    unit-testing windows 
    opened by nemequ 9
  • Working on MSVC support

    Working on MSVC support

    As discussed in #86.

    The goal so far is just to get it to compile and run, here are some notes:

    • Most of the plugins included strings.h, but only zpaq appeared to use it. I removed it, and added a define to zpaq to use the equivalent MSVC function.
    • I put defines to map functions from strings.h to MSVC functions into internal.h
    • MSVC does not come with getopt, I don't know if you would prefer to include a getopt port, or write your own command line handling. For starters I used a getopt-like parser I wrote a while back called parg.
    • The following plugins did not compile out of the box: gipfeli, brotli, density, pithy, snappy, zling, and zstd.
    opened by jibsen 8
  • Feature : Set plugin search path (esp for MSVC)

    Feature : Set plugin search path (esp for MSVC)

    Apologies if these types of questions are better referred to a forum.

    Where can plugins be located? (especially on windows)

    I'm reading:

        plugin_file_name = squash_strdup_printf ("%s/%ssquash%s-plugin-%s%s", plugin->directory, SQUASH_SHARED_LIBRARY_PREFIX, SQUASH_VERSION_API, plugin->name, SQUASH_SHARED_LIBRARY_SUFFIX);
    

    from plugin.c

    And in config.h:

    #define SQUASH_INSTALL_PREFIX "C:/Program Files/squash"
    #define SQUASH_PLUGIN_DIR "C:/Program Files/squash/lib/squash/0.8/plugins"
    #define SQUASH_SEARCH_PATH "C:/Program Files/squash/lib/squash/0.8/plugins"
    #define SQUASH_SHARED_LIBRARY_PREFIX ""
    #define SQUASH_SHARED_LIBRARY_SUFFIX ".dll"
    #define SQUASH_BUILD_TYPE ""
    

    (side question : is this config.h written by c-make?)

    I'm finding that the Install script (which halted when it got to density) put the dll's into that SQUASH_PLUGIN_DIR with the correct hierarchy structure. So those plugins appear by default.

    Q1) Does squash search for plugins at this location only (i.e. not locally)?

    Q2) Can the user specify a search path for plugins?

    I can see:

    SquashPlugin*
    squash_plugin_new (char* name, char* directory, SquashContext* context) {
      SquashPlugin* plugin = (SquashPlugin*) squash_malloc (sizeof (SquashPlugin));
    

    in plugin.c

    and

    squash_context_add_plugin (SquashContext* context, char* name, char* directory) {
    

    in context.c (which calls squash_plugin_new)

    These functions are not exposed in the headers.

    Q3) Is there a good way to publicly operate this functions?

    I've tried manually copying plugins into this folder: When I call squash_foreach_codec I get

    • brieflz
    • brotli
    • bsc
    • bzip2
    • copy
    • crush
    • density (not valid)

    These plugins (except density) were copied by the INSTALL project to the Program Files location. However density dll is missing, and if I add another plugin manually:

    image

    Then it is not shown in the list.

    Q4) Is this plugin list configured somewhere? (rather than searching the directory)

    Ideally I want to be able to keep my squash and plugin dll's next to my application (to make the application portable, but also to make the development environment portable) Ideally something like this: image (or perhaps in a subdir plugins/squash)

    Thank you

    opened by elliotwoods 7
  • Set up CI for OS X

    Set up CI for OS X

    Squash should work on OS X. It used to. Unfortunately I don't have OS X, so until someone sets up CI for OS X it's hard to say whether or not it currently does.

    Travis supports OS X, but last time I checked you had to choose between Linux and OS X, and there was no way for the two to coexist. If that has changed we could just use Travis, if not the only other one I am aware of is Hosted-CI.

    Since I don't have OS X and know basically nothing about developing for it or with it, this is going to have to be done by someone else.

    enhancement core help-wanted bite-sized 
    opened by nemequ 7
  • Possible Feature : doxygen on headers rather than source?

    Possible Feature : doxygen on headers rather than source?

    it seems that if the doxygen comments (which are really well made) were in the headers, then IDE's could use it to provide more assistance within the text editor

    or maybe there's something i'm not seeing here? (if so please mark this as closed :)

    opened by elliotwoods 6
  • Update the brotli submodule to its latest version.

    Update the brotli submodule to its latest version.

    The brotli encoder got a new faster compression mode, quality 0, which is about 2x faster than the fastest mode in the previous version, and the quality 1 mode also got considerably faster.

    Because of this, it might be worth re-running the brotli benchmarks and updating the benchmark results.

    opened by szabadka 6
  • Add (and use) macro for internal symbols

    Add (and use) macro for internal symbols

    Right now we basically depend on -fvisibility=hidden being set in order to restrict symbol visibility by default. This is fine for GCC (and clang, and icc, etc.), but I'm pretty sure MSVC doesn't know about it. It would probably be wise to add a SQUASH_INTERNAL macro (defined to __attribute__ ((visibility ("hidden")))/__declspec(dllimport), and use that for all our non-static internal functions to make sure internals symbols are hidden more reliably.

    That said, we can't really completely replace -fvisibility=hidden since we're unlikely to be able to modify all plugins to explicitly specify visibility.

    bug windows 
    opened by nemequ 6
  • Add zopfli (gzip) compressor

    Add zopfli (gzip) compressor

    zopfli is gzip-compatible, at least at the command line level. It does not include a decompressor - its binaries delegate to zlib for that - but its compressor is interesting in that it achieves noticeably better compression ratios than gzip --best, at the cost of being horrifically slow, requiring huge amounts of memory, and not supporting streaming. It's a useful option for people who absolutely, positively need the wide compatibility of gzip and expect to have something be read far more often than it's written, e.g. static web assets. The compressed files it produces are also often a bit faster to decompress, depending on your hardware.

    opened by adam-azarchs 0
  • Update ZStandard, Increase Range

    Update ZStandard, Increase Range

    The ZStandard submodule is quite old, the newer module should be 10%+ faster (at least with decoding).

    The compression level "range" has been expanded, from -5 to 22. The line to change is here, but I don't program in C so I'll leave the PR to someone else.

    opened by indolering 1
  • Update BriefLZ to 1.3.0

    Update BriefLZ to 1.3.0

    BriefLZ has gained an interface with compression levels 1-9 and a (sometimes very) slow optimal level. I put the optimal parsing as a separate boolean option.

    opened by jibsen 0
Releases(v0.7.0)
Multi-format archive and compression library

Welcome to libarchive! The libarchive project develops a portable, efficient C library that can read and write streaming archives in a variety of form

null 1.9k Nov 28, 2022
Superfast compression library

DENSITY Superfast compression library DENSITY is a free C99, open-source, BSD licensed compression library. It is focused on high-speed compression, a

Centaurean 984 Nov 24, 2022
data compression library for embedded/real-time systems

heatshrink A data compression/decompression library for embedded/real-time systems. Key Features: Low memory usage (as low as 50 bytes) It is useful f

Atomic Object 1.1k Nov 21, 2022
Small strings compression library

SMAZ - compression for very small strings ----------------------------------------- Smaz is a simple compression library suitable for compressing ver

Salvatore Sanfilippo 1k Nov 24, 2022
Brotli compression format

SECURITY NOTE Please consider updating brotli to version 1.0.9 (latest). Version 1.0.9 contains a fix to "integer overflow" problem. This happens when

Google 11.7k Nov 25, 2022
Heavily optimized zlib compression algorithm

Optimized version of longest_match for zlib Summary Fast zlib longest_match function. Produces slightly smaller compressed files for significantly fas

Konstantin Nosov 123 Oct 29, 2022
Fastest Integer Compression

TurboPFor: Fastest Integer Compression TurboPFor: The new synonym for "integer compression" ?? (2019.11) ALL functions now available for 64 bits ARMv8

powturbo 643 Nov 28, 2022
is a c++20 compile and runtime Struct Reflections header only library.

is a c++20 compile and runtime Struct Reflections header only library. It allows you to iterate over aggregate type's member variables.

RedSkittleFox 4 Apr 18, 2022
A simple C library for compressing lists of integers using binary packing

The SIMDComp library A simple C library for compressing lists of integers using binary packing and SIMD instructions. The assumption is either that yo

Daniel Lemire 406 Nov 21, 2022
A portable, simple zip library written in C

A portable (OSX/Linux/Windows), simple zip library written in C This is done by hacking awesome miniz library and layering functions on top of the min

Kuba Podgórski 1k Nov 27, 2022
Compile and execute C "scripts" in one go!

c "There isn't much that's special about C. That's one of the reasons why it's fast." I love C for its raw speed (although it does have its drawbacks)

Ryan Jacobs 2k Nov 26, 2022
distributed builds for C, C++ and Objective C

distcc -- a free distributed C/C++ compiler system by Martin Pool Current Documents: https://distcc.github.io/ Formally http://distcc.org/ "pump" func

distcc 1.8k Nov 20, 2022
Roaring bitmaps in C (and C++)

CRoaring Portable Roaring bitmaps in C (and C++) with full support for your favorite compiler (GNU GCC, LLVM's clang, Visual Studio). Included in the

Roaring bitmaps: A better compressed bitset 1.1k Nov 23, 2022
New generation entropy codecs : Finite State Entropy and Huff0

New Generation Entropy coders This library proposes two high speed entropy coders : Huff0, a Huffman codec designed for modern CPU, featuring OoO (Out

Yann Collet 1.1k Nov 15, 2022
Easing the task of comparing code generated by cc65, vbcc, and 6502-gcc

6502 C compilers benchmark Easing the way to compare code generated by cc65, 6502-gcc, vbcc, and KickC. This repository contains scripts to: Compile t

Sylvain Gadrat 17 Sep 4, 2022
Secure ECC-based DID intersection in Go, Java and C.

SecureUnionID Secure ECC-based DID intersection. ABSTRACT This project is used to protect device ID using Elliptic Curve Cryptography algorithm. The d

Volcengine 19 Aug 22, 2022
nanoc is a tiny subset of C and a tiny compiler that targets 32-bit x86 machines.

nanoc is a tiny subset of C and a tiny compiler that targets 32-bit x86 machines. Tiny? The only types are: int (32-bit signed integer) char (8-

Ajay Tatachar 18 Oct 23, 2022
Smaller C is a simple and small single-pass C compiler

Smaller C is a simple and small single-pass C compiler, currently supporting most of the C language common between C89/ANSI C and C99 (minus some C89 and plus some C99 features).

Alexey Frunze 1.2k Nov 22, 2022
Microvm is a virtual machine and compiler

The aim of this project is to create a stack based language and virtual machine for microcontrollers. A mix of approaches is used. Separate memory is used for program and variable space (Harvard architecture). An interpreter, virtual machine and compiler are available. A demostration of the interpreter in action is presented below.

null 10 Aug 14, 2022