Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s and faster.

Overview

Lizard - efficient compression with very fast decompression

Lizard (formerly LZ5) is a lossless compression algorithm which contains 4 compression methods:

  • fastLZ4 : compression levels -10...-19 are designed to give better decompression speed than LZ4 i.e. over 2000 MB/s
  • LIZv1 : compression levels -20...-29 are designed to give better ratio than LZ4 keeping 75% decompression speed
  • fastLZ4 + Huffman : compression levels -30...-39 add Huffman coding to fastLZ4
  • LIZv1 + Huffman : compression levels -40...-49 give the best ratio (comparable to zlib and low levels of zstd/brotli) at decompression speed of 1000 MB/s

Lizard library is based on frequently used LZ4 library by Yann Collet but the Lizard compression format is not compatible with LZ4. Lizard library is provided as open-source software using BSD 2-Clause license. The high compression/decompression speed is achieved without any SSE and AVX extensions.

Branch Status
lz5_v1.5 Build Status Build status
lizard Build Status Build status

Benchmarks

The following results are obtained with lzbench and -t16,16 using 1 core of Intel Core i5-4300U, Windows 10 64-bit (MinGW-w64 compilation under gcc 6.2.0) with silesia.tar which contains tarred files from Silesia compression corpus.

Compressor name Compression Decompress. Compr. size Ratio
memcpy 7332 MB/s 8719 MB/s 211947520 100.00
lz4 1.7.3 440 MB/s 2318 MB/s 100880800 47.60
lz4hc 1.7.3 -1 98 MB/s 2121 MB/s 87591763 41.33
lz4hc 1.7.3 -4 55 MB/s 2259 MB/s 79807909 37.65
lz4hc 1.7.3 -9 22 MB/s 2315 MB/s 77892285 36.75
lz4hc 1.7.3 -12 17 MB/s 2323 MB/s 77849762 36.73
lz4hc 1.7.3 -16 10 MB/s 2323 MB/s 77841782 36.73
lizard 1.0 -10 346 MB/s 2610 MB/s 103402971 48.79
lizard 1.0 -12 103 MB/s 2458 MB/s 86232422 40.69
lizard 1.0 -15 50 MB/s 2552 MB/s 81187330 38.31
lizard 1.0 -19 3.04 MB/s 2497 MB/s 77416400 36.53
lizard 1.0 -21 157 MB/s 1795 MB/s 89239174 42.10
lizard 1.0 -23 30 MB/s 1778 MB/s 81097176 38.26
lizard 1.0 -26 6.63 MB/s 1734 MB/s 74503695 35.15
lizard 1.0 -29 1.37 MB/s 1634 MB/s 68694227 32.41
lizard 1.0 -30 246 MB/s 909 MB/s 85727429 40.45
lizard 1.0 -32 94 MB/s 1244 MB/s 76929454 36.30
lizard 1.0 -35 47 MB/s 1435 MB/s 73850400 34.84
lizard 1.0 -39 2.94 MB/s 1502 MB/s 69807522 32.94
lizard 1.0 -41 126 MB/s 961 MB/s 76100661 35.91
lizard 1.0 -43 28 MB/s 1101 MB/s 70955653 33.48
lizard 1.0 -46 6.25 MB/s 1073 MB/s 65413061 30.86
lizard 1.0 -49 1.27 MB/s 1064 MB/s 60679215 28.63
zlib 1.2.8 -1 66 MB/s 244 MB/s 77259029 36.45
zlib 1.2.8 -6 20 MB/s 263 MB/s 68228431 32.19
zlib 1.2.8 -9 8.37 MB/s 266 MB/s 67644548 31.92
zstd 1.1.1 -1 235 MB/s 645 MB/s 73659468 34.75
zstd 1.1.1 -2 181 MB/s 600 MB/s 70168955 33.11
zstd 1.1.1 -5 88 MB/s 565 MB/s 65002208 30.67
zstd 1.1.1 -8 31 MB/s 619 MB/s 61026497 28.79
zstd 1.1.1 -11 16 MB/s 613 MB/s 59523167 28.08
zstd 1.1.1 -15 4.97 MB/s 639 MB/s 58007773 27.37
zstd 1.1.1 -18 2.87 MB/s 583 MB/s 55294241 26.09
zstd 1.1.1 -22 1.44 MB/s 505 MB/s 52731930 24.88
brotli 0.5.2 -0 217 MB/s 244 MB/s 78226979 36.91
brotli 0.5.2 -2 96 MB/s 283 MB/s 68066621 32.11
brotli 0.5.2 -5 24 MB/s 312 MB/s 60801716 28.69
brotli 0.5.2 -8 5.56 MB/s 324 MB/s 57382470 27.07
brotli 0.5.2 -11 0.39 MB/s 266 MB/s 51138054 24.13

Documentation

The raw Lizard block compression format is detailed within lizard_Block_format.

To compress an arbitrarily long file or data stream, multiple blocks are required. Organizing these blocks and providing a common header format to handle their content is the purpose of the Frame format, defined into lizard_Frame_format. Interoperable versions of Lizard must respect this frame format.

Comments
  • Way better now, yet, LZSSE2 and LzTurbo cast long shadows

    Way better now, yet, LZSSE2 and LzTurbo cast long shadows

    Hi inikep, some textual decompression feedback, again.

    @github_staff Shouldn't it be better to add 'Feedback' section, thus feedbacks won't go to 'Issues', inappropriate it is.

    Anyway, seeing how v2.0 is way better than the one I played with before, wanted to see my new toy (nothing special) Nakamichi 'Zato' against LZSSE2, LZ5, and the rest superfast decompressors. TurboBench from Sep 4 2016 was used.

    Dind't have enough machine power and time to test more files, yet, the 19 ones serve well the overview.

    The benchmark is downloadable at: [https://1drv.ms/u/s!AmWWFXGMzDmEgmEpIiqenL43pg4N]

    On a LZSSE2, two notes:

    • I believe this buggy i5, that is used, mischaracterize the beauty of utilizing the available registers, surely i7 will give all the needed registers. A retest is needed on i7, and 4th+ gen on top of that;
    • When decompressing from L1/L2/L3 cache, or fetching fast from main RAM, AVX compile is much much faster than SSE4.1 one, see 'Dictionary_of_Native_American_Mythology.tar' testfile.

    The benchmark package contains:

    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>dir/on
    
    10/02/2016  08:55 PM             9,025 _BENCH.BAT
    10/02/2016  08:55 PM        10,687,517 Agatha_Christie_85-ebooks_(French)_TXT.tar.L17.LZSSE2
    10/02/2016  08:55 PM         9,803,106 Agatha_Christie_85-ebooks_(French)_TXT.tar.Nakamichi
    10/02/2016  08:55 PM        11,649,537 Agatha_Christie_89-ebooks_TXT.tar.L17.LZSSE2
    10/02/2016  08:55 PM        11,441,644 Agatha_Christie_89-ebooks_TXT.tar.Nakamichi
    10/02/2016  08:55 PM         5,524,854 Arabian_Nights_complete.html.L17.LZSSE2
    10/02/2016  08:55 PM         5,660,839 Arabian_Nights_complete.html.Nakamichi
    10/02/2016  08:55 PM         1,566,277 Blacklegs_Card_Sharps_and_Confidence_Men.pdf.L17.LZSSE2
    10/02/2016  08:55 PM         1,552,936 Blacklegs_Card_Sharps_and_Confidence_Men.pdf.Nakamichi
    10/02/2016  08:55 PM         4,840,938 Complete_Works_of_Fyodor_Dostoyevsky.txt.L17.LZSSE2
    10/02/2016  08:55 PM         4,859,306 Complete_Works_of_Fyodor_Dostoyevsky.txt.Nakamichi
    10/02/2016  08:55 PM        10,359,314 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.L17.LZSSE2
    10/02/2016  08:55 PM         9,730,008 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.Nakamichi
    10/02/2016  08:55 PM           668,060 Dictionary_of_Native_American_Mythology.tar.L17.LZSSE2
    10/02/2016  08:55 PM           717,938 Dictionary_of_Native_American_Mythology.tar.Nakamichi
    10/02/2016  08:55 PM         1,830,701 Duden_Synonymworterbuch_(De-De).dsl.L17.LZSSE2
    10/02/2016  08:55 PM         1,610,899 Duden_Synonymworterbuch_(De-De).dsl.Nakamichi
    10/02/2016  08:55 PM         4,573,102 Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.L17.LZSSE2
    10/02/2016  08:55 PM         4,564,126 Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.Nakamichi
    10/02/2016  08:55 PM        38,063,182 enwik8.L17.LZSSE2
    10/02/2016  08:55 PM        38,261,116 enwik8.Nakamichi
    10/02/2016  08:55 PM         1,768,505 Large_traffic_log_file_of_a_popular_website_fp.log.L17.LZSSE2
    10/02/2016  08:55 PM         1,990,428 Large_traffic_log_file_of_a_popular_website_fp.log.Nakamichi
    10/02/2016  08:55 PM           451,490 lz5-2.0_beta.exe
    10/02/2016  08:55 PM           433,552 Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.L17.LZSSE2
    10/02/2016  08:55 PM           468,913 Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.Nakamichi
    10/02/2016  08:55 PM             1,632 MokujIN 224 prompt.lnk
    10/02/2016  08:55 PM        17,363,811 Nakamichi_(Zato)_256MB-Sliding-Window_vs_LZSSE2.zip
    10/02/2016  08:55 PM           146,944 Nakamichi_Zato_XMM_PREFETCH_4096_Intel_15.0_64bit_AVX.exe
    10/02/2016  08:55 PM           154,624 Nakamichi_Zato_XMM_PREFETCH_4096_Intel_15.0_64bit_SSE41.exe
    10/02/2016  08:55 PM         3,496,527 Slownik_jezyka_polskiego_(Pol-Pol).dsl.L17.LZSSE2
    10/02/2016  08:55 PM         3,101,206 Slownik_jezyka_polskiego_(Pol-Pol).dsl.Nakamichi
    10/02/2016  08:55 PM           505,572 Smyslov_(Vasily_Smyslov_2627_games).pgn.L17.LZSSE2
    10/02/2016  08:55 PM           541,608 Smyslov_(Vasily_Smyslov_2627_games).pgn.Nakamichi
    10/02/2016  08:55 PM         5,160,110 The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.L17.LZSSE2
    10/02/2016  08:55 PM         5,031,882 The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.Nakamichi
    10/02/2016  08:55 PM         2,556,494 The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.L17.LZSSE2
    10/02/2016  08:55 PM         2,540,805 The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.Nakamichi
    10/02/2016  08:55 PM               348 turbobench_singlefile_noNakamichi_2G.bat
    10/02/2016  08:55 PM         9,129,173 turbobenchs.exe
    10/02/2016  08:55 PM         1,142,536 University_of_Canterbury_The_Calgary_Corpus.tar.L17.LZSSE2
    10/02/2016  08:55 PM         1,311,813 University_of_Canterbury_The_Calgary_Corpus.tar.Nakamichi
    10/02/2016  08:55 PM         2,152,105 Word-list_584879_words_Russian_Spell-Check.txt.L17.LZSSE2
    10/02/2016  08:55 PM         2,004,811 Word-list_584879_words_Russian_Spell-Check.txt.Nakamichi
    10/02/2016  08:55 PM         1,297,189 www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.L17.LZSSE2
    10/02/2016  08:55 PM         1,456,466 www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.Nakamichi
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>
    

    And the actual console results on i5-2430M @3.00GHz, RAM @666MHz:

    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Large_traffic_log_file_of_a_popular_website_fp.log -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 19:12:09 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
          411108     2.0       0.36       0.36   zpaq 5          Large_traffic_log_file_of_a_popular_website_fp.log.tbb
          709242     3.4       1.00     403.91   lzma 9          Large_traffic_log_file_of_a_popular_website_fp.log.tbb
          723419     3.5       5.45      49.40   bzip2           Large_traffic_log_file_of_a_popular_website_fp.log.tbb
          742667     3.6       0.24     793.82   brotli 11       Large_traffic_log_file_of_a_popular_website_fp.log.tbb
          745809     3.6       0.76    2184.24   lzturbo 39      Large_traffic_log_file_of_a_popular_website_fp.log.tbb
          801029     3.9       1.21    1750.18   zstd 22         Large_traffic_log_file_of_a_popular_website_fp.log.tbb
          872740     4.2       1.39     735.04   lzham 4         Large_traffic_log_file_of_a_popular_website_fp.log.tbb
          970589     4.7       6.37    1603.19   xpack 9         Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1010821     4.9      48.77    1718.66   zstd 12         Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1064459     5.2       0.87    3725.53   lzturbo 29      Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1084431     5.3       1.23    2447.71   lz5 15          Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1085963     5.3       7.57     121.05   zpaq 2          Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1143618     5.5       0.94    1043.22   crush 2         Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1178879                       3420.8    lz5-2.0_beta 18 ! Outside TurboBench !
         1333117     6.5      41.81     637.39   zlib 9          Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1408548     6.8      18.63    1386.39   libdeflate 9    Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1554934     7.5       1.12    5195.83   lzturbo 19      Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1561495                       4302.4    lz5-2.0_beta 9  ! Outside TurboBench !
         1567022     7.6      34.31    3805.99   lz4 16          Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1724307     8.4     257.46     603.83   brieflz         Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1768505                       4416      LZSSE2 17 SSE4.1! Outside TurboBench !
         1768505                       4672      LZSSE2 17 AVX   ! Outside TurboBench !
         1768509     8.6       2.96    4414.79   lzsse2 17       Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1939991     9.4     445.99     339.39   density 3       Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         1990428                       2560      Nakamichi-Zato  ! Outside TurboBench !
         2041871     9.9      11.78    3969.40   lzsse2 1        Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         2458195    11.9     513.22     935.48   fastlz 2        Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         2754356    13.4     971.13    1760.19   snappy_c        Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         2806326    13.6     528.68     944.44   lzf             Large_traffic_log_file_of_a_popular_website_fp.log.tbb
         3016983    14.6     143.55    3652.92   yappy           Large_traffic_log_file_of_a_popular_website_fp.log.tbb
        11053474    53.6    1203.35    2929.39   chameleon 2     Large_traffic_log_file_of_a_popular_website_fp.log.tbb
        20511136    99.5     143.57    1228.89   trle            Large_traffic_log_file_of_a_popular_website_fp.log.tbb
        20617075   100.0    6159.87    5509.64   memcpy          Large_traffic_log_file_of_a_popular_website_fp.log.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Word-list_584879_words_Russian_Spell-Check.txt -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 19:17:03 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
          395907     5.7       0.27       0.27   zpaq 5          Word-list_584879_words_Russian_Spell-Check.txt.tbb
          961241    13.7       1.36      81.69   lzma 9          Word-list_584879_words_Russian_Spell-Check.txt.tbb
          972129    13.9       0.30     282.44   brotli 11       Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1038938    14.8       2.50     552.26   lzturbo 39      Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1053174    15.0       0.93     462.60   zstd 22         Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1068797    15.3       1.09     133.01   lzham 4         Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1134768    16.2      12.76     498.47   libdeflate 9    Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1331729    19.0      15.38     501.86   zstd 12         Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1384453    19.8       2.75     492.78   xpack 9         Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1410045    20.1       1.64     283.05   zlib 9          Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1647224    23.5      13.32      34.23   bzip2           Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1818611    26.0       0.12     254.46   crush 2         Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1884914    26.9       5.18      64.44   zpaq 2          Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1958090    28.0     139.17     211.29   brieflz         Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1958925    28.0       2.67     951.28   lzturbo 29      Word-list_584879_words_Russian_Spell-Check.txt.tbb
         1959563    28.0       0.83     828.85   lz5 15          Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2004811                        832      Nakamichi-Zato  ! Outside TurboBench !
         2073997                       1399.8    lz5-2.0_beta 18 ! Outside TurboBench !
         2109594    30.1      70.40    1957.07   yappy           Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2152105                       1664      LZSSE2 17 SSE4.1! Outside TurboBench !
         2152105                       1664      LZSSE2 17 AVX   ! Outside TurboBench !
         2152109    30.7       8.05    1716.22   lzsse2 17       Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2236063    31.9      16.07    1530.82   lzsse2 1        Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2339944    33.4       3.23    1229.66   lzturbo 19      Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2346546                       1547.4    lz5-2.0_beta 9  ! Outside TurboBench !
         2363922    33.8       6.91    1595.00   lz4 16          Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2422149    34.6     281.36     480.54   lzf             Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2445093    34.9     287.86     387.69   fastlz 2        Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2449587    35.0     373.98     915.45   snappy_c        Word-list_584879_words_Russian_Spell-Check.txt.tbb
         2536517    36.2     277.47     248.53   density 3       Word-list_584879_words_Russian_Spell-Check.txt.tbb
         3888621    55.5    1169.47    2460.62   chameleon 2     Word-list_584879_words_Russian_Spell-Check.txt.tbb
         7000457   100.0     137.35    5672.98   trle            Word-list_584879_words_Russian_Spell-Check.txt.tbb
         7000457   100.0    6346.74    5595.89   memcpy          Word-list_584879_words_Russian_Spell-Check.txt.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Agatha_Christie_85-ebooks_(French)_TXT.tar -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 14:13:46 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         5639052    17.6       0.30       0.30   zpaq 5          Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         6739693    21.1       0.95      81.88   lzma 9          Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         6758120    21.1       0.79     166.72   lzham 4         Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         6797247    21.2       0.92     389.67   lzturbo 39      Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         6802328    21.3       1.10     301.01   zstd 22         Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         6910154    21.6       0.37     245.03   brotli 11       Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         7680438    24.0       9.49      20.61   bzip2           Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         8367282    26.1       0.61     355.33   xpack 9         Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         8897625    27.8       3.28      58.95   zpaq 2          Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         8965799    28.0       4.68     499.64   zstd 12         Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         8990951    28.1       0.93     610.22   lzturbo 29      Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         9147309    28.6       0.12     263.86   crush 2         Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
         9332360                        769.3    lz5-2.0_beta 18 ! Outside TurboBench !
         9803106                        960      Nakamichi-Zato  ! Outside TurboBench !
        10107487    31.6       1.30     396.87   lz5 15          Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        10460666    32.7       6.97     454.65   libdeflate 9    Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        10687517                       2176      LZSSE2 17 SSE4.1! Outside TurboBench !
        10687517                       2304      LZSSE2 17 AVX   ! Outside TurboBench !
        10687521    33.4       2.07    2267.44   lzsse2 17       Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        10973383    34.3      10.73     232.96   zlib 9          Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        12107435    37.8       1.01    2658.62   lzturbo 19      Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        12132016                       1840.8    lz5-2.0_beta 9  ! Outside TurboBench !
        12266943    38.3      16.35    1897.96   lz4 16          Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        13238993    41.4      83.58     154.50   brieflz         Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        14364428    44.9     283.27     251.05   density 3       Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        15448533    48.3      12.31    1691.80   lzsse2 1        Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        17262978    53.9      61.89    1265.41   yappy           Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        17678955    55.2     178.84     361.43   lzf             Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        17868118    55.8     182.69     313.48   fastlz 2        Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        17915390    56.0    1049.93    2004.08   chameleon 2     Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        18694206    58.4     222.62     644.77   snappy_c        Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        31952980    99.8     148.95    1323.10   trle            Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
        32007172   100.0    6225.86    5379.36   memcpy          Agatha_Christie_85-ebooks_(French)_TXT.tar.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Agatha_Christie_89-ebooks_TXT.tar -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 14:31:39 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         6267713    18.8       0.31       0.31   zpaq 5          Agatha_Christie_89-ebooks_TXT.tar.tbb
         7881393    23.7       0.94      72.93   lzma 9          Agatha_Christie_89-ebooks_TXT.tar.tbb
         7917303    23.8       0.79     147.72   lzham 4         Agatha_Christie_89-ebooks_TXT.tar.tbb
         7963999    23.9       1.04     260.59   zstd 22         Agatha_Christie_89-ebooks_TXT.tar.tbb
         7973327    24.0       0.89     348.96   lzturbo 39      Agatha_Christie_89-ebooks_TXT.tar.tbb
         8014306    24.1       0.35     218.91   brotli 11       Agatha_Christie_89-ebooks_TXT.tar.tbb
         8416486    25.3      10.01      21.03   bzip2           Agatha_Christie_89-ebooks_TXT.tar.tbb
         9595005    28.8       0.55     319.11   xpack 9         Agatha_Christie_89-ebooks_TXT.tar.tbb
         9851022    29.6       4.71     471.32   zstd 12         Agatha_Christie_89-ebooks_TXT.tar.tbb
         9926145    29.8       3.03      56.95   zpaq 2          Agatha_Christie_89-ebooks_TXT.tar.tbb
        10076254    30.3       0.14     247.08   crush 2         Agatha_Christie_89-ebooks_TXT.tar.tbb
        10334621    31.1       0.89     567.62   lzturbo 29      Agatha_Christie_89-ebooks_TXT.tar.tbb
        11133495                         774.0   lz5-2.0_beta 18 ! Outside TurboBench !
        11145205    33.5       1.33     386.04   lz5 15          Agatha_Christie_89-ebooks_TXT.tar.tbb
        11175291    33.6       7.26     437.57   libdeflate 9    Agatha_Christie_89-ebooks_TXT.tar.tbb
        11441644                        960      Nakamichi-Zato  ! Outside TurboBench !
        11649537                       2112      LZSSE2 17 SSE4.1! Outside TurboBench !
        11649537                       2240      LZSSE2 17 AVX   ! Outside TurboBench !
        11649541    35.0       4.65    2172.48   lzsse2 17       Agatha_Christie_89-ebooks_TXT.tar.tbb
        11716396    35.2      10.54     226.92   zlib 9          Agatha_Christie_89-ebooks_TXT.tar.tbb
        13165738    39.6       1.04    2570.80   lzturbo 19      Agatha_Christie_89-ebooks_TXT.tar.tbb
        13194355                       1813.8    lz5-2.0_beta 9  ! Outside TurboBench !
        13327732    40.1      16.41    1870.35   lz4 16          Agatha_Christie_89-ebooks_TXT.tar.tbb
        14305207    43.0      77.90     148.64   brieflz         Agatha_Christie_89-ebooks_TXT.tar.tbb
        15146404    45.5     277.03     245.52   density 3       Agatha_Christie_89-ebooks_TXT.tar.tbb
        15890698    47.8      12.23    1702.86   lzsse2 1        Agatha_Christie_89-ebooks_TXT.tar.tbb
        18054497    54.3      61.85    1292.14   yappy           Agatha_Christie_89-ebooks_TXT.tar.tbb
        18191878    54.7     176.19     357.75   lzf             Agatha_Christie_89-ebooks_TXT.tar.tbb
        18478571    55.6     180.34     307.93   fastlz 2        Agatha_Christie_89-ebooks_TXT.tar.tbb
        18773896    56.4    1103.36    1976.50   chameleon 2     Agatha_Christie_89-ebooks_TXT.tar.tbb
        19377968    58.3     245.84     647.92   snappy_c        Agatha_Christie_89-ebooks_TXT.tar.tbb
        33156061    99.7     149.02    1321.56   trle            Agatha_Christie_89-ebooks_TXT.tar.tbb
        33258500   100.0    6312.11    5575.61   memcpy          Agatha_Christie_89-ebooks_TXT.tar.tbb
    
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Arabian_Nights_complete.html -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 14:40:27 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         2975964    19.1       0.30       0.30   zpaq 5          Arabian_Nights_complete.html.tbb
         3892786    25.0       1.05      68.74   lzma 9          Arabian_Nights_complete.html.tbb
         3913051    25.1       0.34     212.05   brotli 11       Arabian_Nights_complete.html.tbb
         3961082    25.4       0.89     145.14   lzham 4         Arabian_Nights_complete.html.tbb
         3988669    25.6       1.05     299.58   zstd 22         Arabian_Nights_complete.html.tbb
         3992884    25.6       1.04     385.67   lzturbo 39      Arabian_Nights_complete.html.tbb
         4038902    25.9      10.07      21.41   bzip2           Arabian_Nights_complete.html.tbb
         4589061    29.4       0.64     324.11   xpack 9         Arabian_Nights_complete.html.tbb
         4620802    29.7       5.18     464.37   zstd 12         Arabian_Nights_complete.html.tbb
         4724482    30.3       3.02      56.53   zpaq 2          Arabian_Nights_complete.html.tbb
         4757444    30.5       0.16     243.38   crush 2         Arabian_Nights_complete.html.tbb
         5035317    32.3       1.05     635.15   lzturbo 29      Arabian_Nights_complete.html.tbb
         5223100    33.5       1.03     409.66   lz5 15          Arabian_Nights_complete.html.tbb
         5259086    33.7       7.66     416.98   libdeflate 9    Arabian_Nights_complete.html.tbb
         5478319    35.2       9.98     223.95   zlib 9          Arabian_Nights_complete.html.tbb
         5509438                        928.2    lz5-2.0_beta 18 ! Outside TurboBench !
         5524854                       2048      LZSSE2 17 SSE4.1! Outside TurboBench !
         5524854                       2240      LZSSE2 17 AVX   ! Outside TurboBench !
         5524858    35.5       5.26    2108.15   lzsse2 17       Arabian_Nights_complete.html.tbb
         5660839                        960      Nakamichi-Zato  ! Outside TurboBench !
         6183326    39.7       1.23    2527.32   lzturbo 19      Arabian_Nights_complete.html.tbb
         6185910                       1814.8    lz5-2.0_beta 9  ! Outside TurboBench !
         6258303    40.2      15.74    1815.83   lz4 16          Arabian_Nights_complete.html.tbb
         6740391    43.3      79.89     150.83   brieflz         Arabian_Nights_complete.html.tbb
         7101428    45.6     250.89     227.77   density 3       Arabian_Nights_complete.html.tbb
         7294087    46.8      12.03    1700.32   lzsse2 1        Arabian_Nights_complete.html.tbb
         8585442    55.1     168.81     356.18   lzf             Arabian_Nights_complete.html.tbb
         8613199    55.3     179.31     313.49   fastlz 2        Arabian_Nights_complete.html.tbb
         8630338    55.4      62.13    1302.20   yappy           Arabian_Nights_complete.html.tbb
         9027159    57.9     247.54     632.03   snappy_c        Arabian_Nights_complete.html.tbb
         9049558    58.1    1093.34    1864.05   chameleon 2     Arabian_Nights_complete.html.tbb
        15580578   100.0     144.65    1319.96   trle            Arabian_Nights_complete.html.tbb
        15583444   100.0    6495.81    5477.48   memcpy          Arabian_Nights_complete.html.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Blacklegs_Card_Sharps_and_Confidence_Men.pdf -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 14:42:24 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         1433545    91.5       0.22       0.22   zpaq 5          Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1437059    91.7       0.29     107.33   brotli 11       Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1443674    92.2       3.83    2782.02   lzturbo 39      Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1445273    92.3       4.97      92.01   lzham 4         Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1447307    92.4       5.42    3361.11   zstd 22         Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1448653    92.5       1.91     409.16   xpack 9         Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1449223    92.5       4.37    6166.44   lzturbo 29      Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1450837    92.6      34.31    4437.05   zstd 12         Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1452139    92.7      23.39     286.44   libdeflate 9    Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1452300    92.7       4.95      37.43   zpaq 2          Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1454011    92.8       2.84    3975.32   lz5 15          Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1454076    92.8       3.64      17.07   lzma 9          Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1454251                       7285.0    lz5-2.0_beta 18 ! Outside TurboBench !
         1455523    92.9       4.85    7530.18   lzturbo 19      Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1456105    93.0      25.75     234.33   zlib 9          Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1458737                       7950.6    lz5-2.0_beta 9  ! Outside TurboBench !
         1458993    93.2       6.00      11.92   bzip2           Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1459844    93.2      34.26    6190.82   lz4 16          Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1474902    94.2    1857.98    4819.31   snappy_c        Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1513157    96.6     333.89     317.19   density 3       Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1513539    96.6     256.01    1596.61   fastlz 2        Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1515421    96.8     165.74    1687.80   lzf             Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1528008    97.6      58.24    5844.32   yappy           Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1552936                       2944      Nakamichi-Zato  ! Outside TurboBench !
         1559959    99.6     988.81    1220.79   chameleon 2     Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1566281   100.0     165.01    7677.83   trle            Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1566277                      95552      LZSSE2 17 SSE4.1! Outside TurboBench !
         1566277                      95552      LZSSE2 17 AVX   ! Outside TurboBench !
         1566281   100.0      13.58    7677.83   lzsse2 17       Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1566281   100.0      16.82    7640.38   lzsse2 1        Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1566281   100.0    7458.46    4789.84   memcpy          Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1617666   103.3       9.15     181.96   crush 2         Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
         1634540   104.4      99.78     634.89   brieflz         Blacklegs_Card_Sharps_and_Confidence_Men.pdf.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Complete_Works_of_Fyodor_Dostoyevsky.txt -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 14:50:16 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         2650827    19.3       0.31       0.31   zpaq 5          Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         3470856    25.3      10.08      20.71   bzip2           Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         3474265    25.3       1.08      70.01   lzma 9          Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         3488656    25.4       0.36     227.95   brotli 11       Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         3490560    25.5       0.92     147.68   lzham 4         Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         3502860    25.5       1.25     314.05   zstd 22         Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         3512645    25.6       1.03     396.60   lzturbo 39      Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4097920    29.9       4.59     469.52   zstd 12         Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4104258    29.9       0.62     342.84   xpack 9         Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4203354    30.7       0.13     246.45   crush 2         Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4211285    30.7       3.03      56.49   zpaq 2          Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4555584    33.2       1.07     615.66   lzturbo 29      Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4618517    33.7       7.25     437.03   libdeflate 9    Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4697463    34.3       1.38     420.74   lz5 15          Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4840938                       2112      LZSSE2 17 SSE4.1! Outside TurboBench !
         4840938                       2176      LZSSE2 17 AVX   ! Outside TurboBench !
         4840942    35.3       4.71    2161.27   lzsse2 17       Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4852707    35.4      10.44     227.07   zlib 9          Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         4859306                        960      Nakamichi-Zato  ! Outside TurboBench !
         4885062                        867.2    lz5-2.0_beta 18 ! Outside TurboBench !
         5488915    40.0       1.26    2532.93   lzturbo 19      Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         5491134                       1805      lz5-2.0_beta 9  ! Outside TurboBench !
         5554582    40.5      15.92    1845.41   lz4 16          Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         5944346    43.3      74.60     147.71   brieflz         Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         6378991    46.5     273.52     243.70   density 3       Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         6664446    48.6      12.22    1679.31   lzsse2 1        Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         7434625    54.2      62.38    1279.70   yappy           Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         7557433    55.1     168.58     358.05   lzf             Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         7662064    55.9     176.15     307.64   fastlz 2        Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         7741282    56.5    1025.52    1966.62   chameleon 2     Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
         8028636    58.5     246.64     629.16   snappy_c        Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
        13704843    99.9     144.97    1312.28   trle            Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
        13713279   100.0    6624.77    6086.67   memcpy          Complete_Works_of_Fyodor_Dostoyevsky.txt.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 15:16:32 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         5881208    14.6       0.34       0.34   zpaq 5          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         6942121    17.2       0.92      96.16   lzma 9          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         6996681    17.4       0.77     198.87   lzham 4         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         7028270    17.4       0.80     437.34   lzturbo 39      Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         7055102    17.5       1.08     355.74   zstd 22         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         7110795    17.6       0.33     297.52   brotli 11       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         7125187    17.7       9.43      23.06   bzip2           Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         8984648    22.3       3.62      67.43   zpaq 2          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         9221434    22.9       1.14     510.86   xpack 9         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         9253662                        874.7    lz5-2.0_beta 18 ! Outside TurboBench !
         9263242    23.0       0.06     318.59   crush 2         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         9298174    23.1       7.08     588.24   zstd 12         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         9491604    23.6       0.80     710.56   lzturbo 29      Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
         9730008                        896      Nakamichi-Zato  ! Outside TurboBench !
        10048812    24.9       1.35     555.35   lz5 15          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        10125158    25.1       5.37     605.01   libdeflate 9    Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        10359314                       2752      LZSSE2 17 SSE4.1! Outside TurboBench !
        10359314                       2880      LZSSE2 17 AVX   ! Outside TurboBench !
        10359318    25.7       4.28    2803.69   lzsse2 17       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        10771358    26.7       4.40     275.13   zlib 9          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        11806303    29.3       0.95    2915.87   lzturbo 19      Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        11817690                       1983.0    lz5-2.0_beta 9  ! Outside TurboBench !
        12065538    29.9       8.37    1983.71   lz4 16          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        15196140    37.7      95.09     159.80   brieflz         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        15278814    37.9      57.32    1675.18   yappy           Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        17802562    44.2     341.20     281.89   density 3       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        18298845    45.4      12.66    1793.96   lzsse2 1        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        19974577    49.6     280.56     718.80   snappy_c        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        20525594    50.9     201.42     387.03   lzf             Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        20596670    51.1     209.58     340.90   fastlz 2        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        21693398    53.8    1168.54    2562.83   chameleon 2     Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        40278350    99.9     150.33    1323.24   trle            Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
        40303108   100.0    6211.95    5505.14   memcpy          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Dictionary_of_Native_American_Mythology.tar -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 15:19:11 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
          304124    10.4       0.33       0.33   zpaq 5          Dictionary_of_Native_American_Mythology.tar.tbb
          419209    14.3       8.96      34.37   bzip2           Dictionary_of_Native_American_Mythology.tar.tbb
          422997    14.4       0.46     399.98   brotli 11       Dictionary_of_Native_American_Mythology.tar.tbb
          427665    14.6       1.34     111.56   lzma 9          Dictionary_of_Native_American_Mythology.tar.tbb
          447293    15.3       1.60     828.00   zstd 22         Dictionary_of_Native_American_Mythology.tar.tbb
          448088    15.3       1.60    1176.63   lzturbo 39      Dictionary_of_Native_American_Mythology.tar.tbb
          451937    15.4       1.41     243.61   lzham 4         Dictionary_of_Native_American_Mythology.tar.tbb
          489933    16.7      14.65     821.73   zstd 12         Dictionary_of_Native_American_Mythology.tar.tbb
          492053    16.8       3.63     846.92   xpack 9         Dictionary_of_Native_American_Mythology.tar.tbb
          513025    17.5       0.85     404.51   crush 2         Dictionary_of_Native_American_Mythology.tar.tbb
          533603    18.2       6.45      82.20   zpaq 2          Dictionary_of_Native_American_Mythology.tar.tbb
          563084    19.2       0.62    1046.32   lz5 15          Dictionary_of_Native_American_Mythology.tar.tbb
          563974    19.3       1.81    1571.16   lzturbo 29      Dictionary_of_Native_American_Mythology.tar.tbb
          617741    21.1      12.25     607.35   libdeflate 9    Dictionary_of_Native_American_Mythology.tar.tbb
          617824                       2122.2    lz5-2.0_beta 18 ! Outside TurboBench !
          629521    21.5      23.63     319.20   zlib 9          Dictionary_of_Native_American_Mythology.tar.tbb
          668060                       2816      LZSSE2 17 SSE4.1! Outside TurboBench !
          668060                       3712      LZSSE2 17 AVX   ! Outside TurboBench !
          668064    22.8       1.82    2557.76   lzsse2 17       Dictionary_of_Native_American_Mythology.tar.tbb
          705547                       2655.2    lz5-2.0_beta 9  ! Outside TurboBench !
          706652    24.1       2.12    3358.53   lzturbo 19      Dictionary_of_Native_American_Mythology.tar.tbb
          709584    24.2      22.47    2418.36   lz4 16          Dictionary_of_Native_American_Mythology.tar.tbb
          717938                       1600      Nakamichi-Zato  ! Outside TurboBench !
          738627    25.2     111.96     246.60   brieflz         Dictionary_of_Native_American_Mythology.tar.tbb
          799115    27.3      13.67    2412.39   lzsse2 1        Dictionary_of_Native_American_Mythology.tar.tbb
          885362    30.2     224.09     221.16   density 3       Dictionary_of_Native_American_Mythology.tar.tbb
          977236    33.4     257.58     476.82   fastlz 2        Dictionary_of_Native_American_Mythology.tar.tbb
         1055618    36.0     245.24     498.49   lzf             Dictionary_of_Native_American_Mythology.tar.tbb
         1064801    36.4     390.07     959.58   snappy_c        Dictionary_of_Native_American_Mythology.tar.tbb
         1364314    46.6      77.02    1847.72   yappy           Dictionary_of_Native_American_Mythology.tar.tbb
         1695276    57.9    1098.51    1914.14   chameleon 2     Dictionary_of_Native_American_Mythology.tar.tbb
         2563763    87.5     167.14    1430.70   trle            Dictionary_of_Native_American_Mythology.tar.tbb
         2928644   100.0    7143.02    7666.60   memcpy          Dictionary_of_Native_American_Mythology.tar.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Duden_Synonymworterbuch_(De-De).dsl -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 15:25:19 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
          811958     7.2       0.37       0.38   zpaq 5          Duden_Synonymworterbuch_(De-De).dsl.tbb
         1004261     8.9       1.18     166.17   lzma 9          Duden_Synonymworterbuch_(De-De).dsl.tbb
         1068742     9.4       1.19     919.77   lzturbo 39      Duden_Synonymworterbuch_(De-De).dsl.tbb
         1070058     9.4       1.00     353.88   lzham 4         Duden_Synonymworterbuch_(De-De).dsl.tbb
         1081825     9.5       0.41     456.20   brotli 11       Duden_Synonymworterbuch_(De-De).dsl.tbb
         1088298     9.6       1.25     716.11   zstd 22         Duden_Synonymworterbuch_(De-De).dsl.tbb
         1147399    10.1       7.13      33.48   bzip2           Duden_Synonymworterbuch_(De-De).dsl.tbb
         1335007    11.8       4.51      93.77   zpaq 2          Duden_Synonymworterbuch_(De-De).dsl.tbb
         1420344    12.5       1.25    1430.67   lzturbo 29      Duden_Synonymworterbuch_(De-De).dsl.tbb
         1424372    12.6       3.16     733.06   xpack 9         Duden_Synonymworterbuch_(De-De).dsl.tbb
         1466311    12.9      15.76     823.66   zstd 12         Duden_Synonymworterbuch_(De-De).dsl.tbb
         1485383    13.1       0.13     482.12   crush 2         Duden_Synonymworterbuch_(De-De).dsl.tbb
         1489744    13.1       1.00    1011.44   lz5 15          Duden_Synonymworterbuch_(De-De).dsl.tbb
         1493790                       1467.7    lz5-2.0_beta 18 ! Outside TurboBench !
         1610899                       1472      Nakamichi-Zato  ! Outside TurboBench !
         1740676    15.4       8.44     822.53   libdeflate 9    Duden_Synonymworterbuch_(De-De).dsl.tbb
         1744697    15.4       4.34     393.79   zlib 9          Duden_Synonymworterbuch_(De-De).dsl.tbb
         1830701                       3648      LZSSE2 17 SSE4.1! Outside TurboBench !
         1830701                       3648      LZSSE2 17 AVX   ! Outside TurboBench !
         1830705    16.2       3.28    3288.43   lzsse2 17       Duden_Synonymworterbuch_(De-De).dsl.tbb
         2037204    18.0       1.59    2809.23   lzturbo 19      Duden_Synonymworterbuch_(De-De).dsl.tbb
         2042405                       2175.3    lz5-2.0_beta 9  ! Outside TurboBench !
         2090628    18.4      11.50    1909.57   lz4 16          Duden_Synonymworterbuch_(De-De).dsl.tbb
         2479859    21.9      92.42    2067.72   yappy           Duden_Synonymworterbuch_(De-De).dsl.tbb
         2588673    22.8      10.74    2962.68   lzsse2 1        Duden_Synonymworterbuch_(De-De).dsl.tbb
         2605920    23.0     156.21     246.93   brieflz         Duden_Synonymworterbuch_(De-De).dsl.tbb
         3015462    26.6     317.64     271.72   density 3       Duden_Synonymworterbuch_(De-De).dsl.tbb
         3040630    26.8     371.51     620.77   lzf             Duden_Synonymworterbuch_(De-De).dsl.tbb
         3049421    26.9     355.83     532.30   fastlz 2        Duden_Synonymworterbuch_(De-De).dsl.tbb
         3233776    28.5     441.40     993.10   snappy_c        Duden_Synonymworterbuch_(De-De).dsl.tbb
         6029314    53.2    1216.62    3272.30   chameleon 2     Duden_Synonymworterbuch_(De-De).dsl.tbb
        11335236   100.0     139.52    5470.67   trle            Duden_Synonymworterbuch_(De-De).dsl.tbb
        11335236   100.0    6339.62    5452.25   memcpy          Duden_Synonymworterbuch_(De-De).dsl.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 15:31:51 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         2375112    18.4       0.30       0.30   zpaq 5          Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3087799    23.9       0.36     234.26   brotli 11       Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3102796    24.0       1.23      70.16   lzma 9          Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3149792    24.4       1.00     152.52   lzham 4         Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3163361    24.5       1.37     345.08   zstd 22         Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3176170    24.6       1.07     437.88   lzturbo 39      Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3423837    26.5      10.31      21.69   bzip2           Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3610798    27.9       0.74     353.93   xpack 9         Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3702916    28.6       6.20     479.15   zstd 12         Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3726436    28.8       3.32      59.29   zpaq 2          Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3795056    29.3       0.25     244.54   crush 2         Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         3999478    30.9       1.12     677.81   lzturbo 29      Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         4188499    32.4       1.51     436.06   lz5 15          Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         4327921                        946.6    lz5-2.0_beta 18 ! Outside TurboBench !
         4354767    33.7       8.04     417.97   libdeflate 9    Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         4521811    35.0      14.10     222.49   zlib 9          Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         4564126                        896      Nakamichi-Zato  ! Outside TurboBench !
         4573102                       2048      LZSSE2 17 SSE4.1! Outside TurboBench !
         4573102                       2240      LZSSE2 17 AVX   ! Outside TurboBench !
         4573106    35.4       5.89    2139.98   lzsse2 17       Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         5095655    39.4       1.32    2609.78   lzturbo 19      Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         5097610                       1864.2    lz5-2.0_beta 9  ! Outside TurboBench !
         5138931    39.7      21.52    1869.08   lz4 16          Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         5399037    41.7      80.69     153.48   brieflz         Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         5762189    44.6      11.93    1790.43   lzsse2 1        Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         5929465    45.8     230.92     211.71   density 3       Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         6858738    53.0      64.45    1444.34   yappy           Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         6989183    54.0     168.78     357.56   lzf             Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         7078866    54.7     172.08     309.69   fastlz 2        Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         7285714    56.3     249.62     673.72   snappy_c        Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
         7713222    59.6    1018.75    1764.06   chameleon 2     Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
        12928445   100.0     148.79    1311.90   trle            Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
        12934059   100.0    6387.19    5613.74   memcpy          Encyclopedia_of_Pain_(3_Vol_Set).pdf.txt.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe enwik8 -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 16:16:52 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
        20596996    20.6       0.30       0.29   zpaq 5          enwik8.tbb
        24763998    24.8       0.85      64.99   lzma 9          enwik8.tbb
        25021508    25.0       0.71     127.88   lzham 4         enwik8.tbb
        25390181    25.4       0.95     220.17   zstd 22         enwik8.tbb
        25403540    25.4       0.79     301.23   lzturbo 39      enwik8.tbb
        25743413    25.7       0.31     195.25   brotli 11       enwik8.tbb
        29008762    29.0      10.20      21.06   bzip2           enwik8.tbb
        30520879    30.5       0.44     245.40   xpack 9         enwik8.tbb
        31056443    31.1       5.94     427.89   zstd 12         enwik8.tbb
        31373273    31.4       0.81     549.03   lzturbo 29      enwik8.tbb
        31699553    31.7       0.26     222.10   crush 2         enwik8.tbb
        32073935    32.1       2.89      54.35   zpaq 2          enwik8.tbb
        34427078    34.4       1.51     382.56   lz5 15          enwik8.tbb
        35197151    35.2       8.78     375.05   libdeflate 9    enwik8.tbb
        36475796    36.5      14.57     212.84   zlib 9          enwik8.tbb
        36576568                        822.0    lz5-2.0_beta 18 ! Outside TurboBench !
        38063182                       1856      LZSSE2 17 SSE4.1! Outside TurboBench !
        38063182                       1984      LZSSE2 17 AVX   ! Outside TurboBench !
        38063186    38.1       6.05    1981.30   lzsse2 17       enwik8.tbb
        38261116                        768      Nakamichi-Zato  ! Outside TurboBench !
        41924195    41.9       0.94    2443.32   lzturbo 19      enwik8.tbb
        41956938                       1809.8    lz5-2.0_beta 9  ! Outside TurboBench !
        42196877    42.2      23.96    1802.61   lz4 16          enwik8.tbb
        43196169    43.2      76.49     152.52   brieflz         enwik8.tbb
        45854489    45.9      11.65    1712.27   lzsse2 1        enwik8.tbb
        47817696    47.8     218.21     205.05   density 3       enwik8.tbb
        53844730    53.8      65.94    1450.22   yappy           enwik8.tbb
        53915996    53.9     177.09     358.29   lzf             enwik8.tbb
        54163017    54.2     173.58     311.66   fastlz 2        enwik8.tbb
        56539849    56.5     249.67     631.71   snappy_c        enwik8.tbb
        61528422    61.5    1096.56    1661.82   chameleon 2     enwik8.tbb
        99138398    99.1     147.45    1277.33   trle            enwik8.tbb
       100000004   100.0    6125.95    5728.36   memcpy          enwik8.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 16:17:59 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
          263042    22.0       0.32       0.31   zpaq 5          Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          312058    26.1      10.06      20.84   bzip2           Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          337771    28.3       0.43     272.73   brotli 11       Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          343393    28.7       1.91      61.27   lzma 9          Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          346407    29.0       2.33     489.11   zstd 22         Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          347791    29.1       1.75     704.00   lzturbo 39      Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          354786    29.7       1.54     136.34   lzham 4         Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          376650    31.5       6.94     491.73   zstd 12         Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          380752    31.9       2.43     521.55   xpack 9         Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          396876    33.2       0.56     229.75   crush 2         Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          410524    34.3       7.00     425.26   libdeflate 9    Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          414525    34.7       4.81      59.35   zpaq 2          Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          432292    36.2       9.60     224.40   zlib 9          Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          433552                       2176      LZSSE2 17 SSE4.1! Outside TurboBench !
          433552                       2176      LZSSE2 17 AVX   ! Outside TurboBench !
          433556    36.3       5.36    2127.04   lzsse2 17       Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          451898    37.8       2.11     643.72   lz5 15          Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          454747    38.0       1.80     862.48   lzturbo 29      Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          468913                       1152      Nakamichi-Zato  ! Outside TurboBench !
          475502                       1419.7    lz5-2.0_beta 18 ! Outside TurboBench !
          489322    40.9       2.17    2565.23   lzturbo 19      Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          489595                       1853.3    lz5-2.0_beta 9  ! Outside TurboBench !
          495240    41.4      15.24    1864.89   lz4 16          Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          529655    44.3      75.61     147.96   brieflz         Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          594040    49.7      12.24    1655.67   lzsse2 1        Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          616213    51.5     189.90     190.68   density 3       Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          650156    54.4      61.36    1319.42   yappy           Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          666147    55.7     175.41     357.05   lzf             Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          677269    56.7     174.23     301.94   fastlz 2        Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          707035    59.1    1084.75    1747.66   chameleon 2     Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
          708040    59.2     241.06     634.16   snappy_c        Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
         1195311   100.0     147.47    1262.30   trle            Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
         1195401   100.0    7565.80    7762.32   memcpy          Mike_Tyson_-_Undisputed_Truth_-_My_Autobiography_-_2013.epub.txt.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Slownik_jezyka_polskiego_(Pol-Pol).dsl -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 16:28:53 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         1851513    10.8       0.37       0.37   zpaq 5          Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         2141578    12.4       1.11     124.47   lzma 9          Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         2179075    12.7       0.90     263.14   lzham 4         Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         2260440    13.1       0.99     683.08   lzturbo 39      Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         2276496    13.2       1.30     503.93   zstd 22         Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         2293510    13.3       0.38     374.40   brotli 11       Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         2394859    13.9       9.09      27.47   bzip2           Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         2840567    16.5       3.64      80.16   zpaq 2          Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         2997101    17.4       2.47     589.93   xpack 9         Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         3011302    17.5      10.79     676.74   zstd 12         Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         3060604                       1107.2    lz5-2.0_beta 18 ! Outside TurboBench !
         3060960    17.8       0.07     380.21   crush 2         Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         3091927    18.0       1.00    1037.46   lzturbo 29      Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         3101206                       1216      Nakamichi-Zato  ! Outside TurboBench !
         3231730    18.8       1.44     720.80   lz5 15          Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         3353571    19.5       5.80     736.70   libdeflate 9    Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         3496527                       3200      LZSSE2 17 SSE4.1! Outside TurboBench !
         3496527                       3328      LZSSE2 17 AVX   ! Outside TurboBench !
         3496531    20.3       4.07    3243.61   lzsse2 17       Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         3572553    20.8       3.10     319.69   zlib 9          Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         4006357    23.3       1.22    2934.36   lzturbo 19      Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         4012459                       1970.0    lz5-2.0_beta 9  ! Outside TurboBench !
         4138755    24.1       6.95    1812.87   lz4 16          Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         4901067    28.5      70.29    2109.64   yappy           Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         5500763    32.0     126.29     194.35   brieflz         Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         5730218    33.3      12.13    2397.12   lzsse2 1        Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         6101970    35.5     341.66     289.44   density 3       Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         6356114    36.9     309.19     541.25   lzf             Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         6652854    38.7     294.85     447.56   fastlz 2        Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         6936171    40.3     330.45     868.41   snappy_c        Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
         9150049    53.2    1256.05    3298.34   chameleon 2     Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
        17204132   100.0    6325.05    5638.85   memcpy          Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
        17204132   100.0     144.01    5562.28   trle            Slownik_jezyka_polskiego_(Pol-Pol).dsl.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe Smyslov_(Vasily_Smyslov_2627_games).pgn -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 16:30:22 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
          257330    15.5       0.34       0.33   zpaq 5          Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          314975    19.0       8.51      25.07   bzip2           Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          363995    21.9       1.53      69.25   lzma 9          Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          364389    22.0       0.40     277.83   brotli 11       Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          389023    23.4       1.89     525.16   zstd 22         Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          390007    23.5       1.49     787.49   lzturbo 39      Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          394133    23.7       1.53     154.98   lzham 4         Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          418780    25.2       6.92     560.26   zstd 12         Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          420151    25.3       1.95     599.08   xpack 9         Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          456748    27.5       0.56     287.75   crush 2         Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          466362    28.1       5.36      67.65   zpaq 2          Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          474789    28.6       8.79     490.12   libdeflate 9    Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          490598    29.6      12.43     269.27   zlib 9          Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          495673    29.9       1.90     834.19   lz5 15          Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          500488    30.1       1.64    1158.44   lzturbo 29      Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          505572                       3008      LZSSE2 17 SSE4.1! Outside TurboBench !
          505572                       3008      LZSSE2 17 AVX   ! Outside TurboBench !
          505576    30.5       4.45    2299.22   lzsse2 17       Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          520201                       1576.5    lz5-2.0_beta 18 ! Outside TurboBench !
          541608                       1280      Nakamichi-Zato  ! Outside TurboBench !
          557400    33.6       2.01    2585.73   lzturbo 19      Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          557733                       1964.5    lz5-2.0_beta 9  ! Outside TurboBench !
          566194    34.1      23.06    1928.04   lz4 16          Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          623758    37.6      96.30     184.37   brieflz         Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          688581    41.5     220.43     233.22   density 3       Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          710927    42.8      10.47    1790.76   lzsse2 1        Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          768969    46.3     235.83     442.09   lzf             Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          770887    46.4     242.16     418.78   fastlz 2        Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          782311    47.1      75.40    1277.94   yappy           Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          850739    51.2     295.33     676.19   snappy_c        Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
          937681    56.5    1097.18    2004.88   chameleon 2     Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
         1659995   100.0     148.80    1300.97   trle            Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
         1660043   100.0    7410.89    7377.95   memcpy          Smyslov_(Vasily_Smyslov_2627_games).pgn.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 16:38:19 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         2767461    20.2       0.31       0.30   zpaq 5          The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         3554802    26.0       1.06      69.16   lzma 9          The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         3562585    26.0       0.35     206.99   brotli 11       The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         3567992    26.1       0.89     142.08   lzham 4         The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         3585035    26.2       1.19     284.31   zstd 22         The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         3591509    26.2       1.05     366.58   lzturbo 39      The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         3703637    27.0       9.99      19.22   bzip2           The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         4197653    30.7       0.58     323.83   xpack 9         The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         4256970    31.1       4.27     452.97   zstd 12         The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         4294809    31.4       2.97      55.37   zpaq 2          The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         4371236    31.9       0.13     241.10   crush 2         The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         4632254    33.8       1.07     624.71   lzturbo 29      The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         4855421    35.5       1.30     372.19   lz5 15          The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         4924148    36.0       7.39     408.26   libdeflate 9    The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         5031882                        960      Nakamichi-Zato  ! Outside TurboBench !
         5032797                        826.5    lz5-2.0_beta 18 ! Outside TurboBench !
         5158479    37.7       9.68     217.04   zlib 9          The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         5160110                       2048      LZSSE2 17 SSE4.1! Outside TurboBench !
         5160110                       2112      LZSSE2 17 AVX   ! Outside TurboBench !
         5160114    37.7       5.00    2034.31   lzsse2 17       The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         5828546    42.6       1.26    2456.94   lzturbo 19      The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         5838641                       1758.9    lz5-2.0_beta 9  ! Outside TurboBench !
         5892721    43.0      14.96    1811.50   lz4 16          The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         6211745    45.4      77.62     145.52   brieflz         The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         6526796    47.7     266.83     239.25   density 3       The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         7029733    51.3      12.33    1592.99   lzsse2 1        The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         7814850    57.1    1084.32    1902.87   chameleon 2     The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         7913296    57.8     174.46     352.14   lzf             The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         7978444    58.3      61.33    1223.09   yappy           The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         8050844    58.8     170.32     306.06   fastlz 2        The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
         8508270    62.1     235.40     611.22   snappy_c        The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
        13674844    99.9     142.59    1323.70   trle            The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
        13694980   100.0    6396.53    5440.99   memcpy          The_complete_Aubrey_Maturin_by_Patrick_O'Brian_(21-TXTs).tar.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 16:43:20 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
         1399417    19.6       0.31       0.31   zpaq 5          The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         1835483    25.7       1.23      70.09   lzma 9          The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         1839607    25.8       0.37     248.10   brotli 11       The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         1858314    26.0      10.04      20.92   bzip2           The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         1858842    26.0       1.04     151.87   lzham 4         The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         1860631    26.1       1.45     380.19   zstd 22         The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         1865477    26.1       1.22     477.92   lzturbo 39      The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2133573    29.9       4.78     482.44   zstd 12         The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2135754    29.9       0.74     413.11   xpack 9         The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2184827    30.6       0.14     249.13   crush 2         The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2223226    31.1       3.47      58.98   zpaq 2          The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2407621    33.7       1.24     692.67   lzturbo 29      The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2421732    33.9       7.13     432.41   libdeflate 9    The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2445769    34.3       1.43     447.98   lz5 15          The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2539892    35.6       8.70     228.14   zlib 9          The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2540805                       1088      Nakamichi-Zato  ! Outside TurboBench !
         2556494                       2112      LZSSE2 17 SSE4.1! Outside TurboBench !
         2556494                       2304      LZSSE2 17 AVX   ! Outside TurboBench !
         2556498    35.8       5.22    2141.40   lzsse2 17       The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2595499                       1056.1    lz5-2.0_beta 18 ! Outside TurboBench !
         2891200    40.5       1.44    2364.12   lzturbo 19      The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         2896240                       1809.7    lz5-2.0_beta 9  ! Outside TurboBench !
         2929699    41.0      13.19    1850.47   lz4 16          The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         3047250    42.7      79.41     151.27   brieflz         The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         3405772    47.7     261.79     237.63   density 3       The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         3505515    49.1      12.34    1662.93   lzsse2 1        The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         3793663    53.2      61.37    1295.80   yappy           The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         3917217    54.9     171.14     360.47   lzf             The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         3984251    55.8     179.46     307.76   fastlz 2        The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         4054300    56.8    1026.06    1958.10   chameleon 2     The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         4167383    58.4     248.95     629.89   snappy_c        The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         7108504    99.6     148.08    1317.81   trle            The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
         7137284   100.0    6553.98    5765.17   memcpy          The_Project_Gutenberg_12_Fairy_Books_by_Andrew_Lang.tar.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe University_of_Canterbury_The_Calgary_Corpus.tar -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 16:46:20 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
          688059    21.1       0.31       0.31   zpaq 5          University_of_Canterbury_The_Calgary_Corpus.tar.tbb
          851327    26.1       1.75      60.80   lzma 9          University_of_Canterbury_The_Calgary_Corpus.tar.tbb
          863141    26.4       0.38     233.77   brotli 11       University_of_Canterbury_The_Calgary_Corpus.tar.tbb
          890859    27.3      11.64      25.20   bzip2           University_of_Canterbury_The_Calgary_Corpus.tar.tbb
          899273    27.5       2.23     493.73   zstd 22         University_of_Canterbury_The_Calgary_Corpus.tar.tbb
          902432    27.6       1.59     686.90   lzturbo 39      University_of_Canterbury_The_Calgary_Corpus.tar.tbb
          913218    28.0       1.61     137.29   lzham 4         University_of_Canterbury_The_Calgary_Corpus.tar.tbb
          958042    29.3       1.89     493.21   xpack 9         University_of_Canterbury_The_Calgary_Corpus.tar.tbb
          968043    29.6       9.54     532.63   zstd 12         University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1021136    31.3       9.31     422.94   libdeflate 9    University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1022374    31.3       0.70     235.13   crush 2         University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1060310    32.5       9.04     235.18   zlib 9          University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1077695    33.0       4.90      60.46   zpaq 2          University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1114797    34.1       1.77     966.13   lzturbo 29      University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1125796    34.5       1.83     688.06   lz5 15          University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1142536                       2112      LZSSE2 17 SSE4.1! Outside TurboBench !
         1142536                       2496      LZSSE2 17 AVX   ! Outside TurboBench !
         1142540    35.0       0.44    2042.24   lzsse2 17       University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1192352                       1509.7    lz5-2.0_beta 18 ! Outside TurboBench !
         1231161    37.7       2.07    2625.03   lzturbo 19      University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1237244                       2014.5    lz5-2.0_beta 9  ! Outside TurboBench !
         1240210    38.0       4.17    1942.62   lz4 16          University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1298840    39.8      89.11     180.28   brieflz         University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1311813                        960      Nakamichi-Zato  ! Outside TurboBench !
         1405647    43.0      12.14    1749.08   lzsse2 1        University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1596322    48.9     199.22     394.06   lzf             University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1607581    49.2     194.25     360.83   fastlz 2        University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1610706    49.3     206.42     194.15   density 3       University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1624766    49.8      60.18    1554.28   yappy           University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         1682025    51.5     289.47     734.82   snappy_c        University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         2084324    63.8    1077.73    1576.79   chameleon 2     University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         2810417    86.1     191.46    5224.86   trle            University_of_Canterbury_The_Calgary_Corpus.tar.tbb
         3265540   100.0    6903.88    6610.40   memcpy          University_of_Canterbury_The_Calgary_Corpus.tar.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>turbobenchs.exe www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt -ezlib,9/libdeflate,9/xpack,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,12,22/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2,5/lzf/yappy/trle/memcpy/lzsse2,1,17 -g -k0 -J13 -B2G
    TurboBench:  - Sun Oct 02 16:52:02 2016
    
          C Size  ratio%     C MB/s     D MB/s   Name            File
          973794    21.7       0.26       0.26   zpaq 5          www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1098241    24.5       0.28     111.41   brotli 11       www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1107450    24.7       1.02      78.87   lzma 9          www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1115796    24.9       1.26     446.60   zstd 22         www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1124307    25.1       0.89     152.31   lzham 4         www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1156565    25.8      10.30      19.93   bzip2           www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1163771    25.9       4.55     542.94   libdeflate 9    www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1205730    26.9       1.08     294.70   zlib 9          www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1229904    27.4       2.81     553.32   xpack 9         www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1231733    27.4       7.63     478.05   zstd 12         www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1297189                       2432      LZSSE2 17 SSE4.1! Outside TurboBench !
         1297189                       2880      LZSSE2 17 AVX   ! Outside TurboBench !
         1297193    28.9       1.21    2429.58   lzsse2 17       www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1366183    30.4       0.04     282.19   crush 2         www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1396907    31.1       4.25      61.98   zpaq 2          www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1437041    32.0       1.23     727.18   lz5 15          www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1441772    32.1       1.47     867.14   lzturbo 29      www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1456466                       1152      Nakamichi-Zato  ! Outside TurboBench !
         1503145                       1516.5    lz5-2.0_beta 18 ! Outside TurboBench !
         1509154    33.6       1.76    2556.94   lzturbo 19      www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1510839                       2000.6    lz5-2.0_beta 9  ! Outside TurboBench !
         1556225    34.7      50.22    2329.92   yappy           www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1596981    35.6       1.79    2074.63   lz4 16          www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1598279    35.6     145.80     224.33   brieflz         www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         1995712    44.5     362.36     731.09   snappy_c        www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         2161251    48.2     437.41     367.88   density 3       www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         2385361    53.2    1224.07    3230.69   chameleon 2     www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         2403235    53.6     240.33     441.11   lzf             www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         2408470    53.7     219.97     406.62   fastlz 2        www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         2559866    57.0      15.08    1358.59   lzsse2 1        www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         3677928    82.0     159.99     856.38   trle            www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         3849226    85.8       1.38    3717.84   lzturbo 39      www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
         4487437   100.0    6758.18    6198.11   memcpy          www.ncbi.nlm.nih.gov_(Ladona_fulva_-_whole_genome_shotgun_sequence)_sequence.fasta.txt.tbb
    
    D:\LZ5_v2beta_VS_LZSSE2_VS_Zato>
    

    Hope that someone will give the above picture on some serious machine, not like my crappy i5 ruining the beautiful ... exploitation of registers.

    Still haven't given up on making the richest textual roster with 400+ files... https://1drv.ms/i/s!AmWWFXGMzDmEgmI7l-33IsTzHnMy

    opened by Sanmayce 24
  • Tight, but not that much!

    Tight, but not that much!

    Hi, inikep, happy new 2016!

    Your lz5 fills the gap between lz4 and Zstd, nice. However, your next statement is valid mainly for mixed/binary data, yes? Or to be more precise, for compressors using up to 4MB dictionary, yes?

    In my experiments there is no open-source bytewise compressor that gives better ratio than lz5hc.

    For textual e.g. English texts it is not in my quick tests, my latest&last Nakamichi 'Goldenboy' yields better ratio while being faster in decompression (try Skylake):

    Note: I used for all files below '>lz5 -18 infile' command line.

    
    D:\_Deathship_textual_corpus\lz5>dir
     Volume in drive D is S640_Vol5
     Volume Serial Number is 5861-9E6C
    
     Directory of D:\_Deathship_textual_corpus\lz5
    
    01/06/2016  11:14 PM    <DIR>          .
    01/06/2016  11:14 PM    <DIR>          ..
    09/11/2015  05:25 AM        33,258,496 Agatha_Christie_89-ebooks_TXT.tar
    01/06/2016  10:40 PM        11,894,354 Agatha_Christie_89-ebooks_TXT.tar.lz5
    10/14/2015  02:29 PM        10,434,674 Agatha_Christie_89-ebooks_TXT.tar.Nakamichi
    09/11/2015  05:25 AM        13,713,275 Complete_Works_of_Fyodor_Dostoyevsky.txt
    01/06/2016  09:42 PM         4,981,395 Complete_Works_of_Fyodor_Dostoyevsky.txt.lz5
    10/12/2015  09:00 AM         4,544,039 Complete_Works_of_Fyodor_Dostoyevsky.txt.Nakamichi
    09/11/2015  05:25 AM        10,192,446 dickens
    01/06/2016  09:35 PM         3,936,253 dickens.lz5
    10/11/2015  04:14 PM         3,722,075 dickens.Nakamichi
    01/06/2016  10:26 PM           544,256 lz5-1.3.3.tar
    01/06/2016  10:27 PM           112,315 lz5-1.3.3.tar.lz5
    01/06/2016  11:14 PM           145,822 lz5-1.3.3.tar.Nakamichi
    01/05/2016  07:47 PM           217,556 lz5.exe
    01/01/2002  04:41 AM           120,832 Nakamichi_Kintaro_Intel_15.0_64bit.exe
    01/01/2002  05:50 AM         3,028,450 Nakamichi_Kintaro_source_booklet_executables_32bit_64bit_GCC510_Intel150.zip
    11/02/2015  10:33 PM       132,728,832 New_Shorter_Oxford_English_Dictionary_fifth_edition.tar
    01/06/2016  10:20 PM        24,712,948 New_Shorter_Oxford_English_Dictionary_fifth_edition.tar.lz5
    12/01/2015  08:56 AM        25,592,405 New_Shorter_Oxford_English_Dictionary_fifth_edition.tar.Nakamichi
    09/11/2015  05:25 AM        12,030,464 New_York_Times_Bestsellers_-_August_2015_-_20_ebooks.tar
    01/06/2016  09:47 PM         4,519,211 New_York_Times_Bestsellers_-_August_2015_-_20_ebooks.tar.lz5
    10/14/2015  05:18 PM         4,328,336 New_York_Times_Bestsellers_-_August_2015_-_20_ebooks.tar.Nakamichi
    09/11/2015  05:25 AM        14,613,183 The_Book_of_The_Thousand_Nights_and_a_Night.txt
    01/06/2016  10:26 PM         5,502,069 The_Book_of_The_Thousand_Nights_and_a_Night.txt.lz5
    10/12/2015  03:26 AM         5,228,912 The_Book_of_The_Thousand_Nights_and_a_Night.txt.Nakamichi
    01/06/2016  10:59 PM            92,096 The_Little_Prince_-_Antoine_de_Saint-Exupery.epub.txt
    01/06/2016  10:59 PM            37,008 The_Little_Prince_-_Antoine_de_Saint-Exupery.epub.txt.lz5
    10/11/2015  02:12 PM            43,944 The_Little_Prince_-_Antoine_de_Saint-Exupery.epub.txt.Nakamichi
    01/06/2016  10:56 PM         4,445,260 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt
    01/06/2016  10:22 PM         1,435,859 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt.lz5
    10/11/2015  11:15 AM         1,420,630 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt.Nakamichi
                  30 File(s)    337,577,395 bytes
                   2 Dir(s)  83,927,027,712 bytes free
    
    D:\_Deathship_textual_corpus\lz5>Nakamichi_Kintaro_Intel_15.0_64bit.exe The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt.Nakamichi /bench
    Nakamichi 'Kintaro', written by Kaze, based on Nobuo Ito's LZSS source, babealicious suggestion by m^2 enforced, muffinesque suggestion by Jim Dempsey enforced.
    Note: This compile can handle files up to 1711MB.
    Current priority class is HIGH_PRIORITY_CLASS.
    Decompressing 1420630 bytes ...
    RAM-to-RAM performance: 704 MB/s.
    Compression Ratio (bigger-the-better): 3.13:1
    
    D:\_Deathship_textual_corpus\lz5>Nakamichi_Kintaro_Intel_15.0_64bit.exe dickens.Nakamichi /bench
    Nakamichi 'Kintaro', written by Kaze, based on Nobuo Ito's LZSS source, babealicious suggestion by m^2 enforced, muffinesque suggestion by Jim Dempsey enforced.
    Note: This compile can handle files up to 1711MB.
    Current priority class is HIGH_PRIORITY_CLASS.
    Decompressing 3722075 bytes ...
    RAM-to-RAM performance: 512 MB/s.
    Compression Ratio (bigger-the-better): 2.74:1
    
    D:\_Deathship_textual_corpus\lz5>Nakamichi_Kintaro_Intel_15.0_64bit.exe The_Book_of_The_Thousand_Nights_and_a_Night.txt.Nakamichi /bench
    Nakamichi 'Kintaro', written by Kaze, based on Nobuo Ito's LZSS source, babealicious suggestion by m^2 enforced, muffinesque suggestion by Jim Dempsey enforced.
    Note: This compile can handle files up to 1711MB.
    Current priority class is HIGH_PRIORITY_CLASS.
    Decompressing 5228912 bytes ...
    RAM-to-RAM performance: 448 MB/s.
    Compression Ratio (bigger-the-better): 2.79:1
    
    D:\_Deathship_textual_corpus\lz5>Nakamichi_Kintaro_Intel_15.0_64bit.exe Agatha_Christie_89-ebooks_TXT.tar.Nakamichi /bench
    Nakamichi 'Kintaro', written by Kaze, based on Nobuo Ito's LZSS source, babealicious suggestion by m^2 enforced, muffinesque suggestion by Jim Dempsey enforced.
    Note: This compile can handle files up to 1711MB.
    Current priority class is HIGH_PRIORITY_CLASS.
    Decompressing 10434674 bytes ...
    RAM-to-RAM performance: 320 MB/s.
    Compression Ratio (bigger-the-better): 3.19:1
    
    D:\_Deathship_textual_corpus\lz5>lz5.exe -h
    LZ5 command line interface 64-bit v1.3.3 by Y.Collet & P.Skibinski (Jan  5 2016)
    Usage :
          lz5.exe [arg] [input] [output]
    
    input   : a filename
              with no FILE, or when FILE is - or stdin, read standard input
    Arguments :
     -0       : Fast compression (default)
     -1...-18 : High compression; higher number == more compression but slower
     -d       : decompression (default for .lz5 extension)
     -z       : force compression
     -f       : overwrite output without prompting
     -h/-H    : display help/long help and exit
    
    Advanced arguments :
     -V     : display Version number and exit
     -v     : verbose mode
     -q     : suppress warnings; specify twice to suppress errors too
     -c     : force write to standard output, even if it is the console
     -t     : test compressed file integrity
     -m     : multiple input files (implies automatic output filenames)
     -l     : compress using Legacy format (Linux kernel compression)
     -B#    : Block size [4-7](default : 7)
     -BD    : Block dependency (improve compression ratio)
    --no-frame-crc : disable stream checksum (default:enabled)
    --content-size : compressed frame includes original size (default:not present)
    --[no-]sparse  : sparse mode (default:enabled on file, disabled on stdout)
    Benchmark arguments :
     -b     : benchmark file(s)
     -i#    : iteration loops [1-9](default : 3), benchmark mode only
    
    D:\_Deathship_textual_corpus\lz5>\lz5 -18 -b --no-frame-crc The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt
    The_Project_Gute :   4445260 ->   1435836 (32.30%),    0.1 MB/s ,  455.6 MB/s
    
    D:\_Deathship_textual_corpus\lz5>\lz5 -18 -b --no-frame-crc dickens
    dickens          :  10192446 ->   3936226 (38.62%),    0.1 MB/s ,  390.3 MB/s
    
    D:\_Deathship_textual_corpus\lz5>\lz5 -18 -b --no-frame-crc The_Book_of_The_Thousand_Nights_and_a_Night.txt
    The_Book_of_The_ :  14613183 ->   5502038 (37.65%),    0.1 MB/s ,  400.1 MB/s
    
    D:\_Deathship_textual_corpus\lz5>\lz5 -18 -b --no-frame-crc Agatha_Christie_89-ebooks_TXT.tar
    Agatha_Christie_ :  33258496 ->  11894307 (35.76%),    0.1 MB/s ,  410.8 MB/s
    
    D:\_Deathship_textual_corpus\lz5>
    

    Note: The results were obtained on my laptop with Core 2 Q9550s, Win7 64bit.

    As a final note, I wonder why Hamid didn't implement LzTurbo 19 with bigger window as well?! I fully expect his state-of-the-art parser to kick our asses right away, hee-hee!

    opened by Sanmayce 16
  • Format description

    Format description

    I want to implement an lz5 encoder. Sadly there doesn't seem to be any format description beyond the source code. Is the lz5 format stable enough for you to document things already?

    opened by JoernEngel 14
  • Please consider a name change

    Please consider a name change

    Recently, someone asked about an LZ5 plugin for Squash, which pushed me to speak about something which has been bothering me for a while. LZ5 seems like a very interesting project, and AFAICT you've done an excellent job with it. However, copying part of my answer to that issue:

    Ever since LZ5 was released I've been uncomfortable with it because of the name. "LZ5" implies that it is the "successor to LZ4" (i.e., LZ4 2.0), or at least an "upgraded" version, which isn't fair to LZ4. LZ5 simply makes different trade-offs.

    To be clear, I'm completely fine with the idea of forking LZ4 to create a new codec; it's only the project name I'm uncomfortable with.

    There are a lot of problems with implying such a link with LZ4, such as

    Support

    People are likely to go to LZ4 for support, which isn't fair to the authors of LZ4.

    My favorite example of the type of issue this could cause is a comment from SQLite:

    The default prefix used to be "sqlite_". But then Mcafee started using SQLite in their anti-virus product and it started putting files with the "sqlite" name in the c:/temp folder. This annoyed many windows users. Those users would then do a Google search for "sqlite", find the telephone numbers of the developers and call to wake them up at night and complain. For this reason, the default name prefix is changed to be "sqlite" spelled backwards. So the temp files are still identified, but anybody smart enough to figure out the code is also likely smart enough to know that calling the developer will not help get rid of the file.

    Diluting LZ4's brand

    If people think there is a link between LZ4 and LZ5, any problems with LZ5 are likely to influence their perception of LZ4, which isn't fair.

    Leeching off of LZ4's brand

    LZ4 has created a very good image of itself by being really good at what it does. Other projects should be able to build their own reputations based on their own quality instead of using LZ4 for a head-start.


    With all that said, I want to be very clear that, AFAICT (I haven't really looked at LZ5's code) everything seems to be pretty well-done. However, all the points above still stand, even if LZ5

    • Maintains a high standard of quality, including portability and security
    • Provide clear attribution of the history of LZ5 (as a fork of LZ4)
    • Makes it clear that LZ5 is not associated with LZ4
    • Provides reasonable support, and bug fixes for any issues discovered.

    I'm going to assume that LZ5 does, and continues to do, all those things. Every indication I've seen is that they do. No matter what else they do, though, the "LZ5" name still muddies the waters.

    To be clear, these are my own opinions. I don't think we should involve Yann in this conversation (especially not publicly), as it would put him in a very difficult position.

    I'd like to add that, because of the name, I've been hesitant to add an LZ5 plugin. It's quite possible other people will have the same concern, so it may actually be in LZ5's best interest to change their name.

    In the end, I think that yes, I'd be willing to accept a patch for an LZ5 plugin, so long as the documentation made its relationship with LZ4 extremely clear. I'd feel a lot better about it if LZ5 had a different name, though.

    opened by nemequ 5
  • My own Win-64 build version get worse compression ratio than lizard-1.0-windows

    My own Win-64 build version get worse compression ratio than lizard-1.0-windows

    Compiler: Visual Studio 2017 64-Bit Community Edition OS: Windows 10 64-bit Binary size: my version lizard-MyBuild.exe (200+ KB) v.s. official version lizard.exe (500+KB) File to compress: some Android firmware image (namely, "system-other.img")

    As you can see from the results attached below, compression ratio of lizard-MyBuild.exe is always not as good as the lizard.exe given the same level (e.g., 25, 29, 45, 49). Any idea on this difference?

    Compression ratio of lizard-MyBuild.exe:

    • .\lizard-MyBuild.exe -25f .\system_other.img .\system_other.img.liz.25 (Compressed 371409248 bytes into 187947816 bytes ==> 50.60%)
    • .\lizard-MyBuild.exe -29f .\system_other.img .\system_other.img.liz.29 (Compressed 371409248 bytes into 186106560 bytes ==> 50.11%)
    • .\lizard-MyBuild.exe -45f .\system_other.img .\system_other.img.liz.45 (Compressed 371409248 bytes into 172025694 bytes ==> 46.32%)
    • .\lizard-MyBuild.exe -49f .\system_other.img .\system_other.img.liz.49 (Compressed 371409248 bytes into 168488030 bytes ==> 45.36%)

    Compression ratio of lizard.exe:

    • lizard -25f .\system_other.img .\system_other.img.liz.25 (Compressed 371409248 bytes into 182706984 bytes ==> 49.19%)
    • lizard -29f .\system_other.img .\system_other.img.liz.29 (Compressed 371409248 bytes into 178186870 bytes ==> 47.98%)
    • lizard -45f .\system_other.img .\system_other.img.liz.45 (Compressed 371409248 bytes into 167356722 bytes ==> 45.06%)
    • lizard -49f .\system_other.img .\system_other.img.liz.49 (Compressed 371409248 bytes into 165760838 bytes ==> 44.63%)
    opened by yunhua-deng 3
  • Enlarging the Window

    Enlarging the Window

    Hi inikep, as a future enhancement, could you consider adding bigger windows than 256MB to your superb Lizard?! I just did it, despite the general trend to work with smaller windows (and more threads) and the lack of computational power for gigantic windows. The toy is called 'Ryuugan+' - a strong and fully functional one, it inherits all the good stuff from 'Washigan+' - my mainstay. Haven't been happy with the limited Eagleeye's 512MB sliding window, so the successor Dragoneye adds to all previous windows and the 37bit one - 128GB. The underlying notion is to have (in the future) one compressor able to see deeply in the whole English Wikipedia XML dump from start to the end - it steadily is growing 2GB per year, now it is 60GB.

    The idea is, when the right time comes, the compile&source of such performer to be available - not to rush and write when the machine nomenclature matches the requirements for 15GB+60GB=75GB RAM. Looking back, say, 10 years, many tools are blind to the rich resources that machines of today offer. It is good, for a change, to have the software ready beforehand, no?

    Nakamichi_Ryuugan+_(32bit_64bit_executables_source).zip 5.87 MB (6,160,495 bytes): https://drive.google.com/file/d/0BzKgu_YpO6uZYnpGY0d0NHNHQWc/view?usp=sharing

    Booklet_Ryuugan+.pdf 763 KB (781,446 bytes): https://drive.google.com/file/d/0BzKgu_YpO6uZb1VsSC02cE9WUlk/view?usp=sharing

    F:\TEXTUAL_MADNESS\OLD_DOCs\Nakamichi_Ryuugan+_(32bit_64bit_source)>"Nakamichi_Ryuugan+_(1xQWORD+1xXMM)_Intel_15.0_32bit_SSE41.exe" University_of_Canterbury_The_Calgary_Corpus.tar
    Nakamichi 'Ryuugan+', written by Kaze, based on Nobuo Ito's LZSS source, babealicious suggestion by m^2 enforced, muffinesque suggestion by Jim Dempsey enforced.
    Note0: Nakamichi 'Dragoneye' is 100% FREE, licenseless that is.
    Note1: Hamid Buzidi's LzTurbo ([a] FASTEST [Textual] Decompressor, Levels 19/29/39) retains kingship, his TurboBench (2017-Apr-07) proves the supremacy of LzTurbo, Turbo-Amazing!
    Note2: Conor Stokes' LZSSE2 ([a] FASTEST Textual Decompressor, Level 17) is embedded, all credits along with many thanks go to him.
    Note3: 'Ryuugan' predecessors are Washigan, Okamigan, Zato, Tsubame, Tengu-Tsuyo, Tengu, Rakka, Kokuen, Kinroba, Yoko, Kinutora, Jiten, Butsuhira, Suiken, Keigan, Kumataka, Washi, Aratama, Hitomi, Nekomata, Kitsune, Kinezumi, Sanbashi, Kaiko, Inazuma, Zangetsu, Hanabi, Hanazakari, Sanshi and Kaidanji.
    Note4: This compile can handle files up to 5120MB.
    Note5: The matchfinder/memmem() is 'Railgun_Trolldom'.
    Note6: Instead of '_mm_loadu_si128' '_mm_lddqu_si128' is used.
    Note7: The lookahead 'Tsuyo' heuristic which looks one char ahead is applied thrice, still not strengthened, though.
    Note8: The compile made 2017-May-04, the decompression time measuring is done in 16x8 passes choosing the top score from 64 back-to-back runs - the goal - to enter [maximal] Turbo Mode.
    Note9: Just to reduce the codesize, the 3xQWORD become (in this compile) 1xQWORD+1xXMMWORD.
    NoteA: Please send me (at [email protected]) decompression results obtained on machines with fast CPU-RAM subsystems.
    Current priority class is REALTIME_PRIORITY_CLASS.
    Allocating Source-Buffer 3 MB ...
    Allocating Source-Buffer 3 MB (REVERSED) ...
    Allocating Target-Buffer 35 MB ...
    Allocating Verification-Buffer 3 MB ...
    Compressing 3,265,536 bytes ...
    /; Each rotation means 64KB are encoded; Speed: 000,588B/s; Done 100%; Compression Ratio: 2.67:1; Matches(24/32/48): 4,233/317/1,483; Washigan Matches(5bytes offsets): 8
    NumberOfFullLiterals (lower-the-better): 8276
    Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 14
    NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 3852
    NumberOfMatches[Bheema]Window [128GB window]: 1
    RAM-to-RAM performance: 588 B/s.
    Compressed to 1,223,800 bytes.
    Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xe6d1,9b78
    Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xeacf,b4c9
    Decompressing 1,223,800 (being the compressed stream) bytes ...
    RAM-to-RAM performance: 494 MB/s.
    Verification (input and output sizes match) OK.
    Verification (input and output blocks match) OK.
    LZSSE2: Compressing with LZSSE2 (level 17) 3,265,536 bytes ...
    LZSSE2: Compressed to 1,142,536 bytes.
    LZSSE2: RAM-to-RAM performance: 376 KB/s.
    LZSSE2: Decompressing 1,142,536 bytes (being the compressed stream) ...
    LZSSE2: RAM-to-RAM performance: 671 MB/s.
    LZSSE2: Verification (input and output sizes match) OK.
    LZSSE2: Verification (input and output blocks match) OK.
    LZSSE2 vs Nakamichi 'Ryuugan', c.size: 0.93x
    LZSSE2 vs Nakamichi 'Ryuugan', d.rate: 1.36x
    
    07/02/2017  10:06 AM         3,265,536 University_of_Canterbury_The_Calgary_Corpus.tar
    07/02/2017  10:06 AM           667,746 University_of_Canterbury_The_Calgary_Corpus.tar.cc.nz
    07/02/2017  10:06 AM           768,579 University_of_Canterbury_The_Calgary_Corpus.tar.co.nz
    07/02/2017  10:06 AM         1,142,536 University_of_Canterbury_The_Calgary_Corpus.tar.L17.LZSSE2
    07/02/2017  10:06 AM           853,356 University_of_Canterbury_The_Calgary_Corpus.tar.L9Dict512.xz
    07/02/2017  10:06 AM         1,028,558 University_of_Canterbury_The_Calgary_Corpus.tar.M49Block256.lz5
    07/02/2017  10:06 AM         1,079,861 University_of_Canterbury_The_Calgary_Corpus.tar.method29.zpaq
    07/02/2017  10:06 AM           683,957 University_of_Canterbury_The_Calgary_Corpus.tar.method59.zpaq
    07/02/2017  10:06 AM         1,015,733 University_of_Canterbury_The_Calgary_Corpus.tar.MX9.zip
    07/02/2017  10:06 AM           853,789 University_of_Canterbury_The_Calgary_Corpus.tar.MX9Dict512.7z
    08/27/2017  04:42 PM         1,223,800 University_of_Canterbury_The_Calgary_Corpus.tar.Nakamichi
    07/02/2017  10:06 AM           784,803 University_of_Canterbury_The_Calgary_Corpus.tar.O16.PPMd_varI
    07/02/2017  10:06 AM           796,337 University_of_Canterbury_The_Calgary_Corpus.tar.O6.PPMd_varI
    07/02/2017  10:06 AM           825,124 University_of_Canterbury_The_Calgary_Corpus.tar.ST6Block512.bsc
    
    04/06/2017  08:44 PM           152,089 alice29.txt
    08/27/2017  12:48 PM            56,526 alice29.txt.L17.LZSSE2
    08/27/2017  12:48 PM            67,579 alice29.txt.Nakamichi
    
    04/06/2017  08:44 PM           260,569 Fahrenheit_451_-_Ray_Bradbury.txt
    08/27/2017  12:50 PM           100,387 Fahrenheit_451_-_Ray_Bradbury.txt.L17.LZSSE2
    08/27/2017  12:50 PM           115,818 Fahrenheit_451_-_Ray_Bradbury.txt.Nakamichi
    
    04/06/2017  08:44 PM         1,820,160 Fleurs_du_mal.tar
    08/27/2017  10:58 AM           557,422 Fleurs_du_mal.tar.L17.LZSSE2
    08/27/2017  10:58 AM           414,512 Fleurs_du_mal.tar.Nakamichi
    
    04/06/2017  08:44 PM             8,798 IBM_Deep_Blue_vs_Kasparov.pgn
    08/27/2017  12:49 PM             4,803 IBM_Deep_Blue_vs_Kasparov.pgn.L17.LZSSE2
    08/27/2017  12:49 PM             5,153 IBM_Deep_Blue_vs_Kasparov.pgn.Nakamichi
    
    04/06/2017  08:44 PM         3,680,256 Resident_Evil_(10-books).tar
    08/27/2017  02:49 PM         1,343,243 Resident_Evil_(10-books).tar.L17.LZSSE2
    08/27/2017  02:49 PM         1,289,461 Resident_Evil_(10-books).tar.Nakamichi
    
    04/06/2017  08:44 PM           698,072 The_Death_Ship_-_B.Traven.pdf.txt
    08/27/2017  12:55 PM           258,683 The_Death_Ship_-_B.Traven.pdf.txt.L17.LZSSE2
    08/27/2017  12:55 PM           277,889 The_Death_Ship_-_B.Traven.pdf.txt.Nakamichi
    
    07/02/2017  10:06 AM         4,445,260 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt
    08/27/2017  12:44 PM         1,390,894 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt.L17.LZSSE2
    08/27/2017  12:44 PM         1,356,427 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt.Nakamichi
    
    opened by Sanmayce 3
  • Fix fallthrough warning in gcc

    Fix fallthrough warning in gcc

    Hi. While trying to make C-Blosc2 as free of warnings as possible I fixed a fall-through warning in Lizard that only shows up in GCC. The fix here works for GCC, Clang and MSVC (for a detailed discussion see https://dzone.com/articles/implicit-fallthrough-in-gcc-7).

    opened by FrancescAlted 2
  • Negative size passed to memcpy resulting in memory corruption (lib/lizard_decompress_liz.h:207)

    Negative size passed to memcpy resulting in memory corruption (lib/lizard_decompress_liz.h:207)

    Hey there,

    I have come across a vulnerability in the lizard decompressor, whereby a negative size integer is passed to memcpy(). This occurs in lib/lizard_decompress_liz.h:207.

    memcpy(op, ctx->literalsPtr, length);
    

    Since memcpy() takes the length and treats it as a size_t, this results in an excessive copy from ctx->literalsPtr into op.

    void *memcpy(void *dest, const void *src, size_t n);
    

    This was found while fuzzing commit 6a1ed71450148c8aed57de3179b1bdd81800bada (current latest as of this date).

    By controlling the number of bytes passed as length, this results in memory corruption which can lead to execution control.

    Find the POC file here: poc.zip.

    As follows is the ASAN output:

    $ ./lizard_afl -d crash.file /dev/null
    =================================================================
    ==26522==ERROR: AddressSanitizer: negative-size-param: (size=-8)
        #0 0x4ae273 in __asan_memcpy (/root/lizard/lizard_afl+0x4ae273)
        #1 0x51a9b2 in Lizard_decompress_LIZv1 /root/lizard/programs/../lib/lizard_decompress_liz.h:207:5
        #2 0x51a9b2 in Lizard_decompress_generic /root/lizard/programs/../lib/lizard_decompress.c:241
        #3 0x51a9b2 in Lizard_decompress_safe_usingDict /root/lizard/programs/../lib/lizard_decompress.c:362
        #4 0x634386 in LizardF_decompress /root/lizard/programs/../lib/lizard_frame.c:1197:31
        #5 0x645fc1 in LizardIO_decompressLizardF /root/lizard/programs/lizardio.c:676:26
        #6 0x645fc1 in selectDecoder /root/lizard/programs/lizardio.c:767
        #7 0x645fc1 in LizardIO_decompressSrcFile /root/lizard/programs/lizardio.c:803
        #8 0x644379 in LizardIO_decompressDstFile /root/lizard/programs/lizardio.c:831:5
        #9 0x643b80 in LizardIO_decompressFilename /root/lizard/programs/lizardio.c:850:30
        #10 0x64caf8 in main /root/lizard/programs/lizardcli.c:560:32
        #11 0x7f4d32b69f44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
        #12 0x419cb5 in _start (/root/lizard/lizard_afl+0x419cb5)
    
    0x7f4d33c1e86e is located 110 bytes inside of 131072-byte region [0x7f4d33c1e800,0x7f4d33c3e800)
    allocated by thread T0 here:
        #0 0x4c418c in calloc (/root/lizard/lizard_afl+0x4c418c)
        #1 0x636a0b in LizardF_decodeHeader /root/lizard/programs/../lib/lizard_frame.c:840:33
    
    SUMMARY: AddressSanitizer: negative-size-param (/root/lizard/lizard_afl+0x4ae273) in __asan_memcpy
    ==26522==ABORTING
    

    In the example above, with a memcpy size of -8, this value is interpreted as an unsigned integer and the memcpy continues until there is a page fault due to unmapped memory and the process terminates:

    root at s127422 in ~/lizard (lizard●●)
    $ ./lizard -d crash.file /dev/null
    [1]    17128 segmentation fault  ./lizard -d crash.file /dev/null
    

    To demonstrate that the memcpy size value can be controlled by an attacker, I have attached a series of crashes that result from a differing memcpy size value. lizard_crashes.zip

    opened by glen-mac 2
  • Valgrind complains about

    Valgrind complains about "Conditional jump or move depends on uninitialised value(s)"

    Hi,

    I am a happy user of the Lizard codec (really shine in some of my use cases inside Blosc). However, lately I am getting some complains from valgrind (they do not happen at all with the other codecs inside Blosc, like LZ4, Zlib or Zstd). The final result is fine; do you think this is something to be worried about? Using Lizard v1.0 here on a Linux box (Ubuntu 16.04).

    Thanks for your fine work!

    $ valgrind tests/test_btune 
    ==30189== Memcheck, a memory error detector
    ==30189== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
    ==30189== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
    ==30189== Command: tests/test_btune
    ==30189== 
    ==30189== Thread 5:
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E926B8: Lizard_compress_fastBig (lizard_parser_fastbig.h:85)
    ==30189==    by 0x4E926B8: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E926B8: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E92AC6: Lizard_compress_fastBig (lizard_parser_fastbig.h:138)
    ==30189==    by 0x4E92AC6: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E92AC6: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Thread 4:
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E92AD1: Lizard_compress_fastBig (lizard_parser_fastbig.h:138)
    ==30189==    by 0x4E92AD1: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E92AD1: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E926BF: Lizard_compress_fastBig (lizard_parser_fastbig.h:85)
    ==30189==    by 0x4E926BF: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E926BF: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Thread 3:
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E92AE5: Lizard_compress_fastBig (lizard_parser_fastbig.h:138)
    ==30189==    by 0x4E92AE5: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E92AE5: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E92AF0: Lizard_compress_fastBig (lizard_parser_fastbig.h:140)
    ==30189==    by 0x4E92AF0: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E92AF0: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E92B02: Lizard_compress_fastBig (lizard_parser_fastbig.h:142)
    ==30189==    by 0x4E92B02: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E92B02: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Use of uninitialised value of size 8
    ==30189==    at 0x4E92B08: Lizard_compress_fastBig (lizard_parser_fastbig.h:143)
    ==30189==    by 0x4E92B08: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E92B08: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E926D1: Lizard_compress_fastBig (lizard_parser_fastbig.h:85)
    ==30189==    by 0x4E926D1: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E926D1: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E926D8: Lizard_compress_fastBig (lizard_parser_fastbig.h:87)
    ==30189==    by 0x4E926D8: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E926D8: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Conditional jump or move depends on uninitialised value(s)
    ==30189==    at 0x4E926EB: Lizard_compress_fastBig (lizard_parser_fastbig.h:89)
    ==30189==    by 0x4E926EB: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E926EB: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    ==30189== Use of uninitialised value of size 8
    ==30189==    at 0x4E926ED: Lizard_compress_fastBig (lizard_parser_fastbig.h:90)
    ==30189==    by 0x4E926ED: Lizard_compress_generic (lizard_compress.c:515)
    ==30189==    by 0x4E926ED: Lizard_compress_extState (lizard_compress.c:592)
    ==30189==    by 0x4E93367: Lizard_compress (lizard_compress.c:602)
    ==30189==    by 0x4E5025D: lizard_wrap_compress (blosc.c:398)
    ==30189==    by 0x4E5025D: blosc_c (blosc.c:733)
    ==30189==    by 0x4E50F0A: t_blosc (blosc.c:2155)
    ==30189==    by 0x54CA6B9: start_thread (pthread_create.c:333)
    ==30189== 
    
    opened by FrancescAlted 2
  • LZ5v2: Levels are a bit confusing.

    LZ5v2: Levels are a bit confusing.

    As of 1.5 RC (9e8d558d81e958407edf676329972174e0c9b0de) levels are somewhat confusing.

    Normally when app-level dev calls library, it is expected higher level gives better ratio at cost of slower compression speed. Builtin help of LZ5 also suggests it is a case:

    -1...-18 : compression level; higher number == more compression but slower

    However, it isn't like this in LZ5 v2:

    $ ./lz5 -b9 lz5 9#lz5 : 192320 -> 102547 (1.875), 1.2 MB/s , 404.0 MB/s

    $ ./lz5 -b10 lz5 10#lz5 : 192320 -> 113877 (1.689), 11.1 MB/s , 358.8 MB/s

    So increasing level from 9 to 10 quite unxepectedly boosts compression speed and drops ratio.

    I guess either levels may need to be fine-tuned to behave like expected or, maybe, devs should get explicit knowledge these are 2 different sub-algos, not sure which option is better.

    opened by xcrh 2
  • Add x86 Windows builds to releases

    Add x86 Windows builds to releases

    For reasons unknown to man, x86 Windows is still a thing. I didn't notice any issues when building the 32bit version. Please add x86 builds to the releases for people who don't want to build from source.

    Thanks!

    opened by turbo 2
  • The status of the project.

    The status of the project.

    The library is very impressive as it looks like a pareto frontier.

    But I see there are no updates in about half a year. E.g. trivial fix for UBSan did not get merged: https://github.com/inikep/lizard/pull/33

    It makes me wondering about the perspectives of this project.

    opened by alexey-milovidov 1
  • The library is not annotated for Memory Sanitizer.

    The library is not annotated for Memory Sanitizer.

    Example:

    ==341==WARNING: MemorySanitizer: use-of-uninitialized-value
        #0 0x41021543 in Lizard_compress_fastSmall obj-x86_64-linux-gnu/../contrib/lizard/lib/lizard_parser_fastsmall.h:83:45
        #1 0x41021543 in Lizard_compress_generic obj-x86_64-linux-gnu/../contrib/lizard/lib/lizard_compress.c:506:19
        #2 0x41021543 in Lizard_compress_extState obj-x86_64-linux-gnu/../contrib/lizard/lib/lizard_compress.c:592:12
        #3 0x4102c480 in Lizard_compress obj-x86_64-linux-gnu/../contrib/lizard/lib/lizard_compress.c:602:13
    
    opened by alexey-milovidov 1
  • Symbol names clash with ZSTD

    Symbol names clash with ZSTD

    Some symbols like HUF_decompress, FSE_decompress have identical names to the same symbols in ZSTD library. This prevents using Lizard and ZSTD in the same binary.

    opened by alexey-milovidov 1
  • Fixed heap-buffer-overflow error when decoding variable literal length.

    Fixed heap-buffer-overflow error when decoding variable literal length.

    Heap-buffer-overflow READ 2 · Lizard_decompress_safe

        #0 0x760da0 in MEM_read16 c-blosc2/internal-complibs/lizard-1.0/entropy/mem.h:146:14
        #1 0x760da0 in MEM_readLE16 c-blosc2/internal-complibs/lizard-1.0/entropy/mem.h:226:16
        #2 0x760da0 in MEM_readLE24 c-blosc2/internal-complibs/lizard-1.0/entropy/mem.h:246:12
        #3 0x760da0 in Lizard_decompress_LIZv1 c-blosc2/internal-complibs/lizard-1.0/lizard_decompress_liz.h:70:34
        #4 0x760da0 in Lizard_decompress_generic c-blosc2/internal-complibs/lizard-1.0/lizard_decompress.c:241:19
        #5 0x760da0 in Lizard_decompress_safe c-blosc2/internal-complibs/lizard-1.0/lizard_decompress.c:269:12
        #6 0x5bab43 in lizard_wrap_decompress c-blosc2/blosc/blosc2.c:407:12
        #7 0x5bab43 in blosc_d c-blosc2/blosc/blosc2.c:1118:18
        #8 0x5b098c in serial_blosc c-blosc2/blosc/blosc2.c:1219:16
        #9 0x5b098c in do_job c-blosc2/blosc/blosc2.c:1377:15
        #10 0x5b7389 in blosc_run_decompression_with_context c-blosc2/blosc/blosc2.c:2159:13
        #11 0x5b7d4b in blosc2_decompress c-blosc2/blosc/blosc2.c:2227:12
        #12 0x5613c3 in LLVMFuzzerTestOneInput c-blosc2/tests/fuzz/fuzz_decompress_chunk.c:34:5
    
    SUMMARY: AddressSanitizer: heap-buffer-overflow (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds_c-blosc2_c26e8220b80bfc0b5ac207ffcb887c5ee8c63325/revisions/decompress_chunk_fuzzer+0x760da0)
    

    https://oss-fuzz.com/testcase-detail/5137671143555072

    Originally submitted to Blosc/c-blosc2#178.

    Testcase binary file liz.biz zipped up: liz.zip

    opened by nmoinvaz 1
  • Binary?

    Binary?

    Hi, now that I have access to two modern and fast (more in the link) machines, could @inikep you provide the lzbench binary, please. It would be great if updated with latest Brotli (the 30bit window) and Zstd!

    Wanna benchmark 2+GB testfiles: http://www.sanmayce.com/Nakamichi/index.html#2020Dec19

    opened by Sanmayce 0
  • Compression algorithm explanation

    Compression algorithm explanation

    Hi, @inikep !

    I really appreciate your works and I want to ask you: Do you have any clear explanations of ZSTD, LZ4 and Lizard? I want to explain their internals in my coursework.All references would be included.

    opened by fibersel 1
Releases(v1.0)
  • v1.0(Mar 15, 2017)

    Changes:

    • LZ5 v2.0 was renamed to Lizard
    • improved compression speed with many small files

    Lizard v1.0 contains 4 compression methods:

    • fastLZ4 : compression levels -10...-19 are designed to give better decompression speed than LZ4 i.e. over 2000 MB/s
    • LIZv1 : compression levels -20...-29 are designed to give better ratio than LZ4 keeping 75% decompression speed
    • fastLZ4 + Huffman : compression levels -30...-39 add Huffman coding to fastLZ4
    • LIZv1 + Huffman : compression levels -40...-49 give the best ratio (comparable to zlib and low levels of zstd/brotli) at decompression speed of 1000 MB/s
    Source code(tar.gz)
    Source code(zip)
    lizard-1.0-windows.zip(388.38 KB)
  • v2.0(Feb 5, 2017)

  • v1.5(Aug 16, 2016)

    Changes:

    • introduced compatibility with Visual C++ 2010 and newer
    • attached Visual Studio 2010 project
    • thoroughly tested with 21 Travis CI and 7 AppVeyor CI tests
    • fixed bug with reusing a context in lz5frame.c (LZ5F_compressBegin and LZ5_compress_HC_continue)
    • fixed rare bug in match finder (concerns levels 4 - 15)
    Source code(tar.gz)
    Source code(zip)
    lz5.exe(420.75 KB)
    lz5_32.exe(223.02 KB)
  • v1.4.1(Feb 17, 2016)

  • v1.4(Feb 11, 2016)

    Changes:

    • improved: levels from 13 to 15 (maximum compression ratio)
    • added a new parser: LZ5HC_optimal_price_bt
    • updated documentation: lz5_Block_format.md and lz5_Frame_format.md
    • changed lz5.exe: the "-B" option with block size [1-7] = 64KB, 256KB, 1MB, 4MB, 16MB, 64MB, 256MB (default : 4 = 4MB)
    Source code(tar.gz)
    Source code(zip)
    lz5.exe(215.57 KB)
  • v1.3.3(Jan 5, 2016)

  • r132(Dec 3, 2015)

    • improved compression ratio
    • added: new parsers: LZ5HC_fast, LZ5HC_price_fast, LZ5HC_lowest_price
    • added: a special 1-byte codeword for the last occured offset
    • added: support for 3-byte long matches (MINMATCH = 3)
    Source code(tar.gz)
    Source code(zip)
    lz5.exe(202.95 KB)
  • r131b(Nov 2, 2015)

Owner
Przemyslaw Skibinski
Przemyslaw Skibinski
Heavily optimized zlib compression algorithm

Optimized version of longest_match for zlib Summary Fast zlib longest_match function. Produces slightly smaller compressed files for significantly fas

Konstantin Nosov 119 Sep 5, 2022
Parallel, indexed xz compressor

pixz Pixz (pronounced pixie) is a parallel, indexing version of xz. Repository: https://github.com/vasi/pixz Downloads: https://github.com/vasi/pixz/r

Dave Vasilevsky 605 Sep 19, 2022
A portable, simple zip library written in C

A portable (OSX/Linux/Windows), simple zip library written in C This is done by hacking awesome miniz library and layering functions on top of the min

Kuba Podgórski 982 Sep 22, 2022
zlib replacement with optimizations for "next generation" systems.

zlib-ng zlib data compression library for the next generation systems Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org) C

zlib-ng 9 Aug 31, 2022
lunatic is an ARM (32-bit) dynamic recompiler for low-level emulators.

lunatic lunatic is an ARM (32-bit) dynamic recompiler for low-level emulators. At the moment this is alpha level software and is not ready for general

fleroviux 82 Sep 12, 2022
Compression abstraction library and utilities

Squash - Compresion Abstraction Library

null 367 Sep 14, 2022
Multi-format archive and compression library

Welcome to libarchive! The libarchive project develops a portable, efficient C library that can read and write streaming archives in a variety of form

null 1.8k Sep 20, 2022
Superfast compression library

DENSITY Superfast compression library DENSITY is a free C99, open-source, BSD licensed compression library. It is focused on high-speed compression, a

Centaurean 980 Sep 13, 2022
data compression library for embedded/real-time systems

heatshrink A data compression/decompression library for embedded/real-time systems. Key Features: Low memory usage (as low as 50 bytes) It is useful f

Atomic Object 1.1k Sep 23, 2022
Small strings compression library

SMAZ - compression for very small strings ----------------------------------------- Smaz is a simple compression library suitable for compressing ver

Salvatore Sanfilippo 1k Sep 16, 2022
Fastest Integer Compression

TurboPFor: Fastest Integer Compression TurboPFor: The new synonym for "integer compression" ?? (2019.11) ALL functions now available for 64 bits ARMv8

powturbo 630 Sep 8, 2022
Compile and execute C "scripts" in one go!

c "There isn't much that's special about C. That's one of the reasons why it's fast." I love C for its raw speed (although it does have its drawbacks)

Ryan Jacobs 2k Sep 24, 2022
distributed builds for C, C++ and Objective C

distcc -- a free distributed C/C++ compiler system by Martin Pool Current Documents: https://distcc.github.io/ Formally http://distcc.org/ "pump" func

distcc 1.7k Sep 18, 2022
Roaring bitmaps in C (and C++)

CRoaring Portable Roaring bitmaps in C (and C++) with full support for your favorite compiler (GNU GCC, LLVM's clang, Visual Studio). Included in the

Roaring bitmaps: A better compressed bitset 1.1k Sep 19, 2022
New generation entropy codecs : Finite State Entropy and Huff0

New Generation Entropy coders This library proposes two high speed entropy coders : Huff0, a Huffman codec designed for modern CPU, featuring OoO (Out

Yann Collet 1.1k Sep 21, 2022
Easing the task of comparing code generated by cc65, vbcc, and 6502-gcc

6502 C compilers benchmark Easing the way to compare code generated by cc65, 6502-gcc, vbcc, and KickC. This repository contains scripts to: Compile t

Sylvain Gadrat 17 Sep 4, 2022
Secure ECC-based DID intersection in Go, Java and C.

SecureUnionID Secure ECC-based DID intersection. ABSTRACT This project is used to protect device ID using Elliptic Curve Cryptography algorithm. The d

Volcengine 19 Aug 22, 2022
nanoc is a tiny subset of C and a tiny compiler that targets 32-bit x86 machines.

nanoc is a tiny subset of C and a tiny compiler that targets 32-bit x86 machines. Tiny? The only types are: int (32-bit signed integer) char (8-

Ajay Tatachar 16 Feb 13, 2022
Smaller C is a simple and small single-pass C compiler

Smaller C is a simple and small single-pass C compiler, currently supporting most of the C language common between C89/ANSI C and C99 (minus some C89 and plus some C99 features).

Alexey Frunze 1.1k Sep 21, 2022