Is TurboPFor a library for int compression? If yes, why does the page look like some compare tool for different libs. Put the docs/hello world right there, not the compare results. It's extremely painful to find anything how to use the lib (I hope it's wasn't assumed that users should read icbench or source of any other tool to decifir how to use the lib).

I use visual studio, and VS build doesn't work at all. Also, that approach that some files result in multiple different output obj files doesn't help overall: it would be better if the lib had tiny wrappers that included those c-files and set required defines to modify compilation.

Even after I made it build on VS it absolutely doesn't work and completely fails for me. I'm not even sure what function I need to use to compress. After that huge table of irrelevant info, if anybody ever gets as far as "Function usage" section, you'll see some cryptic explanation that doesn't seem to be current anymore. From that section, it seems that I need p4fmencXXX but that doesn't exist (nothing p4fm exist at all). Why is that there that pack/unpack thing doing, to confuse people? From the docs, it seems like p4packXXX should be a valid function, while it's not.

After digging, it seem that I should try p4enc32, but so far I get stack corruption, and it doesn't seem like it could ever work at all. It crashes for me in _p4enc32 on line while(i != n) MISS; In my case I call p4enc32 with an array of 2731 uints. That while loop will clearly corrupt stack, because the MISS expands to _in[i] = in[i], that is, it will attempt to assign _in[2730] because that loop will run up to the length of the input n=2731. I don't get it, how come that code could even work, as it writes to stack array of 287 elements?! Also, it seems like there are no checks, I don't see a single assert anywhere to check/show conditions/expectations, while in that function it had to make that check!

While making some changes in my code, I tried to verify that the resulting binary data (compressed with p4nzenc256v32) is identical. To my surprise, the data had many changes as if something corrupted it. I tried to dump hex of compressed data and it clearly had a few modified bytes here and there. What's even more surprising, when I uncompressed the data using p4nzdec256v32 output from these two different buffers was identical. Any idea why, is that normal? If so, what makes it produce different outputs with identical inputs?

save the file as ictest.c in the TurboPFor directory and type

make ictest ./ictest

output: compress size is 6 uncompress size is 6

#include <stdio.h>
#define NTURBOPFOR_DAC
#include "vp4.h"

#define P4NENC_BOUND(n) ((n+127)/128+(n+32)*sizeof(uint32_t))

int main(int argc, char* argv[]) {
      printf("Hello TurboPFor\n");
      int ar[32] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
      unsigned elnum = 10;
      unsigned char* compress_buf =  malloc(P4NENC_BOUND(elnum));
      int *uncompress_buf =  malloc((elnum+32)*sizeof(ar[0]));
  
      size_t compress_size = p4nenc32((uint32_t*)ar, elnum, compress_buf);
      printf("compress size is %lu\n", compress_size);
  
      size_t uncompress_size = p4ndec32(compress_buf, elnum, (uint32_t*)uncompress_buf);
      printf("uncompress size is %lu\n", uncompress_size);
}

#include "conf.h"
#include "bitpack.h"
#include "vint.h"		
#include "bitutil.h"
#include "vp4.h"


typedef uint64_t LONG;


int main(LONG argc, char** argv) {
	LONG *array = (LONG *)malloc(sizeof(LONG)*5);
	array[0] = 24520120;
	array[1] = 29620120;
	array[2] = 42420120;
	array[3] = 20124222;
	array[4] = 4294967295;

	unsigned char* out = (unsigned char *)malloc(5*10*sizeof(unsigned char));
	unsigned char * op = p4enc64(array, 5, out);
	printf("%d\n",(int)(op-out) );
	

	LONG *capacity = (LONG *)malloc(sizeof(LONG)*5);
	unsigned char * op2 =  p4dec64(out, 5, capacity);
	printf("%d\n",(int)(op2-out) );

}

When attempting to use p4enc64 in above code, the following error is raised.

malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.

Program received signal SIGABRT, Aborted. 0x00007ffff700dc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.

No issue occurs with uint32_t and also when malloc is not used.

Please correct me if there is any issue with the code. I am trying to use TurboPFor in my code for compressing 64-bit unsorted integer arrays. Also I am having trouble with allocating the right buffer size for the compressed array.What is the minimum buffer size I can allocate safely while using p4enc64 for integer array of size n?

Just to make sure it's tracked properly, decoding speed reduced by more than 10% for p4nzdec256v32 (my personal project seems to have best results with this function).

9dad490 doesn't have the problem, 5dff9d3f has the problem. It's absolutely clearly almost 10% slower for entire test where p4nzdec256v32 connsumes perhaps 25% of the cpu time of the entire test.

Times of my tests with these TurboPFor versions: 9dad490 - 16.605sec ~MyAlgo profiler data:

p4nzdec256v32 roughly takes 3.560sec (21.44% of 16.605sec)

5dff9d3f - 17.636sec ~MyAlgo profiler data:

p4nzdec256v32 roughly takes 4.213sec (23.89% of 17.636sec) These tests are performed with absolutely identical code, simply TurboPFor version changed. Also, the test uses 3 threads (according to the debugger, so, actual performance hit in p4nzdec256v32 seems to be way more than 10%, and might easily be in the range of 20%-30%). Also, times were measured without profiler (just to make it clear, it was fully optimized runs of 64-bit builds of TurboPFor)

Unfortunately, due to the way TurboPFor is managed, it's in permanently broken state and I wasn't able to bisect to find offending commit: if I try to check out anything in between, it's always broken and doesn't build with all kinds of compilation errors. I'd strongly recommend adjusting your approach: it's not suitable for projects that are worked on and used by more than one person.

ctz64 for non-windows builds is:

for example:

#define ctz64(x) __builtin_ctzll(x)

while for windows it's:

static inline int ctz64(uint64_t x) { unsigned long z; _BitScanForward64(&z, x); return x?z:64; }

__builtin_ctzll isn't defined for x=0, however, for windows builds it does the check: x?z:64.

https://github.com/powturbo/TurboPFor/blob/4df4bcea29b670dab3acb985aac83a7562bfa2eb/conf.h#L65

I have just seen your project and I see that you are using Blosc for comparison. For completeness, I would recommend to include other Blosc codecs as they give different results that may be a good fit in different scenarios.

Below, I have selected several combinations that works well for this problem, and here are my results (Linux, GCC 4.9.2, Xeon E3-1240 v3 @ 3.40GHz):

$ time LD_LIBRARY_PATH=/home/francesc/c-blosc/build/blosc ./icbench -a1.5 -m0 -M255 -n100m
zipf alpha=1.50 range[0..255].n=100000000

bits histogram:0:40.20% 1:14.22% 2:12.75% 3:10.28% 4:7.77% 5:5.69% 6:4.09% 7:2.92% 8:2.07%
62064288    15.52    4.97      27.85          130.24   blosc (zlib, nthreads=1, clevel=3)
62064288    15.52    4.97      42.85          214.16   blosc (zlib, nthreads=4, clevel=3)
63392801    15.85    5.07     396.74         1412.46   TurboPFor
77187330    19.30    6.17      51.44          663.17   blosc (lz4hc, nthreads=1, clevel=3)
77187330    19.30    6.17      78.52          984.59   blosc (lz4hc, nthreads=4, clevel=3)
101473443    25.37    8.12     862.60          896.15  blosc (lz4, nthreads=1, clevel=3)
101473443    25.37    8.12    1193.86         1392.87  blosc (lz4, nthreads=2, clevel=3)
101473443    25.37    8.12    1608.41         1549.34  blosc (lz4, nthreads=4, clevel=3)
102491006    25.62    8.20     595.36         1815.32  blosc (blosclz, nthreads=1, clevel=3)
102491006    25.62    8.20     949.23         2049.91  blosc (blosclz,  nthreads=2, clevel=3)
102491006    25.62    8.20    1293.24         1873.03  blosc (blosclz,  nthreads=4, clevel=3)

The timings with clang 3.5 are very close to these, so I am not reproducing them.

By the way, very nice work! I did not realized that compressing integers was that important!

https://github.com/powturbo/TurboPFor/blob/5dff9d3fc120fafa13cedbc8ea0f249e8291215a/vs/vs2017/TurboPFor.vcxproj#L92

If you remove USE_SSE & USE_AVX2, why don't you grep entire repo to see if you have left overs here and there?..

$ javah -jni jic or $ javah -jni jic.java
fails - no such file. Could you please add jic.h to the repository ? So far, I've tried to make a shared library without it and succeeded with following make file: differences are added flag - -fPIC and

    #$(CC) $(OBJS) -lm -o -shared libic.so $(LFLAGS)
    $(CC) -shared -Wl,-soname,libic.so -o libic.so.1 -lm $(LFLAGS) $(OBJS)

# Linux: "export CC=clang" windows mingw: "set CC=gcc" or uncomment one of following lines
# CC=clang
# CC=gcc

MARCH=-march=native
#MARCH=-msse2
CFLAGS=-DNDEBUG -fstrict-aliasing -m64 $(MARCH) -Iext -fPIC 

UNAME := $(shell uname)
ifeq ($(UNAME), Linux)
LIBTHREAD=-lpthread
LIBRT=-lrt
else
CC=gcc
endif

BIT=./
all: icbench idxcr idxqry idxseg libic

bitpack.o: $(BIT)bitpack.c $(BIT)bitpack.h $(BIT)bitpack64_.h
    $(CC) -O2 $(CFLAGS) -c $(BIT)bitpack.c

bitpackv.o: $(BIT)bitpackv.c $(BIT)bitpack.h $(BIT)bitpackv32_.h
    $(CC) -O2 $(CFLAGS) -c $(BIT)bitpackv.c

vp4dc.o: $(BIT)vp4dc.c
    $(CC) -O3 $(CFLAGS) -funroll-loops -c $(BIT)vp4dc.c

vp4dd.o: $(BIT)vp4dd.c
    $(CC) -O3 $(CFLAGS) -funroll-loops -c $(BIT)vp4dd.c

varintg8iu.o: $(BIT)ext/varintg8iu.c $(BIT)ext/varintg8iu.h 
    $(CC) -O2 $(CFLAGS) -c -funroll-loops -std=c99 $(BIT)ext/varintg8iu.c

idxqryp.o: $(BIT)idxqry.c
    $(CC) -O3 $(CFLAGS) -c $(BIT)idxqry.c -o idxqryp.o

SIMDCOMPD=ext/simdcomp/
SIMDCOMP=$(SIMDCOMPD)bitpacka.o $(SIMDCOMPD)src/simdintegratedbitpacking.o $(SIMDCOMPD)src/simdcomputil.o $(SIMDCOMPD)src/simdbitpacking.o

#LIBFOR=ext/for/for.o
MVB=ext/MaskedVByte/src/varintencode.o ext/MaskedVByte/src/varintdecode.o

# Lzturbo not included
#LZT=../lz/lz8c0.o ../lz/lz8d.o ../lz/lzbc0.o ../lz/lzbd.o

# blosc. Set the env. variable "EXT=blosc" to include 
#EXT=blosc
ifeq ($(EXT), blosc)
B=ext/
CFLAGS+=-DSHUFFLE_SSE2_ENABLED -DHAVE_LZ4 -DHAVE_ZLIB -Iext/
LFLAGS+=-lpthread 
BLOSC=$(B)lz4hc.o $(B)c-blosc/blosc/blosc.o $(B)c-blosc/blosc/blosclz.o $(B)c-blosc/blosc/shuffle.o $(B)c-blosc/blosc/shuffle-generic.o $(B)c-blosc/blosc/shuffle-sse2.o
endif

LZ4=ext/lz4.o 

#ZLIB=-lz

#BSHUFFLE=ext/bitshuffle/src/bitshuffle.o

OBJS=icbench.o bitutil.o vint.o bitpack.o bitunpack.o eliasfano.o vsimple.o vp4dd.o vp4dc.o varintg8iu.o bitpackv.o bitunpackv.o $(TRANSP) ext/simple8b.o transpose.o $(BLOSC) $(SIMDCOMP) $(LIBFOR) $(LZT) $(LZ4) $(MVB) $(ZLIB) $(BSHUFFLE)

icbench: $(OBJS)
    $(CC) $(OBJS) -lm -o icbench $(LFLAGS)

libic: $(OBJS)
    #$(CC) $(OBJS) -lm -o -shared libic.so $(LFLAGS)
    $(CC) -shared -Wl,-soname,libic.so -o libic.so.1 -lm $(LFLAGS) $(OBJS)

idxseg:   idxseg.o
    $(CC) idxseg.o -o idxseg

ifeq ($(UNAME), Linux)
para: CFLAGS += -DTHREADMAX=32  
para: idxqry
endif

idxcr:   idxcr.o bitpack.o vp4dc.o bitutil.o
    $(CC) idxcr.o bitpack.o bitpackv.o vp4dc.o bitutil.o -o idxcr $(LFLAGS)

idxqry:   idxqry.o bitunpack.o vp4dd.o bitunpackv.o bitutil.o
    $(CC) idxqry.o bitunpack.o bitunpackv.o vp4dd.o bitutil.o $(LIBTHREAD) $(LIBRT) -o idxqry $(LFLAGS)

.c.o:
    $(CC) -O3 $(CFLAGS) $< -c -o $@

.cc.o:
    $(CXX) -O3 -DNDEBUG -std=c++11 $< -c -o $@

.cpp.o:
    $(CXX) -O3 -DNDEBUG -std=c++11 $< -c -o $@

clean:
    @find . -type f -name "*\.o" -delete -or -name "*\~" -delete -or -name "core" -delete

cleanw:
    del /S ..\*.o
    del /S ..\*~

Hello,

First of all, sorry for my english,

Currently, I am finishing a thesis, and your implementation is cited in the bibliography. For doing this I would like to know some data about your code,

• What are the differences between your code (the PFOR codec) and other state-of-the-art implementations like D. Lemire et al? http://arxiv.org/pdf/1401.6399v12.pdf
  • Are the optimizations based on the implementation?, or has the algorithm changed?
• Is this code part of any published work (e.g article, thesis, etc) which can be cited using a reference manager?
• What is the licence of your code? (in the case it has any)

Thank-you, Julian

There are padfloat32() and padfloat64(), but no padfloat16(), is it straightforward to create a float16 version based on the float32 version?

Also, what if I do the lossy compression for float16 data using icapp -f4 -g.001 float16data?

When compiling with clang I get tons of warnings accessing elements past the end of the array bounds inside transcode.c

Looking into specifics of the warnings, I think all of them are legit bugs that should be addressed. All of the warnings are related to ov variables declared on lines 603, 731, 849, 1107. For example, when compiling AVX version I get these warnings:

Build started...
1>------ Build started: Project: TurboPFor, Configuration: Release x64 ------
1>In file included from ..\transpose_avx2.c:2:
1>In file included from ../../transpose.c:94:
1>../../transpose.c(681,5): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(681,122): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(681,132): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(681,63): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(681,171): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(675,83): message : expanded from macro 'mm256_packus_epi16'
1>C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\Llvm\x64\lib\clang\15.0.1\include\avx2intrin.h(818,56): message : expanded from macro '_mm256_permute4x64_epi64'
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(681,144): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(682,5): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(682,122): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(682,132): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(682,63): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(682,171): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(675,83): message : expanded from macro 'mm256_packus_epi16'
1>C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\Llvm\x64\lib\clang\15.0.1\include\avx2intrin.h(818,56): message : expanded from macro '_mm256_permute4x64_epi64'
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(682,144): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(683,49): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(676,97): message : expanded from macro 'ST128'
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(683,67): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(676,97): message : expanded from macro 'ST128'
1>../../transpose.c(603,5): message : array 'ov' declared here
1>../../transpose.c(734,16): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(712,19): message : expanded from macro 'NBL'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,16): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(713,19): message : expanded from macro 'NBL'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,42): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(716,74): message : expanded from macro 'NB'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,42): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(716,49): message : expanded from macro 'NB'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,42): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(716,3): message : expanded from macro 'NB'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,42): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(717,74): message : expanded from macro 'NB'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,42): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(717,49): message : expanded from macro 'NB'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,42): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(717,3): message : expanded from macro 'NB'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,42): warning : array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(718,45): message : expanded from macro 'NB'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>../../transpose.c(734,42): warning : array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
1>../../transpose.c(718,55): message : expanded from macro 'NB'
1>../../transpose.c(731,5): message : array 'ov' declared here
1>Done building project "TurboPFor.vcxproj".

What is the difference between p4nenc256w32 and p4nenc256v32? The v stands for vertical, I believe. I cannot find any information about what 'vertical' means, nor what the 'w' variant might be. Looking at the decoder, instead of a corresponding decoder p4ndec256w32 there is p4nddec256w32 ('n' vs. 'nd'). Could you please document what these function are and how it makes sense that the 'w' encoder and 'w' decoder do not quite mirror each other? Thanks.

Hi,

You have demonstrated in the results that TurboPfor can reach a good compression rate with random accesses. However, I have only observed the enc/dec logic of whole blocks in the high level API in the source code.

May I ask about what kinds of API supports sequential iteration or random access, and what are some demos using them? Also, I want to ask whether 64 bit integers are supported or not?

Thanks a lot!

We have had some segmentation violations in our software using this library which seem to be caused by the P4 decompression. The decompression algorithm (in particular the function p4nd1dec64) often seems to read outside of the boundaries of the in array that is provided. I have seen occurrences of up to 80 bytes after the in array. Please note that the algorithm never seems to use these values, so the results are valid, independent of what data is to be found after the array.

I was already aware that the compression and decompression write to the out array in locations outside of the strictly required limits (as discussed in #26), but I wasn't aware that it also reads in out of bounds.

It would be very helpful if there would be variants of the function that never read outside of the boundaries, even if it costs some performance. Alternatively, it would be very helpful if we at least had documentation that describes how far outside of the boundaries the functions are reading.

How to reproduce?

• Build and run this program (linking against the library, obviously):

#include <cstdint>
#include <iostream>
#include <cassert>
#include "vp4.h"

using namespace std;

int main() {
    // the original, uncompressed set
    const size_t numData = 4;
    const uint64_t data[] = { 5246L, 12601L, 15235L, 18998L };

    // the compressed set
    // the data array is created on the heap, so that valgrind can find when reading out of the array bounds
    const size_t compressedSize = 8;
    unsigned char* compressed = new unsigned char[compressedSize] 
        { 0x94, 0x7e, 0x0d, 0xba, 0x3c, 0x49, 0xc9, 0x3a };

    // the buffer for decompression (enlarged as temporary memory)
    uint64_t decompressed[numData+32];
    p4nd1dec64(compressed, numData, decompressed);

    // verify that the results are correct
    for (int i = 0; i < numData; i++) {
        cout 
            << "data[" << i << "]=" << data[i] 
            << " uncompressed[" << i << "]=" << decompressed[i] 
            << endl;
        assert(data[i] == decompressed[i]);
    }

   return 0;
}

• Run the program using valgrind, which results in the following output (on may machine):

> valgrind ./testP4Error
==15779== Memcheck, a memory error detector
==15779== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==15779== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==15779== Command: ./testP4Error
==15779== 
==15779== Invalid read of size 8
==15779==    at 0x48E718: bitd1unpack64_13 (bitunpack_.h:3523)
==15779==    by 0x409D9F: p4nd1dec64 (vp4d.c:477)
==15779==    by 0x40129C: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779==  Address 0x5a25043 is 3 bytes inside a block of size 8 alloc'd
==15779==    at 0x4C2AC38: operator new[](unsigned long) (vg_replace_malloc.c:433)
==15779==    by 0x401246: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779== 
==15779== Invalid read of size 8
==15779==    at 0x48E754: bitd1unpack64_13 (bitunpack_.h:3523)
==15779==    by 0x409D9F: p4nd1dec64 (vp4d.c:477)
==15779==    by 0x40129C: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779==  Address 0x5a2504b is 3 bytes after a block of size 8 alloc'd
==15779==    at 0x4C2AC38: operator new[](unsigned long) (vg_replace_malloc.c:433)
==15779==    by 0x401246: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779== 
==15779== Invalid read of size 8
==15779==    at 0x48E7D7: bitd1unpack64_13 (bitunpack_.h:3523)
==15779==    by 0x409D9F: p4nd1dec64 (vp4d.c:477)
==15779==    by 0x40129C: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779==  Address 0x5a25053 is 11 bytes after a block of size 8 alloc'd
==15779==    at 0x4C2AC38: operator new[](unsigned long) (vg_replace_malloc.c:433)
==15779==    by 0x401246: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779== 
==15779== Invalid read of size 8
==15779==    at 0x48E85B: bitd1unpack64_13 (bitunpack_.h:3523)
==15779==    by 0x409D9F: p4nd1dec64 (vp4d.c:477)
==15779==    by 0x40129C: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779==  Address 0x5a2505b is 19 bytes after a block of size 8 alloc'd
==15779==    at 0x4C2AC38: operator new[](unsigned long) (vg_replace_malloc.c:433)
==15779==    by 0x401246: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779== 
==15779== Invalid read of size 8
==15779==    at 0x48E8E2: bitd1unpack64_13 (bitunpack_.h:3523)
==15779==    by 0x409D9F: p4nd1dec64 (vp4d.c:477)
==15779==    by 0x40129C: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779==  Address 0x5a25063 is 19 bytes after a block of size 16 in arena "client"
==15779== 
==15779== Invalid read of size 8
==15779==    at 0x48E975: bitd1unpack64_13 (bitunpack_.h:3523)
==15779==    by 0x409D9F: p4nd1dec64 (vp4d.c:477)
==15779==    by 0x40129C: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779==  Address 0x5a2506b is 27 bytes after a block of size 16 in arena "client"
==15779== 
==15779== Invalid read of size 4
==15779==    at 0x48E9FB: bitd1unpack64_13 (bitunpack_.h:3523)
==15779==    by 0x409D9F: p4nd1dec64 (vp4d.c:477)
==15779==    by 0x40129C: main (in /data/rpm/v93rnd_libic/src/test/testP4Error)
==15779==  Address 0x5a25073 is 29 bytes before an unallocated block of size 4,194,128 in arena "client"
==15779== 
data[0]=5246 uncompressed[0]=5246
data[1]=12601 uncompressed[1]=12601
data[2]=15235 uncompressed[2]=15235
data[3]=18998 uncompressed[3]=18998
==15779== 
==15779== HEAP SUMMARY:
==15779==     in use at exit: 8 bytes in 1 blocks
==15779==   total heap usage: 1 allocs, 0 frees, 8 bytes allocated
==15779== 
==15779== LEAK SUMMARY:
==15779==    definitely lost: 8 bytes in 1 blocks
==15779==    indirectly lost: 0 bytes in 0 blocks
==15779==      possibly lost: 0 bytes in 0 blocks
==15779==    still reachable: 0 bytes in 0 blocks
==15779==         suppressed: 0 bytes in 0 blocks
==15779== Rerun with --leak-check=full to see details of leaked memory
==15779== 
==15779== For lists of detected and suppressed errors, rerun with: -s
==15779== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 0 from 0)

These are the errors:

1>bitutil.h(193,31): warning C4391: 'uint16_t _mm_cvtsi128_si16(__m128i)': incorrect return type for intrinsic function, expected 'short' (compiling source file ..\..\bitpack.c)
1>bitutil.h(193,60): error C2169: '_mm_cvtsi128_si16': intrinsic function, cannot be defined (compiling source file ..\..\bitpack.c)