High-performance regular expression matching library

Overview

Hyperscan

Hyperscan is a high-performance multiple regex matching library. It follows the regular expression syntax of the commonly-used libpcre library, but is a standalone library with its own C API.

Hyperscan uses hybrid automata techniques to allow simultaneous matching of large numbers (up to tens of thousands) of regular expressions and for the matching of regular expressions across streams of data.

Hyperscan is typically used in a DPI library stack.

Documentation

Information on building the Hyperscan library and using its API is available in the Developer Reference Guide.

License

Hyperscan is licensed under the BSD License. See the LICENSE file in the project repository.

Versioning

The master branch on Github will always contain the most recent release of Hyperscan. Each version released to master goes through QA and testing before it is released; if you're a user, rather than a developer, this is the version you should be using.

Further development towards the next release takes place on the develop branch.

Get Involved

The official homepage for Hyperscan is at www.hyperscan.io.

If you have questions or comments, we encourage you to join the mailing list. Bugs can be filed by sending email to the list, or by creating an issue on Github.

If you wish to contact the Hyperscan team at Intel directly, without posting publicly to the mailing list, send email to [email protected].

Issues
  • Build windows DLL

    Build windows DLL

    Update Hyperscan to support building Windows DLL for release & debug.

    Also added a convenience function for validation, suitable for calling from Dotnet.

    opened by wurp 18
  • hyper scan library crash

    hyper scan library crash

    Hello,

    I have been developing an app with very simple function to compile a pattern and use the pattern to match which needs use Hyperscan library:

    After compiling the hyperscan library with following options: -DCMAKE_BUILD_TYPE=debug -DBOOST_ROOT=${BOOST_ROOT} -DBUILD_SHARED_LIBS=1 -DCMAKE_INSTALL_PREFIX=${USR_LIB_PATH}.

    The App runs as expected at my compiling machine[model name : Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz], and then I copied my app and libhs.so to another machine [model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz](Same X86 64bit machine). however it crashed every time at pattern load function when I execute the app.

    The compiling machine and running machine has difference cpuinfo, not sure it has relationship with this.

    Could some one can help me out. Thanks.

    Here are the crash info:

    rgdb.sh core.simplegrep.14441 
    file: compiled magic version [521] does not match with shared library magic version [524]
    gdb ./simplegrep -c core.simplegrep.14441
    GNU gdb (GDB) Amazon Linux (7.6.1-64.33.amzn1)
        Copyright (C) 2013 Free Software Foundation, Inc.
        License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
        This is free software: you are free to change and redistribute it.
        There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
        and "show warranty" for details.
        This GDB was configured as "x86_64-amazon-linux-gnu".
        For bug reporting instructions, please see:
        <http://www.gnu.org/software/gdb/bugs/>...
        Reading symbols from /home/david/simplegrep...done.
        [New LWP 14441]
        Missing separate debuginfo for /usr/lib64/libstdc++.so.6
        Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/0a/90c35d3174805453ea67a785446d628e298b59.debug
        Missing separate debuginfo for /lib64/libgcc_s.so.1
        Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/00/fa2883fb47b1327397bbf167c52f51a723d013.debug
        Core was generated by `./simplegrep outside ss.list'.
        Program terminated with signal 4, Illegal instruction.
    #0  set_range (to=111, from=111, this=0x7ffee99b7910) at /home/tidy/work/app/hyperscan/hyperscan/src/util/bitfield.h:154
        154 /home/tidy/work/app/hyperscan/hyperscan/src/util/bitfield.h: No such file or directory.
        Missing separate debuginfos, use: debuginfo-install glibc-2.17-106.167.amzn1.x86_64
        (gdb) bt
    #0  set_range (to=111, from=111, this=0x7ffee99b7910) at /home/tidy/work/app/hyperscan/hyperscan/src/util/bitfield.h:154
    #1  setRange (to=111 'o', from=111 'o', this=0x7ffee99b7910) at /home/tidy/work/app/hyperscan/hyperscan/src/util/charreach.h:106
    #2  CharReach (to=111 'o', from=111 'o', this=0x7ffee99b7910) at /home/tidy/work/app/hyperscan/hyperscan/src/util/charreach.h:62
    #3  ue2::AsciiComponentClass::add (this=0x1c3aca0, c=111) at /home/tidy/work/app/hyperscan/hyperscan/src/parser/AsciiComponentClass.cpp:124
    #4  0x00007fe4c5fddcb1 in ue2::getLiteralComponentClass ([email protected]=111 'o', nocase=<optimized out>)
            at /home/tidy/work/app/hyperscan/hyperscan/src/parser/ComponentClass.cpp:421
    #5  0x00007fe4c5e0b1cc in ue2::addLiteral ([email protected]=0x1c3ac50, c=111 'o', mode=...)
                at /home/tidy/work/app/hyperscan/hyperscan/src/parser/Parser.rl:181
    #6  0x00007fe4c5fe78ca in ue2::parse ([email protected]=0x7ffee99ba6ca "outside", globalMode=...)
                    at /home/tidy/work/app/hyperscan/hyperscan/src/parser/Parser.rl:1823
    #7  0x00007fe4c5e1b7b1 in ue2::ParsedExpression::ParsedExpression (this=0x7ffee99b8570, index_in=<optimized out>, expression=0x7ffee99ba6ca "outside", flags=2, 
                actionId=<optimized out>, ext=0x0) at /home/tidy/work/app/hyperscan/hyperscan/src/compiler/compiler.cpp:116
    #8  0x00007fe4c5e1d0c4 in ue2::addExpression (ng=..., [email protected]=0, expression=0x7ffee99ba6ca "outside", flags=2, ext=0x0, id=0)
        at /home/tidy/work/app/hyperscan/hyperscan/src/compiler/compiler.cpp:236
    #9  0x00007fe4c5e19305 in ue2::hs_compile_multi_int ([email protected]=0x7ffee99b8a98, [email protected]=0x7ffee99b8a94, [email protected]=0x7ffee99b8aac, 
                [email protected]=0x0, [email protected]=1, [email protected]=1, [email protected]=0x0, [email protected]=0x7ffee99b8bf0, [email protected]=0x7ffee99b8bf8, g=...)
        at /home/tidy/work/app/hyperscan/hyperscan/src/hs.cpp:228
    #10 0x00007fe4c5e19a13 in hs_compile ([email protected]=0x7ffee99ba6ca "outside", [email protected]=2, [email protected]=1, [email protected]=0x0, 
                [email protected]=0x7ffee99b8bf0, [email protected]=0x7ffee99b8bf8) at /home/tidy/work/app/hyperscan/hyperscan/src/hs.cpp:285
    #11 0x0000000000400d31 in main (argc=<optimized out>, argv=<optimized out>) at /home/tidy/exam/simplegrep.c:163
    (gdb) q
    

    Compiling machine cpuinfo: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 69 model name : Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz stepping : 1 microcode : 0x17 cpu MHz : 2300.092 cache size : 4096 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt bogomips : 4600.18 clflush size : 64 cache_alignment : 64 address sizes : 42 bits physical, 48 bits virtual power management:

    Running machine cpuinfo:

    cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 62 model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz stepping : 4 microcode : 0x415 cpu MHz : 2494.028 cache size : 25600 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase smep erms xsaveopt bugs : bogomips : 4988.05 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

    -Tidy

    opened by TidyHuang 17
  • Segment Fault when I'm using Fat Runtime

    Segment Fault when I'm using Fat Runtime

    I have built the hyperscan as '.so' library, and using it by nginx.

    and then I got segment fault error

    Program received signal SIGSEGV, Segmentation fault.
    0x00000000001202b6 in ?? ()
    Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 pcre-8.32-15.el7.x86_64 zlib-1.2.7-15.el7.x86_64
    (gdb) bt
    #0  0x00000000001202b6 in ?? ()
    #1  0x00007ffff5fd14b5 in ?? ()
    #2  0x00007fffffffd920 in ?? ()
    #3  0x00007ffff7de75e8 in _dl_relocate_object () from /lib64/ld-linux-x86-64.so.2
    Backtrace stopped: frame did not save the PC
    (gdb) info register
    rax            0x0	0
    rbx            0x555555b9d8a0	93824998824096
    rcx            0x0	0
    rdx            0x78bfbfd	126614525
    rsi            0x7ffff5fd1490	140737320391824
    rdi            0x0	0
    rbp            0x7fffffffd7e0	0x7fffffffd7e0
    rsp            0x7fffffffd7d8	0x7fffffffd7d8
    r8             0x555555b9d8a0	93824998824096
    r9             0x555555b9d8a0	93824998824096
    r10            0x7fffffffd640	140737488344640
    r11            0x7fffffffd740	140737488344896
    r12            0x7ffff5c45fc8	140737316675528
    r13            0x7	7
    r14            0x7ffff5b5d020	140737315721248
    r15            0x7ffff690b598	140737330066840
    rip            0x1202b6	0x1202b6
    eflags         0x10246	[ PF ZF IF RF ]
    cs             0x33	51
    ss             0x2b	43
    ds             0x0	0
    es             0x0	0
    fs             0x0	0
    gs             0x0	0
    
    # objdump -d libhs.so
    
      00000000001202b0 <[email protected]>:
      1202b0:   ff 25 8a eb ca 00       jmpq   *0xcaeb8a(%rip)        # dcee40 <_GLOBAL_OFFSET_TABLE_+0x5e40>
      1202b6:   68 c5 0b 00 00          pushq  $0xbc5
      1202bb:   e9 90 43 ff ff          jmpq   114650 <_init+0x20>
    
    
    opened by anhk 15
  • Build Error with 5.4.0

    Build Error with 5.4.0

    The latest release dos not longer build in my debian packaging environment: I am building with default flags in Debian Testing.

    Here is a snipped of the errors that occur a lot during build:

    /home/user/tmp/hyperscan-5.4.0/src/hwlm/hwlm.c:136: undefined reference to `avx2_memset'
    /usr/bin/ld: ../lib/libhs.a(stream.c.o): in function `roseCountingMiracleScan':
    /home/user/tmp/hyperscan-5.4.0/src/rose/counting_miracle.h:68: undefined reference to `avx2_memcpy'
    /usr/bin/ld: ../lib/libhs.a(stream.c.o): in function `roseCountingMiracleScanShufti':
    /home/user/tmp/hyperscan-5.4.0/src/rose/counting_miracle.h:118: undefined reference to `avx2_memcpy'
    /usr/bin/ld: /home/user/tmp/hyperscan-5.4.0/src/rose/counting_miracle.h:118: undefined reference to `avx2_memcpy'
    /usr/bin/ld: ../lib/libhs.a(stream.c.o): in function `roseCountingMiracleScan':
    /home/user/tmp/hyperscan-5.4.0/src/rose/counting_miracle.h:68: undefined reference to `avx2_memcpy'
    /usr/bin/ld: ../lib/libhs.a(program_runtime.c.o): in function `roseCountingMiracleScan':
    /home/user/tmp/hyperscan-5.4.0/src/rose/counting_miracle.h:68: undefined reference to `avx2_memcpy'
    /usr/bin/ld: ../lib/libhs.a(program_runtime.c.o):/home/user/tmp/hyperscan-5.4.0/src/rose/counting_miracle.h:118: more undefined references to `avx2_memcpy' follow
    /usr/bin/ld: ../lib/libhs.a(database.c.o): in function `print_database_string':
    /home/user/tmp/hyperscan-5.4.0/src/database.c:405: undefined reference to `avx2_snprintf'
    /usr/bin/ld: ../lib/libhs.a(database.c.o): in function `hs_serialize_database':
    /home/user/tmp/hyperscan-5.4.0/src/database.c:86: undefined reference to `avx2_memset'
    /usr/bin/ld: /home/user/tmp/hyperscan-5.4.0/src/database.c:105: undefined reference to `avx2_memcpy'
    /usr/bin/ld: ../lib/libhs.a(database.c.o): in function `hs_deserialize_database_at':
    /home/user/tmp/hyperscan-5.4.0/src/database.c:227: undefined reference to `avx2_memset'
    /usr/bin/ld: ../lib/libhs.a(database.c.o): in function `db_copy_bytecode':
    /home/user/tmp/hyperscan-5.4.0/src/database.c:196: undefined reference to `avx2_memcpy'
    /usr/bin/ld: ../lib/libhs.a(database.c.o): in function `hs_deserialize_database':
    /home/user/tmp/hyperscan-5.4.0/src/database.c:275: undefined reference to `avx2_memset'
    /usr/bin/ld: ../lib/libhs.a(database.c.o): in function `db_copy_bytecode':
    /home/user/tmp/hyperscan-5.4.0/src/database.c:196: undefined reference to `avx2_memcpy'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough8_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:956: undefined reference to `core2_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough16_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:968: undefined reference to `core2_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough8_queueInitState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:1073: undefined reference to `core2_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `core2_nfaExecGough16_queueInitState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:1075: undefined reference to `core2_memset'
    /usr/bin/ld: ../lib/libhs.a(mpv.c.o): in function `nfaExecMpv_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/mpv.c:886: undefined reference to `core2_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough8_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:956: undefined reference to `corei7_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough16_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:968: undefined reference to `corei7_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough8_queueInitState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:1073: undefined reference to `corei7_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `corei7_nfaExecGough16_queueInitState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:1075: undefined reference to `corei7_memset'
    /usr/bin/ld: ../lib/libhs.a(mpv.c.o): in function `nfaExecMpv_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/mpv.c:886: undefined reference to `corei7_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough8_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:956: undefined reference to `avx2_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough16_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:968: undefined reference to `avx2_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `nfaExecGough8_queueInitState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:1073: undefined reference to `avx2_memset'
    /usr/bin/ld: ../lib/libhs.a(gough.c.o): in function `avx2_nfaExecGough16_queueInitState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/gough.c:1075: undefined reference to `avx2_memset'
    /usr/bin/ld: ../lib/libhs.a(mpv.c.o): in function `nfaExecMpv_initCompressedState':
    /home/user/tmp/hyperscan-5.4.0/src/nfa/mpv.c:886: undefined reference to `avx2_memset'
    collect2: error: ld returned 1 exit status
    
    
    opened by rhaist 11
  • Chimera Release build mode lots of undefined reference error

    Chimera Release build mode lots of undefined reference error

    hyperscan version: 5.1.1&5.1.0&5.00 boost version: 1.57&1.70 pcre version: 8.41 Ragel version: 6.9 cmake version: 3.14.2 & 3.12.1

    os version: CentOS 7.3.1611 gcc version: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)

    I still get this error using Relwithdebinfo mode.

    gcc error:

    ../lib/libchimera.a(ch_compile.cpp.o): In function `ch::ch_compile_multi_int(char const* const*, unsigned int const*, unsigned int const*, unsigned int, unsigned int, unsigned long, unsigned long, hs_platform_info const*, ch_database**)':
    ch_compile.cpp:(.text+0xbf1): undefined reference to `ue2::mmbit_size(unsigned int)'
    ch_compile.cpp:(.text+0x1034): undefined reference to `ue2::CompileError::CompileError(std::string const&)'
    ch_compile.cpp:(.text+0x1050): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x1055): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x119a): undefined reference to `ue2::CompileError::CompileError(unsigned int, std::string const&)'
    ch_compile.cpp:(.text+0x119f): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x11a4): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x12b8): undefined reference to `ue2::CompileError::CompileError(std::string const&)'
    ch_compile.cpp:(.text+0x12d4): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x12d9): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x1320): undefined reference to `ue2::CompileError::CompileError(std::string const&)'
    ch_compile.cpp:(.text+0x133c): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x1341): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x13e8): undefined reference to `ue2::CompileError::CompileError(unsigned int, std::string const&)'
    ch_compile.cpp:(.text+0x13ed): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x13f2): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x1450): undefined reference to `ue2::CompileError::CompileError(std::string const&)'
    ch_compile.cpp:(.text+0x146c): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x1471): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x14ad): undefined reference to `ue2::CompileError::CompileError(std::string const&)'
    ch_compile.cpp:(.text+0x14c9): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x14ce): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x150a): undefined reference to `ue2::CompileError::CompileError(std::string const&)'
    ch_compile.cpp:(.text+0x1526): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x152b): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x1760): undefined reference to `ue2::CompileError::CompileError(std::string const&)'
    ch_compile.cpp:(.text+0x177c): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x1781): undefined reference to `typeinfo for ue2::CompileError'
    ch_compile.cpp:(.text+0x17ce): undefined reference to `ue2::CompileError::CompileError(std::string const&)'
    ch_compile.cpp:(.text+0x17ea): undefined reference to `ue2::CompileError::~CompileError()'
    ch_compile.cpp:(.text+0x17ef): undefined reference to `typeinfo for ue2::CompileError'
    ../lib/libchimera.a(ch_compile.cpp.o):(.gcc_except_table+0x180): undefined reference to `typeinfo for ue2::CompileError'
    ../lib/libchimera.a(ch_compile.cpp.o):(.gcc_except_table+0x1d8): undefined reference to `typeinfo for ue2::CompileError'
    ../lib/libchimera.a(ch_compile.cpp.o):(.gcc_except_table+0x230): undefined reference to `typeinfo for ue2::CompileError'
    ../lib/libchimera.a(ch_runtime.c.o): In function `multiCallback':
    ch_runtime.c:(.text+0x6df): undefined reference to `mmbit_maxlevel_direct_lut'
    ch_runtime.c:(.text+0x6e9): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x728): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x766): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x79c): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x7d6): undefined reference to `mmbit_root_offset_from_level'
    ../lib/libchimera.a(ch_runtime.c.o):ch_runtime.c:(.text+0x808): more undefined references to `mmbit_root_offset_from_level' follow
    ../lib/libchimera.a(ch_runtime.c.o): In function `multiCallback':
    ch_runtime.c:(.text+0x88b): undefined reference to `mmbit_keyshift_lut'
    ch_runtime.c:(.text+0x972): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x98c): undefined reference to `mmbit_keyshift_lut'
    ch_runtime.c:(.text+0xa35): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0xa43): undefined reference to `mmbit_maxlevel_direct_lut'
    ch_runtime.c:(.text+0xab0): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0xaf0): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0xb31): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0xb73): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0xbb4): undefined reference to `mmbit_root_offset_from_level'
    ../lib/libchimera.a(ch_runtime.c.o):ch_runtime.c:(.text+0xbf5): more undefined references to `mmbit_root_offset_from_level' follow
    ../lib/libchimera.a(ch_runtime.c.o): In function `ch_scan':
    ch_runtime.c:(.text+0x13a2): undefined reference to `mmbit_maxlevel_direct_lut'
    ch_runtime.c:(.text+0x1420): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x146e): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x14ac): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x14ec): undefined reference to `mmbit_root_offset_from_level'
    ch_runtime.c:(.text+0x152a): undefined reference to `mmbit_root_offset_from_level'
    ../lib/libchimera.a(ch_runtime.c.o):ch_runtime.c:(.text+0x1568): more undefined references to `mmbit_root_offset_from_level' follow
    collect2: error: ld returned 1 exit status
    make[3]: *** [bin/unit-chimera] Error 1
    make[2]: *** [unit/CMakeFiles/unit-chimera.dir/all] Error 2
    make[1]: *** [unit/CMakeFiles/unit-chimera.dir/rule] Error 2
    make: *** [unit-chimera] Error 2
    

    this happens while linking unit-chimera

    opened by JackLiar 10
  • Compile Error for Hyperscan 5.0.0 with Boost 1.68.0

    Compile Error for Hyperscan 5.0.0 with Boost 1.68.0

    I get a compilation error when compiling Hyperscan 5.0.0 using the latest boost 1.68.0 libraries. I do not get this error when using the 1.67.0 boost libraries (I am able to sucessfully compile Hyperscan 5.0.0 using the older 1.67.0 boost libraries). I am compiling on Ubuntu 18 x64, with gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3).

    Error message is: in limex_compile.cpp, generated: error: call of overloaded ‘distance(...)’ is ambiguous.

    Steps to re-create the issue: Get boost libraries: cd ~/snort_src wget https://dl.bintray.com/boostorg/release/1.68.0/source/boost_1_68_0.tar.gz tar -xvzf boost_1_68_0.tar.gz

    Get hyperscan wget https://github.com/intel/hyperscan/archive/v5.0.0.tar.gz tar -xvzf v5.0.0.tar.gz mkdir hyperscan-5.0.0-build cd hyperscan-5.0.0-build/ cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DBOOST_ROOT=~/snort_src/boost_1_68_0/ ../hyperscan-5.0.0 make

    Results from 'make' command: [ 1%] Generating src/parser/control_verbs.cpp [ 1%] Generating src/parser/Parser.cpp Scanning dependencies of target hs_compile [ 1%] Building CXX object CMakeFiles/hs_compile.dir/src/grey.cpp.o [ 1%] Building CXX object CMakeFiles/hs_compile.dir/src/hs.cpp.o [ 2%] Building CXX object CMakeFiles/hs_compile.dir/src/compiler/asserts.cpp.o [ 2%] Building CXX object CMakeFiles/hs_compile.dir/src/compiler/compiler.cpp.o [ 2%] Building CXX object CMakeFiles/hs_compile.dir/src/compiler/error.cpp.o [ 2%] Building CXX object CMakeFiles/hs_compile.dir/src/fdr/engine_description.cpp.o [ 3%] Building CXX object CMakeFiles/hs_compile.dir/src/fdr/fdr_compile.cpp.o [ 3%] Building CXX object CMakeFiles/hs_compile.dir/src/fdr/fdr_compile_util.cpp.o [ 3%] Building CXX object CMakeFiles/hs_compile.dir/src/fdr/fdr_confirm_compile.cpp.o [ 4%] Building CXX object CMakeFiles/hs_compile.dir/src/fdr/fdr_engine_description.cpp.o [ 4%] Building CXX object CMakeFiles/hs_compile.dir/src/fdr/flood_compile.cpp.o [ 4%] Building CXX object CMakeFiles/hs_compile.dir/src/fdr/teddy_compile.cpp.o [ 4%] Building CXX object CMakeFiles/hs_compile.dir/src/fdr/teddy_engine_description.cpp.o [ 5%] Building CXX object CMakeFiles/hs_compile.dir/src/hwlm/hwlm_build.cpp.o [ 5%] Building CXX object CMakeFiles/hs_compile.dir/src/hwlm/hwlm_literal.cpp.o [ 5%] Building CXX object CMakeFiles/hs_compile.dir/src/hwlm/noodle_build.cpp.o [ 6%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/accel_dfa_build_strat.cpp.o [ 6%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/accelcompile.cpp.o [ 6%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/castlecompile.cpp.o [ 7%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/dfa_build_strat.cpp.o [ 7%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/dfa_min.cpp.o [ 7%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/goughcompile.cpp.o [ 7%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/goughcompile_accel.cpp.o [ 8%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/goughcompile_reg.cpp.o [ 8%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/mcclellancompile.cpp.o [ 8%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/mcclellancompile_util.cpp.o [ 9%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/mcsheng_compile.cpp.o [ 9%] Building CXX object CMakeFiles/hs_compile.dir/src/nfa/limex_compile.cpp.o /home/noah/snort_src/hyperscan-5.0.0/src/nfa/limex_compile.cpp: In function ‘u32 ue2::{anonymous}::addSquashMask(const ue2::{anonymous}::build_info&, const NFAVertex&, std::vector<boost::dynamic_bitset<> >&)’: /home/noah/snort_src/hyperscan-5.0.0/src/nfa/limex_compile.cpp:983:54: error: call of overloaded ‘distance(std::vector<boost::dynamic_bitset<> >::iterator, __gnu_cxx::__normal_iterator<boost::dynamic_bitset<>*, std::vector<boost::dynamic_bitset<> > >&)’ is ambiguous return verify_u32(distance(squash.begin(), it)); ^ In file included from /usr/include/c++/7/bits/stl_algobase.h:66:0, from /usr/include/c++/7/bits/char_traits.h:39, from /usr/include/c++/7/string:40, from /home/noah/snort_src/hyperscan-5.0.0/src/nfa/nfa_kind.h:40, from /home/noah/snort_src/hyperscan-5.0.0/src/nfagraph/ng_holder.h:41, from /home/noah/snort_src/hyperscan-5.0.0/src/nfa/limex_compile.h:37, from /home/noah/snort_src/hyperscan-5.0.0/src/nfa/limex_compile.cpp:34: /usr/include/c++/7/bits/stl_iterator_base_funcs.h:138:5: note: candidate: typename std::iterator_traits<_Iterator>::difference_type std::distance(_InputIterator, _InputIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<boost::dynamic_bitset<>*, std::vector<boost::dynamic_bitset<> > >; typename std::iterator_traits<_Iterator>::difference_type = long int] distance(_InputIterator __first, _InputIterator __last) ^~~~~~~~ In file included from /home/noah/snort_src/boost_1_68_0/boost/range/distance.hpp:18:0, from /home/noah/snort_src/boost_1_68_0/boost/range/functions.hpp:21, from /home/noah/snort_src/boost_1_68_0/boost/range/iterator_range_core.hpp:38, from /home/noah/snort_src/boost_1_68_0/boost/range/iterator_range.hpp:13, from /home/noah/snort_src/hyperscan-5.0.0/src/util/graph_range.h:54, from /home/noah/snort_src/hyperscan-5.0.0/src/util/ue2_graph.h:33, from /home/noah/snort_src/hyperscan-5.0.0/src/nfagraph/ng_holder.h:44, from /home/noah/snort_src/hyperscan-5.0.0/src/nfa/limex_compile.h:37, from /home/noah/snort_src/hyperscan-5.0.0/src/nfa/limex_compile.cpp:34: /home/noah/snort_src/boost_1_68_0/boost/iterator/distance.hpp:49:9: note: candidate: typename boost::iterators::iterator_difference::type boost::iterators::distance_adl_barrier::distance(SinglePassIterator, SinglePassIterator) [with SinglePassIterator = __gnu_cxx::__normal_iterator<boost::dynamic_bitset<>*, std::vector<boost::dynamic_bitset<> > >; typename boost::iterators::iterator_difference::type = long int] distance(SinglePassIterator first, SinglePassIterator last) ^~~~~~~~ CMakeFiles/hs_compile.dir/build.make:720: recipe for target 'CMakeFiles/hs_compile.dir/src/nfa/limex_compile.cpp.o' failed make[2]: *** [CMakeFiles/hs_compile.dir/src/nfa/limex_compile.cpp.o] Error 1 CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/hs_compile.dir/all' failed make[1]: *** [CMakeFiles/hs_compile.dir/all] Error 2 Makefile:129: recipe for target 'all' failed make: *** [all] Error 2

    Results from cmake ( works OK, no errors): DBOOST_ROOT=~/snort_src/boost_1_68_0/ ../hyperscan-5.0.0 -- The C compiler identification is GNU 7.3.0 -- The CXX compiler identification is GNU 7.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Performing Test ARCH_64_BIT -- Performing Test ARCH_64_BIT - Success -- Performing Test ARCH_32_BIT -- Performing Test ARCH_32_BIT - Failed -- Default build type 'Release with debug info' -- Boost version: 1.68.0 -- Found PythonInterp: /usr/bin/python (found version "2.7.15") -- Build date: 2018-09-03 -- gcc version 7.3.0 -- g++ version 7.3.0 -- Looking for include file unistd.h -- Looking for include file unistd.h - found -- Looking for include file intrin.h -- Looking for include file intrin.h - not found -- Looking for C++ include intrin.h -- Looking for C++ include intrin.h - not found -- Looking for include file x86intrin.h -- Looking for include file x86intrin.h - found -- Looking for C++ include x86intrin.h -- Looking for C++ include x86intrin.h - found -- Looking for posix_memalign -- Looking for posix_memalign - found -- Looking for _aligned_malloc -- Looking for _aligned_malloc - not found -- Performing Test HAS_C_HIDDEN -- Performing Test HAS_C_HIDDEN - Success -- Performing Test HAS_CXX_HIDDEN -- Performing Test HAS_CXX_HIDDEN - Success -- Looking for _LIBCPP_VERSION -- Looking for _LIBCPP_VERSION - not found -- generator is Unix Makefiles -- Performing Test HAS_C_ATTR_IFUNC -- Performing Test HAS_C_ATTR_IFUNC - Success -- Performing Test HAVE_SSSE3 -- Performing Test HAVE_SSSE3 - Success -- Performing Test HAVE_AVX2 -- Performing Test HAVE_AVX2 - Success -- Performing Test HAVE_AVX512 -- Performing Test HAVE_AVX512 - Failed -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED - Success -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED - Success -- Performing Test HAVE__BUILTIN_CONSTANT_P -- Performing Test HAVE__BUILTIN_CONSTANT_P - Success -- Performing Test C_FLAG_Wvla -- Performing Test C_FLAG_Wvla - Success -- Performing Test C_FLAG_Wpointer_arith -- Performing Test C_FLAG_Wpointer_arith - Success -- Performing Test C_FLAG_Wstrict_prototypes -- Performing Test C_FLAG_Wstrict_prototypes - Success -- Performing Test C_FLAG_Wmissing_prototypes -- Performing Test C_FLAG_Wmissing_prototypes - Success -- Performing Test CXX_FLAG_Wvla -- Performing Test CXX_FLAG_Wvla - Success -- Performing Test CXX_FLAG_Wpointer_arith -- Performing Test CXX_FLAG_Wpointer_arith - Success -- Performing Test CC_SELF_ASSIGN -- Performing Test CC_SELF_ASSIGN - Failed -- Performing Test CXX_SELF_ASSIGN -- Performing Test CXX_SELF_ASSIGN - Failed -- Performing Test CC_PAREN_EQUALITY -- Performing Test CC_PAREN_EQUALITY - Failed -- Performing Test CXX_UNUSED_CONST_VAR -- Performing Test CXX_UNUSED_CONST_VAR - Success -- Performing Test CXX_IGNORED_ATTR -- Performing Test CXX_IGNORED_ATTR - Success -- Performing Test CXX_WEAK_VTABLES -- Performing Test CXX_WEAK_VTABLES - Failed -- Performing Test CXX_MISSING_DECLARATIONS -- Performing Test CXX_MISSING_DECLARATIONS - Success -- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS -- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS - Success -- Performing Test CXX_WUNUSED_VARIABLE -- Performing Test CXX_WUNUSED_VARIABLE - Success -- Building runtime for multiple microarchitectures -- Looking for mmap -- Looking for mmap - found -- Doxygen not found, unable to generate API reference -- Sphinx not found, unable to generate developer reference -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Checking for module 'libpcre=8.41' -- Requested 'libpcre = 8.41' but version of libpcre is 8.39 -- PCRE version 8.41 not found -- PCRE 8.41 not found -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE
    -- Checking for module 'sqlite3' -- Found sqlite3, version 3.22.0 -- Performing Test SQLITE_VERSION_OK -- Performing Test SQLITE_VERSION_OK - Success -- Looking for sqlite3_open_v2 -- Looking for sqlite3_open_v2 - found -- Looking for C++ include pthread_np.h -- Looking for C++ include pthread_np.h - not found -- Looking for pthread_setaffinity_np -- Looking for pthread_setaffinity_np - found -- Looking for malloc_info -- Looking for malloc_info - found -- Looking for shmget -- Looking for shmget - found -- PCRE 8.41 not found, not building hscollider -- Configuring done -- Generating done -- Build files have been written to: /home/noah/snort_src/hyperscan-5.0.0

    opened by NDietrich 10
  • Compile finds PCRE but also doesn't find it ?

    Compile finds PCRE but also doesn't find it ?

    Hi,

    I'm currently trying to compile Hyperscan 4.70 on Slackware64-14.2 and i'm noticing this strange contradiction:

    -- Checking for module 'libpcre=8.41' -- Found libpcre, version 8.41 -- PCRE version 8.41 -- PCRE 8.41 not found, not building hscollider -- Configuring done -- Generating done

    This seems contradictory to me and is preventing me from building the hs-collider module.

    GCC Version 5.5.0 G++ Version 5.5.0

    Please let me know if you need any other information .

    opened by R3belWithoutAPause 10
  • the Chinese period (。)match the pattern [\u4E00-\u9FA5]

    the Chinese period (。)match the pattern [\u4E00-\u9FA5]

    I compile the pattern of [\u4E00-\u9FA5] with default flags, when i using chines period(。)to match the pattern,the result is matched.According to the actual situation,the resule must be not matched.

    opened by whhitdragon 9
  • hs_alloc_scratch: Conditional jump or move depends on uninitialised value

    hs_alloc_scratch: Conditional jump or move depends on uninitialised value

    When running the test suite (v5.1.1) under valgrind I'm getting the something similar to the following for a number of tests, including HyperscanArgChecks.ScratchSizeNoSize, HyperscanTestBehaviour.ScanSeveralGigabytesNoMatch, HyperscanTestBehaviour.CloseStreamMatch, HyperscanTestBehaviour.NoMainCB, HyperscanTestBehaviour.CloseStreamNoMatch, HyperscanTestBehaviour.CloseStreamAfterTermination, HyperscanTestBehaviour.VectoredX, HyperscanTestBehaviour.MultiStreamX, and many more.

    Currently debugging.

    ==4607== Conditional jump or move depends on uninitialised value(s)
    ==4607==    at 0x3D60B3: hs_alloc_scratch (scratch.c:256)
    ==4607==    by 0x1D7325: (anonymous namespace)::HyperscanArgChecks_ScratchSizeNoSize_Test::TestBody() (arg_checks.cpp:2009)
    ==4607==    by 0x1B85DC: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest-all.cc:3562)
    ==4607==    by 0x1B85DC: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest-all.cc:3598)
    ==4607==    by 0x1AB034: testing::Test::Run() [clone .part.451] (gtest-all.cc:3634)
    ==4607==    by 0x1AB328: Run (gtest-all.cc:3626)
    ==4607==    by 0x1AB328: testing::TestInfo::Run() [clone .part.452] (gtest-all.cc:3810)
    ==4607==    by 0x1AB4D0: Run (gtest-all.cc:3785)
    ==4607==    by 0x1AB4D0: testing::TestCase::Run() [clone .part.453] (gtest-all.cc:3928)
    ==4607==    by 0x1B2242: Run (gtest-all.cc:3914)
    ==4607==    by 0x1B2242: testing::internal::UnitTestImpl::RunAllTests() (gtest-all.cc:5799)
    ==4607==    by 0x1B8ABC: HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (gtest-all.cc:3562)
    ==4607==    by 0x1B8ABC: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest-all.cc:3598)
    ==4607==    by 0x1AB689: testing::UnitTest::Run() (gtest-all.cc:5410)
    ==4607==    by 0x197D10: RUN_ALL_TESTS (gtest.h:20058)
    ==4607==    by 0x197D10: main (main.cpp:35)
    ==4607==  Uninitialised value was created by a heap allocation
    ==4607==    at 0x4837DBF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==4607==    by 0x607707: allocate (new_allocator.h:111)
    ==4607==    by 0x607707: allocate (alloc_traits.h:436)
    ==4607==    by 0x607707: _M_allocate (stl_vector.h:296)
    ==4607==    by 0x607707: void std::vector<scatter_unit_u8, std::allocator<scatter_unit_u8> >::_M_realloc_insert<scatter_unit_u8 const&>(__gnu_cxx::__normal_iterator<scatter_unit_u8*, std::vector<scatter_unit_u8, std::allocator<scatter_unit_u8> > >, scatter_unit_u8 const&) (vector.tcc:427)
    ==4607==    by 0x606E6E: push_back (stl_vector.h:1085)
    ==4607==    by 0x606E6E: void ue2::add_scatter<scatter_unit_u8>(std::vector<scatter_unit_u8, std::allocator<scatter_unit_u8> >*, unsigned int, unsigned long long) (multibit_build.cpp:199)
    ==4607==    by 0x60725B: ue2::mmbBuildInitRangePlan(unsigned int, unsigned int, unsigned int, ue2::scatter_plan_raw*) [clone .part.193] (multibit_build.cpp:232)
    ==4607==    by 0x8A7671: ue2::buildStateScatterPlan(unsigned int, unsigned int, unsigned int, unsigned int, RoseStateOffsets const&, bool, unsigned int, unsigned int, unsigned int) (rose_build_scatter.cpp:81)
    ==4607==    by 0x86B3A1: ue2::RoseBuildImpl::buildFinalEngine(unsigned int) (rose_build_bytecode.cpp:3799)
    ==4607==    by 0x5B669F: ue2::RoseBuildImpl::buildRose(unsigned int) (rose_build_compile.cpp:1796)
    ==4607==    by 0x2A3B98: generateRoseEngine (compiler.cpp:352)
    ==4607==    by 0x2A3B98: ue2::build(ue2::NG&, unsigned int*) (compiler.cpp:422)
    ==4607==    by 0x2A10D2: ue2::hs_compile_multi_int(char const* const*, unsigned int const*, unsigned int const*, hs_expr_ext const* const*, unsigned int, unsigned int, hs_platform_info const*, hs_database**, hs_compile_error**, ue2::Grey const&) (hs.cpp:254)
    ==4607==    by 0x2A1AE9: hs_compile (hs.cpp:301)
    ==4607==    by 0x1D7297: (anonymous namespace)::HyperscanArgChecks_ScratchSizeNoSize_Test::TestBody() (arg_checks.cpp:2004)
    ==4607==    by 0x1B85DC: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest-all.cc:3562)
    ==4607==    by 0x1B85DC: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest-all.cc:3598)
    
    opened by pallas 9
  • Build failure with Boost 1.62.0

    Build failure with Boost 1.62.0

    After upgrading Boost in homebrew/core from 1.61 to 1.62, hyperscan now fails to build.

    The error is

    [ 39%] Building CXX object CMakeFiles/hs.dir/src/nfagraph/ng_dominators.cpp.o
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++    -I/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/build -I/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1 -I/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src -isystem /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/include -isystem /usr/local/include  -Os -w -pipe -march=native -mmacosx-version-min=10.11  -DNDEBUG -std=c++11 -Wall -Wextra -Wshadow -Wswitch -Wreturn-type -Wcast-qual -Wno-deprecated -Wnon-virtual-dtor -fno-strict-aliasing -fno-omit-frame-pointer -fvisibility=hidden -Wvla -Wpointer-arith -Wno-self-assign -Wno-unused-const-variable -Wno-ignored-attributes -Wweak-vtables -Wmissing-declarations -O2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -mmacosx-version-min=10.11   -o CMakeFiles/hs.dir/src/nfagraph/ng_dominators.cpp.o -c /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp
    In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:34:
    In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.h:38:
    In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_holder.h:32:
    In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_graph.h:46:
    In file included from /usr/local/include/boost/graph/adjacency_list.hpp:246:
    In file included from /usr/local/include/boost/graph/detail/adjacency_list.hpp:32:
    /usr/local/include/boost/graph/graph_concepts.hpp:131:13: error: no matching function for call to 'degree'
            n = degree(v, g);
                ^~~~~~
    /usr/local/include/boost/concept/usage.hpp:16:43: note: in instantiation of member function 'boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >::~BidirectionalGraph' requested here
        ~usage_requirements() { ((Model*)0)->~Model(); }
                                              ^
    /usr/local/include/boost/concept/detail/general.hpp:39:42: note: in instantiation of member function 'boost::concepts::usage_requirements<boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> > >::~usage_requirements' requested here
        static void failed() { ((Model*)0)->~Model(); }
                                             ^
    /usr/local/include/boost/graph/graph_concepts.hpp:122:9: note: in instantiation of member function 'boost::concepts::requirement<boost::concepts::failed ************boost::concepts::usage_requirements<boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> > >::************>::failed' requested here
            BOOST_CONCEPT_USAGE(BidirectionalGraph) {
            ^
    /usr/local/include/boost/concept/usage.hpp:29:7: note: expanded from macro 'BOOST_CONCEPT_USAGE'
          BOOST_CONCEPT_ASSERT((boost::concepts::usage_requirements<model>)); \
          ^
    /usr/local/include/boost/concept/assert.hpp:43:5: note: expanded from macro 'BOOST_CONCEPT_ASSERT'
        BOOST_CONCEPT_ASSERT_FN(void(*)ModelInParens)
        ^
    /usr/local/include/boost/concept/detail/general.hpp:71:51: note: expanded from macro 'BOOST_CONCEPT_ASSERT_FN'
        &::boost::concepts::requirement_<ModelFnPtr>::failed>    \
                                                      ^
    /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:67:16: note: in instantiation of function template specialization 'boost_ue2::lengauer_tarjan_dominator_tree<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, boost::iterator_property_map<std::__1::__wrap_iter<unsigned long *>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, unsigned long, unsigned long &>, boost::iterator_property_map<std::__1::__wrap_iter<void **>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, void *, void *&>, std::__1::vector<void *, std::__1::allocator<void *> >, boost::associative_property_map<boost::unordered::unordered_map<void *, void *, boost::hash<void *>, std::__1::equal_to<void *>, std::__1::allocator<std::__1::pair<void *const, void *> > > > >' requested here
        boost_ue2::lengauer_tarjan_dominator_tree(g, source, index_map, dfnum_map,
                   ^
    /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:81:12: note: in instantiation of function template specialization 'ue2::calcDominators<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >' requested here
        return calcDominators(boost::reverse_graph<NFAGraph, const NFAGraph &>(g.g),
               ^
    /usr/local/include/boost/graph/detail/adjacency_list.hpp:1087:5: note: candidate template ignored: could not match 'undirected_graph_helper' against 'reverse_graph'
        degree(typename Config::vertex_descriptor u,
        ^
    /usr/local/include/boost/graph/detail/adjacency_list.hpp:1514:5: note: candidate template ignored: could not match 'bidirectional_graph_helper_with_property' against 'reverse_graph'
        degree(typename Config::vertex_descriptor u,
        ^
    In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:34:
    In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.h:38:
    In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_holder.h:32:
    In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_graph.h:46:
    In file included from /usr/local/include/boost/graph/adjacency_list.hpp:246:
    In file included from /usr/local/include/boost/graph/detail/adjacency_list.hpp:32:
    /usr/local/include/boost/graph/graph_concepts.hpp:138:13: error: no matching function for call to 'degree'
            n = degree(v, cg);
                ^~~~~~
    /usr/local/include/boost/graph/graph_concepts.hpp:133:9: note: in instantiation of member function 'boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >::const_constraints' requested here
            const_constraints(g);
            ^
    /usr/local/include/boost/concept/usage.hpp:16:43: note: in instantiation of member function 'boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >::~BidirectionalGraph' requested here
        ~usage_requirements() { ((Model*)0)->~Model(); }
                                              ^
    /usr/local/include/boost/concept/detail/general.hpp:39:42: note: in instantiation of member function 'boost::concepts::usage_requirements<boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> > >::~usage_requirements' requested here
        static void failed() { ((Model*)0)->~Model(); }
                                             ^
    /usr/local/include/boost/graph/graph_concepts.hpp:122:9: note: in instantiation of member function 'boost::concepts::requirement<boost::concepts::failed ************boost::concepts::usage_requirements<boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> > >::************>::failed' requested here
            BOOST_CONCEPT_USAGE(BidirectionalGraph) {
            ^
    /usr/local/include/boost/concept/usage.hpp:29:7: note: expanded from macro 'BOOST_CONCEPT_USAGE'
          BOOST_CONCEPT_ASSERT((boost::concepts::usage_requirements<model>)); \
          ^
    /usr/local/include/boost/concept/assert.hpp:43:5: note: expanded from macro 'BOOST_CONCEPT_ASSERT'
        BOOST_CONCEPT_ASSERT_FN(void(*)ModelInParens)
        ^
    /usr/local/include/boost/concept/detail/general.hpp:71:51: note: expanded from macro 'BOOST_CONCEPT_ASSERT_FN'
        &::boost::concepts::requirement_<ModelFnPtr>::failed>    \
                                                      ^
    /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:67:16: note: in instantiation of function template specialization 'boost_ue2::lengauer_tarjan_dominator_tree<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, boost::iterator_property_map<std::__1::__wrap_iter<unsigned long *>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, unsigned long, unsigned long &>, boost::iterator_property_map<std::__1::__wrap_iter<void **>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, void *, void *&>, std::__1::vector<void *, std::__1::allocator<void *> >, boost::associative_property_map<boost::unordered::unordered_map<void *, void *, boost::hash<void *>, std::__1::equal_to<void *>, std::__1::allocator<std::__1::pair<void *const, void *> > > > >' requested here
        boost_ue2::lengauer_tarjan_dominator_tree(g, source, index_map, dfnum_map,
                   ^
    /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:81:12: note: in instantiation of function template specialization 'ue2::calcDominators<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >' requested here
        return calcDominators(boost::reverse_graph<NFAGraph, const NFAGraph &>(g.g),
               ^
    /usr/local/include/boost/graph/detail/adjacency_list.hpp:1087:5: note: candidate template ignored: could not match 'undirected_graph_helper' against 'reverse_graph'
        degree(typename Config::vertex_descriptor u,
        ^
    /usr/local/include/boost/graph/detail/adjacency_list.hpp:1514:5: note: candidate template ignored: could not match 'bidirectional_graph_helper_with_property' against 'reverse_graph'
        degree(typename Config::vertex_descriptor u,
        ^
    2 errors generated.
    make[2]: *** [CMakeFiles/hs.dir/src/nfagraph/ng_dominators.cpp.o] Error 1
    make[1]: *** [CMakeFiles/hs.dir/all] Error 2
    make: *** [all] Error 2
    /usr/local/Homebrew/Library/Homebrew/debrew.rb:11:in `raise'
    

    Full build log is here: https://gist.github.com/ilovezfs/3b8db4415d055a04cf0099e585ad12f3

    opened by ilovezfs 9
  • Feature/add arm support

    Feature/add arm support

    (Resubmitting the PR against 'develop' branch as suggested in #286).

    This PR brings the ARM NEON port up to par with the x86. It is based on PR272 (#272), however it can be pulled in on its own as it includes the changes. All tests pass on both ARM and x86 (obviously, we wouldn't want to break x86 functionality). As already described in PR272, x86-specific code has been abstracted away and arch specific code has been put into separate headers in src/util/arch/. This way, maintining arch-specific code of hyperscan and/or porting hyperscan to a new architecture (eg Power/RISC-V/etc) should now be much easier. Please consider merging this PR.

    opened by markos 8
  • Sample data is missing on 01.org

    Sample data is missing on 01.org

    Hi,

    I'm trying to evaluate the performance of hyperscan based on https://www.hyperscan.io/2017/05/08/performance-analysis-hyperscan-hsbench/

    While the below download link for sample data is missing on 01.org https://01.org/downloads/sample-data-hyperscan-hsbench-performance-measurement

    Where can I find them?

    Thanks.

    opened by AsaMyth 2
  • Memory leak / invalid pointer when serializing database in Windows built with Cygwin

    Memory leak / invalid pointer when serializing database in Windows built with Cygwin

    Hello,

    I believe I have found a bug when running Hyperscan on Windows. On linux I have not been able to reproduce the error. It happens when I try to free a serialized database without a custom memory allocator. When I set a custom memory allocator (which is only a malloc and a free) it is able to free without further problem, however without defining this custom allocator it crashes. By using DrMemory I was able to find the following memory errors:

    Error #2: INVALID HEAP ARGUMENT to free 0x0000000002069f30
    # 0 replace_free               [D:\a\drmemory\drmemory\common\alloc_replace.c:2710]
    # 1 main   
    Note: @0:00:07.101 in thread 18456
    Note: refers to -1 byte(s) before next malloc
    Note: next higher malloc: 0x0000000002069f30-0x000000000206b3e0
    Note: refers to -5296 byte(s) beyond last valid byte in prior malloc
    Note: prev lower malloc:  0x0000000002069f30-0x000000000206b3e0
    
    Error #3: LEAK 5296 direct bytes 0x0000000002069f30-0x000000000206b3e0 + 0 indirect bytes
    # 0 replace_malloc                      [D:\a\drmemory\drmemory\common\alloc_replace.c:2580]
    # 1 hs_runtime.dll!hs_serialize_database+0x93     (0x00007ffa94882b44 <hs_runtime.dll+0xc2b44>)
    # 2 main   
    

    (Note that Error #1 is unrelated)

    It seems I am not able to free the serialized database returned by hs_serialize_database. Nevertheless, if I set the custom allocator to the following one:

    void * allocator(size_t bytes)
    {
        return (void *) malloc(bytes);
    }
    
    void deallocator(void * ptr)
    {
        free(ptr);
    }
    
    [...]
    hs_err = hs_set_misc_allocator(allocator, deallocator);
        if(hs_err != HS_SUCCESS)
            throw std::runtime_error("ERROR: Could not use allocator");
    [...]
    

    Then no leak or invalid heap argument is reported and the program finishes correctly.

    Version of the build and reproducible example

    I have built Hyperscan 5.4.0 in Cygwin64 version 3.3.4(0.341/5/3), Windows 11 Enterprise version 21H2 and GCC version x86_64-w64-mingw32-g++ 11.2.0. An isolated (hopefully reproducible) example is shown below:

    #include <iostream>
    #include <vector>
    #include <hs.h>
    
    void * allocator(size_t bytes)
    {
        return (void *) malloc(bytes);
    }
    
    void deallocator(void * ptr)
    {
        free(ptr);
    }
    
    int main(int ac, char ** av)
    {
        // Vectors to hold the patterns expressions, flags and IDs
        std::vector<const char *> cstr_patterns;
        std::vector<unsigned> patterns_flags;
        std::vector<unsigned> patterns_ids;
    
        std::string p1 = "A random regex pattern";
        std::string p2 = "value: [0-9]+";
    
        cstr_patterns.push_back(p1.c_str());
        cstr_patterns.push_back(p2.c_str());
        patterns_flags.push_back(HS_FLAG_DOTALL | HS_FLAG_MULTILINE | HS_FLAG_SOM_LEFTMOST);
        patterns_flags.push_back(HS_FLAG_DOTALL | HS_FLAG_MULTILINE | HS_FLAG_SOM_LEFTMOST);
        patterns_ids.push_back(0);
        patterns_ids.push_back(1);
    
        // Database structures
        hs_database_t * db_block = NULL;
        hs_compile_error_t * compile_err = NULL;
    
        std::cout << "Begin compilation" << std::endl;
        int err = 0;
        
        // Compile
        err = hs_compile_multi(cstr_patterns.data(), patterns_flags.data(),
                        patterns_ids.data(), cstr_patterns.size(), HS_MODE_BLOCK,
                        NULL, &db_block, &compile_err);
        
        // Check if there were any errors
        if (err != HS_SUCCESS)
        {
            std::string error_msg;
            if (compile_err->expression < 0)
                error_msg.append("ERROR: "+std::string(compile_err->message)+"\n"); 
            else
                error_msg.append("ERROR: Pattern [id: "+std::string(std::to_string(compile_err->expression))+"] "+std::string(cstr_patterns[compile_err->expression])
                    + " failed with error "+std::string(compile_err->message)+"\n");
            hs_free_compile_error(compile_err);
            throw std::runtime_error(error_msg);
        }
        else
            hs_free_compile_error(compile_err);
    
        std::cout << "Finished compilation" << std::endl;
        
        // Set custom allocator
        hs_error_t hs_err;
        hs_err = hs_set_misc_allocator(allocator, deallocator);
        if(hs_err != HS_SUCCESS)
            throw std::runtime_error("ERROR: Could not use allocator");
        
    
        std::cout << "Begin serialization " << std::endl;
    
        // Serialize DB
        size_t serialized_size;
        char * serialized_db = NULL;
    
        hs_err = hs_serialize_database(db_block, &serialized_db, &serialized_size);
        if(hs_err != HS_SUCCESS)
            throw std::runtime_error("ERROR: Could not serialize db");
    
        std::cout << "Finished serialization " << std::endl;
        
        // Free database
        std::cout << "Freeing database " << std::endl;
        hs_err = hs_free_database(db_block);
        if(hs_err != HS_SUCCESS)
            throw std::runtime_error("ERROR: Could not free db");
        
        // Do something .........
    
        // Free serialized db
        std::cout << "Freeing serialized database" << std::endl;
        free(serialized_db);
        
        std::cout << "Finished" << std::endl;
    
        return 0;
    }
    

    Also the full compilation line (on cygwin):

     /bin/x86_64-w64-mingw32-g++.exe -static -g -fno-inline -fno-omit-frame-pointer -o isolated.exe -I/<path-to-hyperscan>/hyperscan/src/ -I/<path-to-hyperscan>/hyperscan/ isolated.cpp /<path-to-hyperscan>/hyperscan/bin/hs.dll /<path-to-hyperscan>/hyperscan/bin/hs_runtime.dll
    

    In order to reproduce the error, simply comment the lines regarding the setting of the hs_set_misc_allocator. I believe the problem is that the pointer returned by char *out = hs_misc_alloc(length); in database.c is not freeable by the default system free.

    Thank you for your support, Esteban

    opened by estebanpw 0
  • [Bug] Scan fail for specify corpora and pattern in literal mode

    [Bug] Scan fail for specify corpora and pattern in literal mode

    Hi, everyone. I try to use my custom corpora and simple pattern to benchmark by hsbench, it will abort(RelWithDebug) or scan fail(Release) in literal mode. image

    My pattern:

    # will error
    1:/xt>DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD/i
    # will error
    # 1:/xt<DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD/i
    2:/123/i
    

    I have test in some case:

    • In pattern id=1, either > or <, it will error.
    • In pattern id=1, if i reduce the number of D, or replace the last D to A, it works ok.
    • Without id=2, it works ok.
    • In non-literal mode, it works ok.
    • In pattern id=1, replace xt to xy, it works ok

    The error code is: src/rose/program_runtime.c +3492 In function: roseRunProgram_l image

    This is my corpora: 23414.zip

    I want to know why and how to fix it, really thanks for everyone first.

    opened by pengtianabc 1
  • clang-13 `-Wunused-but-set-variable` causes compilation failure

    clang-13 `-Wunused-but-set-variable` causes compilation failure

    When compiling the project under macOS with the latest command line tools version 13.3.0.0.1.1645755326, which uses clang version Apple clang version 13.1.6 (clang-1316.0.21.2), the project fails to be built, reporting error like:

    FAILED: CMakeFiles/hs_compile.dir/src/nfa/mcsheng_compile.cpp.o 
    /Library/Developer/CommandLineTools/usr/bin/c++  -I/Users/ss/dev/tools/vcpkg/buildtrees/chimera/x64-osx-dbg -I/Users/ss/dev/tools/vcpkg/buildtrees/chimera/src/28a3d3326e-133ce01161.clean -I/Users/ss/dev/tools/vcpkg/buildtrees/chimera/src/28a3d3326e-133ce01161.clean/src -isystem /Users/ss/dev/tools/vcpkg/buildtrees/chimera/src/28a3d3326e-133ce01161.clean/include -isystem /Users/ss/dev/tools/vcpkg/installed/x64-osx/include -fPIC -march=native -mtune=native -O2 -std=c++11 -Wall -Wextra -Wshadow -Wswitch -Wreturn-type -Wcast-qual -Wno-deprecated -Wnon-virtual-dtor -fno-strict-aliasing -Werror -fno-omit-frame-pointer -Wvla -Wpointer-arith -Wno-self-assign -Wno-unused-const-variable -Wno-ignored-attributes -Wno-redundant-move -Wweak-vtables -Wmissing-declarations  -g -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -MD -MT CMakeFiles/hs_compile.dir/src/nfa/mcsheng_compile.cpp.o -MF CMakeFiles/hs_compile.dir/src/nfa/mcsheng_compile.cpp.o.d -o CMakeFiles/hs_compile.dir/src/nfa/mcsheng_compile.cpp.o -c /Users/ss/dev/tools/vcpkg/buildtrees/chimera/src/28a3d3326e-133ce01161.clean/src/nfa/mcsheng_compile.cpp
    /Users/ss/dev/tools/vcpkg/buildtrees/chimera/src/28a3d3326e-133ce01161.clean/src/nfa/mcsheng_compile.cpp:1170:13: error: variable 'total_daddy' set but not used [-Werror,-Wunused-but-set-variable]
            u16 total_daddy = 0;
                ^
    1 error generated.
    

    And it is caused by clang-13 introducing -Wunused-but-set-variable by default, which makes compilation fails. Similar with this issue

    opened by niyue 1
  • Session-aware-allocation supporting with hyperscan

    Session-aware-allocation supporting with hyperscan

    hi, everybody I am using hyperscan do some serious heavy-load query works in c++. For the best performance, our project hire session-aware-memory-allocation technology, just like Google's Arena. pls see Protobuf's Arena Document. I noticed that hyperscan library support use customized allocation function. from hs_set_allocator but this API was not designed for our scenairo, we cannot passing a c++ object's member funtion as c-style-function-ptr, so I need an another parameter just like a void*, then I can pass the Arena object ptr into the stateless global allocation function. hs_error_t hs_set_allocator(void*(size_t, void*) alloc_func, void* ctx, void(void*, void*) free_func, void* ctx) I hope u guys like this idea, and implement it, it can really boost hyperscan in some heavy-load quering services.

    opened by hurricane1026 3
  • SEGV hs_scan with HS_FLAG_SOM_LEFTMOST

    SEGV hs_scan with HS_FLAG_SOM_LEFTMOST

    Hello there,

    I was experimenting with scratch buffer allocation and noticed that one specific pattern would cause a crash. The code I used is as follow, I have 2 similar patterns and only allocate the scratch buffer for pattern1 but not pattern2:

    #include <hs/hs_compile.h>
    #include <stdio.h>
    #include <string.h>
    #include <hs/hs.h>
    
    static int eventHandler(unsigned int id, unsigned long long from,
                            unsigned long long to, unsigned int flags, void *ctx) {
        printf("Match for pattern \"%s\" at offset %llu\n", (char *)ctx, to);
        return 0;
    }
    
    int main(int argc, char *argv[]) {
        const char *pattern1 = "^(ftps?|https?).*?\\?+$";
        const char *pattern2 = "^(ftps?|https?)://.*$";
    
        hs_compile_error_t *compile_err;
        hs_database_t *database1;
        hs_compile(pattern1, HS_FLAG_SOM_LEFTMOST, HS_MODE_BLOCK, NULL, &database1,
                       &compile_err);
    
        hs_database_t *database2;
        hs_compile(pattern2, HS_FLAG_SOM_LEFTMOST, HS_MODE_BLOCK, NULL, &database2,
                       &compile_err);
    
        const char *inputData = "https://foo.bar/";
        hs_scratch_t *scratch = NULL;
        hs_alloc_scratch(database1, &scratch);
        hs_scan(database1, inputData, strlen(inputData), 0, scratch, eventHandler,
                    (void*)pattern1);
        hs_scan(database2, inputData, strlen(inputData), 0, scratch, eventHandler,
                    (void*)pattern2);
    
        return 0;
    }
    

    Expected results: the second hs_scan should return an error, since the scratch buffer was not allocated properly for that pattern Actual results: the second hs_scan is SEGV, here is the stack trace:

    #0  0x00007ffff7c3b5cd in memset (__len=32, __ch=0, __dest=0x0)
        at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71
    #1  fatbit_clear (bits=0x0) at ./src/util/fatbit.h:60
    #2  recordAnchoredLiteralMatch (end=8, anch_id=0, scratch=0x7fffcc00f240, t=<optimized out>)
        at ./src/rose/program_runtime.c:101
    #3  roseRunProgram (t=<optimized out>, scratch=<optimized out>, programOffset=<optimized out>, som=0,
        end=<optimized out>, prog_flags=<optimized out>) at ./src/rose/program_runtime.c:2278
    #4  0x00007ffff7c22fd0 in roseAnchoredCallback ([email protected]=0, end=<optimized out>,
        [email protected]=1888, [email protected]=0x7fffcc00f240) at ./src/rose/match.c:215
    #5  0x00007ffff7bd5a04 in doComplexReport (cached_accept_id=<synthetic pointer>,
        cached_accept_state=<synthetic pointer>, eod=0 '\000', loc=8, s=21, m=<optimized out>,
        ctxt=0x7fffcc00f240, cb=0x7ffff7c22f70 <roseAnchoredCallback>) at ./src/nfa/mcclellan.c:74
    #6  mcclellanExec8_i (mode=CALLBACK_OUTPUT, c_final=0x0, single=0 '\000', ctxt=0x7fffcc00f240,
        cb=0x7ffff7c22f70 <roseAnchoredCallback>, offAdj=0, len=8, buf=0x7fffd4014580 "https://foo.bar/",
        state=<synthetic pointer>, m=<optimized out>) at ./src/nfa/mcclellan.c:502
    #7  nfaExecMcClellan8_Bi (single=0 '\000', context=0x7fffcc00f240,
        length=8, buffer=0x7fffd4014580 "https://foo.bar/",
        offset=0, n=<optimized out>) at ./src/nfa/mcclellan.c:922
    #8  nfaExecMcClellan8_B (n=<optimized out>, offset=0, buffer=0x7fffd4014580 "https://foo.bar/", length=8,
        cb=0x7ffff7c22f70 <roseAnchoredCallback>, context=0x7fffcc00f240) at ./src/nfa/mcclellan.c:945
    #9  0x00007ffff7c0dbe1 in runAnchoredTableBlock (scratch=0x7fffcc00f240, atable=<optimized out>,
        t=0x555557cfde00) at ./src/rose/block.c:63
    #10 roseBlockAnchored (scratch=0x7fffcc00f240, t=0x555557cfde00) at ./src/rose/block.c:212
    #11 roseBlockExec (t=<optimized out>, scratch=<optimized out>) at ./src/rose/block.c:395
    #12 0x00007ffff7b44f4f in rawBlockExec (scratch=0x7fffcc00f240, rose=0x555557cfde00)
        at ./src/runtime.c:188
    #13 hs_scan (db=<optimized out>, data=<optimized out>, length=15, flags=<optimized out>,
        scratch=0x7fffcc00f240, onEvent=<optimized out>, userCtx=<optimized out>) at ./src/runtime.c:419
    

    Note: there is no crash if I remove the HS_FLAG_SOM_LEFTMOST flag. Workaround: reallocating the scratch buffer with the second pattern solves the issue.

    It sounds to me that some sanity checks are missing inside hs_scan.

    opened by noboruma 2
Releases(v5.4.0)
  • v5.4.0(Jan 13, 2021)

    [5.4.0] 2020-12-31

    • Improvement on literal matcher "Fat Teddy" performance, including support for Intel(R) AVX-512 Vector Byte Manipulation Instructions (Intel(R) AVX-512 VBMI).
    • Introduce a new 32-state shuffle-based DFA engine ("Sheng32"). This improves scanning performance by leveraging AVX-512 VBMI.
    • Introduce a new 64-state shuffle-based DFA engine ("Sheng64"). This improves scanning performance by leveraging AVX-512 VBMI.
    • Introduce a new shuffle-based hybrid DFA engine ("McSheng64"). This improves scanning performance by leveraging AVX-512 VBMI.
    • Improvement on exceptional state handling performance for LimEx NFA, including support for AVX-512 VBMI.
    • Improvement on lookaround performance with new models, including support for AVX-512.
    • Improvement on DFA state space efficiency.
    • Optimization on decision of NFA/DFA generation.
    • hsbench: add CSV dump support for hsbench.
    • Bugfix for cmake error on Icelake under release mode.
    • Bugfix in find_vertices_in_cycles() to avoid self-loop checking in SCC.
    • Bugfix for issue #270: fix return value handling in chimera.
    • Bugfix for issue #284: use correct free function in logical combination.
    • Add BUILD_EXAMPLES cmake option to enable example code compilation. (#260)
    • Some typo fixing. (#242, #259)

    By using this document, in addition to any agreements you have with Intel, you accept the terms set forth below.

    You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

    INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.

    Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined.” Do not finalize a design with this information.

    The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

    Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

    Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2019, Intel Corporation. All rights reserved.

    Source code(tar.gz)
    Source code(zip)
  • v5.3.0(May 25, 2020)

    [5.3.0] 2020-05-15

    • Improvement on literal matcher "Teddy" performance, including support for Intel(R) AVX-512 Vector Byte Manipulation Instructions (Intel(R) AVX-512 VBMI).
    • Improvement on single-byte/two-byte matching performance, including support for Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512).
    • hsbench: add hyphen support for -T option.
    • tools/fuzz: add test scripts for synthetic pattern generation.
    • Bugfix for acceleration path analysis in LimEx NFA.
    • Bugfix for duplicate matches for Small-write engine.
    • Bugfix for UTF8 checking problem for hscollider.
    • Bugfix for issue #205: avoid crash of hs_compile_lit_multi() with clang and ASAN.
    • Bugfix for issue #211: fix error in db_check_platform() function.
    • Bugfix for issue #217: fix cmake parsing issue of CPU arch for non-English locale.
    • Bugfix for issue #228: avoid undefined behavior when calling close() after fdopendir() in loadExpressions().
    • Bugfix for issue #239: fix hyperscan compile issue under gcc-10.
    • Add VLAN packets processing capability in pcap analysis script. (#214)
    • Avoid extra convert instruction for "Noodle". (#221)
    • Add Hyperscan version marcro in hs.h. (#222)

    By using this document, in addition to any agreements you have with Intel, you accept the terms set forth below.

    You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

    INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.

    Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined.” Do not finalize a design with this information.

    The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

    Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

    Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2020, Intel Corporation. All rights reserved.

    Source code(tar.gz)
    Source code(zip)
  • v5.2.1(Oct 30, 2019)

    [5.2.1] 2019-10-13

    • Bugfix for issue #186: fix compile issue when BUILD_SHARED_LIBS is on in release mode.
    • Disable redundant move check for older compiler versions.

    By using this document, in addition to any agreements you have with Intel, you accept the terms set forth below.

    You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

    INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.

    Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined.” Do not finalize a design with this information.

    The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

    Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

    Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2019, Intel Corporation. All rights reserved.

    Source code(tar.gz)
    Source code(zip)
  • v5.2.0(Aug 13, 2019)

    [5.2.0] 2019-07-12

    • Literal API: add new API hs_compile_lit() and hs_compile_lit_multi() to process pure literal rule sets. The 2 literal APIs treat each expression text in a literal sense without recognizing any regular grammers.
    • Logical combination: add support for purely negative combinations, which report match at EOD in case of no sub-expressions matched.
    • Windows porting: support shared library (DLL) on Windows with available tools hscheck, hsbench and hsdump.
    • Bugfix for issue #148: fix uninitialized use of scatter_unit_uX due to padding.
    • Bugfix for issue #155: fix numerical result out of range error.
    • Bugfix for issue #165: avoid corruption of pending combination report in streaming mode.
    • Bugfix for issue #174: fix scratch free issue when memory allocation fails.

    By using this document, in addition to any agreements you have with Intel, you accept the terms set forth below.

    You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

    INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.

    Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined.” Do not finalize a design with this information.

    The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

    Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

    Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2019, Intel Corporation. All rights reserved.

    Source code(tar.gz)
    Source code(zip)
  • v5.1.1(Apr 10, 2019)

    [5.1.1] 2019-04-03

    Known Issues and Limitations: There are no known issues with this release.

    Resolved Issues:

    • Add extra detection and handling when invalid rose programs are triggered.
    • Bugfix for issue #136: fix CMake parsing of CPU architecture for GCC-9.
    • Bugfix for issue #137: avoid file path impact on fat runtime build.
    • Bugfix for issue #141: fix rose literal programs for multi-pattern matching when no pattern ids are provided.
    • Bugfix for issue #144: fix library install path in pkg-config files.

    By using this document, in addition to any agreements you have with Intel, you accept the terms set forth below.

    You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

    INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.

    Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined.” Do not finalize a design with this information.

    The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

    Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

    Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2019, Intel Corporation. All rights reserved.

    Source code(tar.gz)
    Source code(zip)
  • v5.1.0(Jan 31, 2019)

    [5.1.0] 2019-01-31

    New Features:

    • Improve DFA state compression by wide-state optimization to reduce bytecode size.
    • Improve interpreter runtime handling for pure literal patterns to boost the performance of pure literal matching.
    • Improve representation of interpreter to boost overall performance.

    Known Issues and Limitations: There are no known issues with this release.

    Resolved Issues:

    • Bugfix for logical combinations: fix error reporting combination's match in case of sub-expression has EOD match under streaming mode.
    • Bugfix for logical combinations: fix miss reporting combination's match under vacuous input.
    • Bugfix for issue #104: fix compile error with Boost 1.68.0.
    • Bugfix for issue #127: avoid pcre error for hscollider with installed PCRE package.
    • Update version of PCRE used by testing tools as a syntax and semantic reference to PCRE 8.41 or above.
    • Fix github repo address in doc.

    By using this document, in addition to any agreements you have with Intel, you accept the terms set forth below.

    You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

    INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.

    Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined.” Do not finalize a design with this information.

    The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

    Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

    Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2019, Intel Corporation. All rights reserved.

    Source code(tar.gz)
    Source code(zip)
  • v5.0.0(Jul 9, 2018)

    [5.0.0] 2018-07-09

    • Introduce chimera hybrid engine of Hyperscan and PCRE, to fully support PCRE syntax as well as to take advantage of the high performance nature of Hyperscan.
    • New API feature: logical combinations (AND, OR and NOT) of patterns in a given pattern set.
    • Windows porting: hsbench, hscheck, hscollider and hsdump tools now available on Windows 8 or newer.
    • Improve undirected graph implementation to avoid graph copy and reduce compile time.
    • Bugfix for issue #86: enable hscollider for installed PCRE package.
    Source code(tar.gz)
    Source code(zip)
  • v4.7.0(Jan 23, 2018)

    [4.7.0] 2018-01-24

    • Introduced hscollider pattern testing tool, for validating Hyperscan match behaviour against PCRE.
    • Introduced hscheck pattern compilation tool.
    • Introduced hsdump development tool for producing information about Hyperscan pattern compilation.
    • New API feature: extended approximate matching support for Hamming distance.
    • Bugfix for issue #69: Force C++ linkage in Xcode.
    • Bugfix for issue #73: More documentation for hs_close_stream().
    • Bugfix for issue #78: Fix for fat runtime initialisation when used as a shared library.
    Source code(tar.gz)
    Source code(zip)
  • v4.6.0(Sep 22, 2017)

    [4.6.0] 2017-09-22

    • New API feature: stream state compression. This allows the user to compress and restore state for streams to reduce memory usage.
    • Many improvements to literal matching performance, including more support for Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512).
    • Compile time improvements, mainly reducing compiler memory allocation. Also results in reduced compile time for some pattern sets.
    • Bugfix for issue #62: fix error building Hyperscan using older versions of Boost.
    • Small updates to fix warnings identified by Coverity.
    Source code(tar.gz)
    Source code(zip)
  • v4.5.2(Jul 26, 2017)

    [4.5.2] 2017-07-26

    • Bugfix for issue #57: Treat characters between \Q..\E as codepoints in UTF8 mode.
    • Bugfix for issue #60: Use a portable flag for mktemp for fat runtime builds.
    • Bugfix for fat runtime builds on AVX-512 capable machines with Hyperscan's AVX-512 support disabled.
    Source code(tar.gz)
    Source code(zip)
  • v4.5.1(Jun 16, 2017)

    [4.5.1] 2017-06-16

    • Bugfix for issue #56: workaround for gcc-4.8 C++11 defect.
    • Bugfix for literal matching table generation, reversing a regression in performance for some literal matching cases.
    • Bugfixes for hsbench, related to multicore benchmarking, portability fixes for FreeBSD, and clarifying output results.
    • CMake: removed a duplicate else branch that causes very recent (v3.9) builds of CMake to fail.
    Source code(tar.gz)
    Source code(zip)
  • v4.5.0(Jun 9, 2017)

    [4.5.0] 2017-06-09

    • New API feature: approximate matching using the "edit distance" extended parameter. This allows the user to request all matches that are a given edit distance from an exact match for a pattern.
    • Initial support for Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512), disabled by default. To enable it, pass -DBUILD_AVX512=1 to cmake.
    • Major compile time improvements in many subsystems, reducing compile time significantly for many large pattern sets.
    • Internal reworking of literal matchers to operate on literals of at most eight characters, with subsequent confirmation done in the Rose interpreter. This reduces complexity and bytecode size and improves performance for many pattern sets.
    • Improve performance of the FDR literal matcher front end.
    • Improve bucket assignment and other heuristics governing the FDR literal matcher.
    • Improve optimisation passes that take advantage of extended parameter constraints (min_offset, etc).
    • Introduce further lookaround specialisations to improve scanning performance.
    • Optimise Rose interpreter construction to reduce the length of programs generated in some situations.
    • Remove the old "Rose" pattern decomposition analysis pass in favour of the new "Violet" pass introduced in Hyperscan 4.3.0.
    • In streaming mode, allow exhaustion (where the stream can no longer produce matchers) to be detected in more situations, improving scanning performance.
    • Improve parsing of control verbs (such as (*UTF8)) that can only occur at the beginning of the pattern. Combinations of supported verbs in any order are now permitted.
    • Update version of PCRE used by testing tools as a syntax and semantic reference to PCRE 8.40.
    • Tuning support for Intel(R) microarchitecture code names Skylake, Skylake Server, Goldmont.
    • CMake: when building a native build with a version of GCC that doesn't recognise the host compiler, tune for the microarch selected by -march=native.
    • CMake: don't fail if SQLite (which is only required to build the hsbench tool) is not present.
    • CMake: detect libc++ directly and use that to inform the Boost version requirement.
    • Bugfix for issue #51: make the fat runtime build wrapper less fragile.
    • Bugfix for issues #46, #52: use sqlite3_errmsg() to allow SQLite 3.6.x to be used. Thanks to @EaseTheWorld for the PR.
    Source code(tar.gz)
    Source code(zip)
  • v4.4.1(Mar 1, 2017)

    [4.4.1] 2017-02-28

    • Bugfixes to fix issues where stale data was being referenced in scratch memory. In particular this may have resulted in hs_close_stream() referencing data from other previously scanned streams. This may result in incorrect matches being been reported.
    Source code(tar.gz)
    Source code(zip)
  • v4.4.0(Jan 20, 2017)

    [4.4.0] 2017-01-20

    • Introduce the "fat runtime" build. This will build several variants of the Hyperscan scanning engine specialised for different processor feature sets, and use the appropriate one for the host at runtime. This uses the "ifunc" indirect function attribute provided by GCC and is currently available on Linux only, where it is the default for release builds.
    • New API function: add the hs_valid_platform() function. This function tests whether the host provides the SSSE3 instruction set required by Hyperscan.
    • Introduce a new standard benchmarking tool, "hsbench". This provides an easy way to measure Hyperscan's performance for a particular set of patterns and corpus of data to be scanned.
    • Introduce a 64-bit GPR LimEx NFA model, which uses 64-bit GPRs on 64-bit hosts and SSE registers on 32-bit hosts.
    • Introduce a new DFA model ("McSheng") which is a hybrid of the existing McClellan and Sheng models. This improves scanning performance for some cases.
    • Introduce lookaround specialisations to improve scanning performance.
    • Improve the handling of long literals by moving confirmation to the Rose interpreter and simplifying the hash table used to track them in streaming mode.
    • Improve compile time optimisation for removing redundant paths from expression graphs.
    • Build: improve support for building with MSVC toolchain.
    • Reduce the size of small write DFAs used for small scans in block mode.
    • Introduce a custom graph type (ue2_graph) used in place of the Boost Graph Library's adjacency_list type. Improves compile time performance and type safety.
    • Improve scanning performance of the McClellan DFA.
    • Bugfix for a very unusual SOM case where the incorrect start offset was reported for a match.
    • Bugfix for issue #37, removing execute permissions from some source files.
    • Bugfix for issue #41, handle Windows line endings in pattern files.
    Source code(tar.gz)
    Source code(zip)
  • v4.3.2(Nov 14, 2016)

    [4.3.2] 2016-11-15

    • Bugfix for issue #39. This small change is a workaround for an issue in Boost 1.62. The fix has been submitted to Boost for inclusion in a future release.
    Source code(tar.gz)
    Source code(zip)
  • v4.3.1(Aug 29, 2016)

    [4.3.1] 2016-08-29

    • Bugfix for issue #30. In recent versions of Clang, a write to a variable was being elided, resulting in corrupted stream state after calling hs_reset_stream().
    Source code(tar.gz)
    Source code(zip)
  • v4.2.1(Aug 29, 2016)

    [4.2.1] 2016-08-29

    • Bugfix for issue #30. In recent versions of Clang, a write to a variable was being elided, resulting in corrupted stream state after calling hs_reset_stream().
    Source code(tar.gz)
    Source code(zip)
  • v4.3.0(Aug 24, 2016)

    [4.3.0] 2016-08-24

    • Introduce a new analysis pass ("Violet") used for decomposition of patterns into literals and smaller engines.
    • Introduce a new container engine ("Tamarama") for infix and suffix engines that can be proven to run exclusively of one another. This reduces stream state for pattern sets with many such engines.
    • Introduce a new shuffle-based DFA engine ("Sheng"). This improves scanning performance for pattern sets where small engines are generated.
    • Improve the analysis used to extract extra mask information from short literals.
    • Reduced compile time spent in equivalence class analysis.
    • Build: frame pointers are now only omitted for 32-bit release builds.
    • Build: Workaround for C++ issues reported on FreeBSD/libc++ platforms. (github issue #27)
    • Simplify the LimEx NFA with a unified "variable shift" model, which reduces the number of different NFA code paths to one per model size.
    • Allow some anchored prefixes that may squash the literal to which they are attached to run eagerly. This improves scanning performance for some patterns.
    • Simplify and improve EOD ("end of data") matching, using the interpreter for all operations.
    • Elide unnecessary instructions in the Rose interpreter at compile time.
    • Reduce the number of inlined instantiations of the Rose interpreter in order to reduce instruction cache pressure.
    • Small improvements to literal matcher acceleration.
    • Parser: ignore \E metacharacters that are not preceded by \Q. This conforms to PCRE's behaviour, rather than returning a compile error.
    • Check for misaligned memory when allocating an error structure in Hyperscan's compile path and return an appropriate error if detected.
    Source code(tar.gz)
    Source code(zip)
  • v4.2.0(Jun 1, 2016)

    [4.2.0] 2016-05-31

    • Introduce an interpreter for many complex actions to replace the use of internal reports within the core of Hyperscan (the "Rose" engine). This improves scanning performance and reduces database size for many pattern sets.
    • Many enhancements to the acceleration framework used by NFA and DFA engines, including more flexible multibyte implementations and more AVX2 support. This improves scanning performance for many pattern sets.
    • Improved prefiltering support for complex patterns containing very large bounded repeats (R{M,N} with large N).
    • Improve scanning performance of pattern sets with a very large number of EOD-anchored patterns.
    • Improve scanning performance of large pattern sets that use the HS_FLAG_SINGLEMATCH flag.
    • Improve scanning performance of pattern sets that contain a single literal by improving the "Noodle" literal matcher.
    • Small reductions in total stream state for many pattern sets.
    • Improve runtime detection of AVX2 support.
    • Disable -Werror for release builds, in order to behave better for packagers and users with different compiler combinations than those that we test.
    • Improve support for building on Windows with MSVC 2015 (github issue #14). Support for Hyperscan on Windows is still experimental.
    • Small updates to fix warnings identified by Coverity.
    • Remove Python codegen for the "FDR" and "Teddy" literal matchers. These are now implemented directly in C code.
    • Remove the specialist "Sidecar" engine in favour of using our more general repeat engines.
    • New API function: add the hs_expression_ext_info() function. This is a variant of hs_expression_info() that can accept patterns with extended parameters.
    • New API error value: add the HS_SCRATCH_IN_USE error, which is returned when Hyperscan detects that a scratch region is already in use on entry to an API function.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.0(Feb 3, 2016)

    [4.1.0] 2015-12-18

    • Update version of PCRE used by testing tools as a syntax and semantic reference to PCRE 8.38.
    • Small updates to fix warnings identified by Coverity.
    • Clean up and unify exception handling behaviour across GPR and SIMD NFA models.
    • Fix bug in handling of bounded repeat triggers with large gaps between them for sparse repeat model.
    • Correctly reject POSIX collating elements ([.ch.], [=ch=]) in the parser. These are not supported by Hyperscan.
    • Add support for quoted sequences (\Q...\E) inside character classes.
    • Simplify FDR literal matcher runtime by removing some static specialization.
    • Fix handling of the POSIX [:graph:], [:print:] and [:punct:] character classes to match the behaviour of PCRE 8.38 in both standard operation and with the UCP flag set. (Note: some bugs were fixed in this area in PCRE 8.38.) Previously Hyperscan's behaviour was the same as versions of PCRE before 8.34.
    • Improve performance when compiling pattern sets that include a large number of similar bounded repeat constructs. (github issue #9)
    Source code(tar.gz)
    Source code(zip)
  • v4.0.1(Aug 24, 2016)

    [4.0.1] 2015-10-30

    • Minor cleanups to test code.
    • CMake and other build system improvements.
    • API update: allow hs_reset_stream() and hs_reset_and_copy_stream() to be supplied with a NULL scratch pointer if no matches are required. This is in line with the behaviour of hs_close_stream().
    • Disallow bounded repeats with a very large minimum repeat but no maximum, i.e. {N,} for very large N.
    • Reduce compile memory usage in literal set explansion for some large cases.
    Source code(tar.gz)
    Source code(zip)
  • v4.0.0(Aug 24, 2016)

regular expression library

Oniguruma https://github.com/kkos/oniguruma Oniguruma is a modern and flexible regular expressions library. It encompasses features from different reg

K.Kosako 1.7k Jun 19, 2022
RE2 is a fast, safe, thread-friendly alternative to backtracking regular expression engines like those used in PCRE, Perl, and Python. It is a C++ library.

This is the source code repository for RE2, a regular expression library. For documentation about how to install and use RE2, visit https://github.co

Google 7k Jun 22, 2022
A Compile time PCRE (almost) compatible regular expression matcher.

Compile time regular expressions v3 Fast compile-time regular expressions with support for matching/searching/capturing during compile-time or runtime

Hana Dusíková 2.4k Jun 25, 2022
Perl Incompatible Regular Expressions library

This is PIRE, Perl Incompatible Regular Expressions library. This library is aimed at checking a huge amount of text against relatively many regular

Yandex 318 Jun 5, 2022
Onigmo is a regular expressions library forked from Oniguruma.

Onigmo (Oniguruma-mod) https://github.com/k-takata/Onigmo Onigmo is a regular expressions library forked from Oniguruma. It focuses to support new exp

K.Takata 566 Jun 19, 2022
The approximate regex matching library and agrep command line tool.

Introduction TRE is a lightweight, robust, and efficient POSIX compliant regexp matching library with some exciting features such as approximate (fuzz

Ville Laurikari 669 Jun 23, 2022
C++ regular expressions made easy

CppVerbalExpressions C++ Regular Expressions made easy VerbalExpressions is a C++11 Header library that helps to construct difficult regular expressio

null 357 Jun 26, 2022
A non-backtracking NFA/DFA-based Perl-compatible regex engine matching on large data streams

Name libsregex - A non-backtracking NFA/DFA-based Perl-compatible regex engine library for matching on large data streams Table of Contents Name Statu

OpenResty 591 Jun 24, 2022
Glob pattern to regex translator in C++11. Optionally, directory traversal with glob pattern in C++17. Header-only library.

Glob pattern to regex translator in C++11. Optionally, directory traversal with glob pattern in C++17. Header-only library.

Takayuki MATSUOKA 3 Oct 27, 2021
SRL-CPP is a Simple Regex Language builder library written in C++11 that provides an easy to use interface for constructing both simple and complex regex expressions.

SRL-CPP SRL-CPP is a Simple Regex Language builder library written in C++11 that provides an easy to use interface for constructing both simple and co

Telepati 0 Mar 9, 2022
A portable fork of the high-performance regular expression matching library

Vectorscan? A fork of Intel's Hyperscan, modified to run on more platforms. Currently ARM NEON/ASIMD is 100% functional, and Power VSX are in developm

VectorCamp 191 Jun 17, 2022
A small implementation of regular expression matching engine in C

cregex cregex is a compact implementation of regular expression (regex) matching engine in C. Its design was inspired by Rob Pike's regex-code for the

Jim Huang 70 Apr 2, 2022
regular expression library

Oniguruma https://github.com/kkos/oniguruma Oniguruma is a modern and flexible regular expressions library. It encompasses features from different reg

K.Kosako 1.7k Jun 19, 2022
RE2 is a fast, safe, thread-friendly alternative to backtracking regular expression engines like those used in PCRE, Perl, and Python. It is a C++ library.

This is the source code repository for RE2, a regular expression library. For documentation about how to install and use RE2, visit https://github.co

Google 7k Jun 22, 2022
A Compile time PCRE (almost) compatible regular expression matcher.

Compile time regular expressions v3 Fast compile-time regular expressions with support for matching/searching/capturing during compile-time or runtime

Hana Dusíková 2.4k Jun 25, 2022
Header-only ECMAScript (JavaScript) compatible regular expression engine

SRELL (std::regex-like library) is a regular expression template library for C++ and has native support for UTF-8, UTF-16, and UTF-32. This is up-to-d

Dmitry Atamanov 4 Mar 11, 2022
Love 6's Regular Expression Engine. Support Concat/Select/Closure Basic function. Hope u can enjoy this tiny engine :)

Regex_Engine Love 6's Blog Website: https://love6.blog.csdn.net/ Love 6's Regular Expression Engine Hope u can love my tiny regex engine :) maybe a fe

Love6 2 May 24, 2022
Fast regular expression grep for source code with incremental index updates

Fast regular expression grep for source code with incremental index updates

Arseny Kapoulkine 247 Jun 19, 2022
C++ Matrix -- High performance and accurate (e.g. edge cases) matrix math library with expression template arithmetic operators

Matrix This is a math and arithmetic matrix library. It has stood many years of performing in mission critical production for financial systems. It ha

Hossein Moein 67 Jun 5, 2022
Experimental and Comparative Performance Measurements of High Performance Computing Based on OpenMP and MPI

High-Performance-Computing-Experiments Experimental and Comparative Performance Measurements of High Performance Computing Based on OpenMP and MPI 实验结

Jiang Lu 1 Nov 27, 2021
An eventing framework for building high performance and high scalability systems in C.

NOTE: THIS PROJECT HAS BEEN DEPRECATED AND IS NO LONGER ACTIVELY MAINTAINED As of 2019-03-08, this project will no longer be maintained and will be ar

Facebook Archive 1.7k Jun 15, 2022
OceanBase is an enterprise distributed relational database with high availability, high performance, horizontal scalability, and compatibility with SQL standards.

What is OceanBase database OceanBase Database is a native distributed relational database. It is developed entirely by Alibaba and Ant Group. OceanBas

OceanBase 4.4k Jun 27, 2022
High Performance 3D Game Engine, with a high emphasis on Rendering

Electro High Performance 3D Game Engine, with a high emphasis on Rendering MainFeatures Rendering PBR Renderer (Cook–Torrance GGX) IBL (Image Based Li

Surge 45 Jun 18, 2022
An eventing framework for building high performance and high scalability systems in C.

NOTE: THIS PROJECT HAS BEEN DEPRECATED AND IS NO LONGER ACTIVELY MAINTAINED As of 2019-03-08, this project will no longer be maintained and will be ar

Meta Archive 1.7k Jun 15, 2022
Perl Incompatible Regular Expressions library

This is PIRE, Perl Incompatible Regular Expressions library. This library is aimed at checking a huge amount of text against relatively many regular

Yandex 318 Jun 5, 2022
Onigmo is a regular expressions library forked from Oniguruma.

Onigmo (Oniguruma-mod) https://github.com/k-takata/Onigmo Onigmo is a regular expressions library forked from Oniguruma. It focuses to support new exp

K.Takata 566 Jun 19, 2022
The dgSPARSE Library (Deep Graph Sparse Library) is a high performance library for sparse kernel acceleration on GPUs based on CUDA.

dgSPARSE Library Introdution The dgSPARSE Library (Deep Graph Sparse Library) is a high performance library for sparse kernel acceleration on GPUs bas

dgSPARSE 49 Jun 17, 2022
Functional programming style pattern-matching library for C++

Mach7: Pattern Matching for C++ by Yuriy Solodkyy, Gabriel Dos Reis, Bjarne Stroustrup Abstract Pattern matching is an abstraction mechanism that can

Yuriy Solodkyy 1.2k Jun 24, 2022
The approximate regex matching library and agrep command line tool.

Introduction TRE is a lightweight, robust, and efficient POSIX compliant regexp matching library with some exciting features such as approximate (fuzz

Ville Laurikari 669 Jun 23, 2022