A portable fork of the high-performance regular expression matching library

Overview

Vectorscan?

A fork of Intel's Hyperscan, modified to run on more platforms. Currently ARM NEON/ASIMD is 100% functional, and Power VSX are in development. ARM SVE2 will be implemented when harwdare becomes accessible to the developers. More platforms will follow in the future, on demand/request.

Vectorscan will follow Intel's API and internal algorithms where possible, but will not hesitate to make code changes where it is thought of giving better performance or better portability. In addition, the code will be gradually simplified and made more uniform and all architecture specific -currently Intel- #ifdefs will be removed and abstracted away.

Why the fork?

Originally, the ARM porting was supposed to be merged into Intel's own Hyperscan, and 2 Pull Requests had been made to the project for this reason (1, 2). Unfortunately, the PRs were rejected for now and the forseeable future, thus we have created Vectorscan for our own multi-architectural and opensource collaborative needs.

What is Hyperscan?

Hyperscan is a high-performance multiple regex matching library. It follows the regular expression syntax of the commonly-used libpcre library, but is a standalone library with its own C API.

Hyperscan uses hybrid automata techniques to allow simultaneous matching of large numbers (up to tens of thousands) of regular expressions and for the matching of regular expressions across streams of data.

Vectorscan is typically used in a DPI library stack, just like Hyperscan.

Cross Compiling for AArch64

  • To cross compile for AArch64, first adjust the variables set in cmake/setenv-arm64-cross.sh.
    • export CROSS=<arm-cross-compiler-dir>/bin/aarch64-linux-gnu-
    • export CROSS_SYS=<arm-cross-compiler-system-dir>
    • export BOOST_PATH=<boost-source-dir>
  • Set the environment variables:
    • source cmake/setenv-arm64-cross.sh
  • Configure Vectorscan:
    • mkdir <build-dir-name>
    • cd <build-dir>
    • cmake -DCROSS_COMPILE_AARCH64=1 <hyperscan-source-dir> -DCMAKE_TOOLCHAIN_FILE=<hyperscan-source-dir>/cmake/arm64-cross.cmake
  • Build Vectorscan:
    • make -jT where T is the number of threads used to compile.
    • cmake --build . -- -j T can also be used instead of make.

Compiling for SVE

The following cmake variables can be set in order to target Arm's Scalable Vector Extension. They are listed in ascending order of strength, with cmake detecting whether the feature is available in the compiler and falling back to a weaker version if not. Only one of these variables needs to be set as weaker variables will be implied as set.

  • BUILD_SVE
  • BUILD_SVE2
  • BUILD_SVE2_BITPERM

Documentation

Information on building the Hyperscan library and using its API is available in the Developer Reference Guide.

License

Vectorscan, like Hyperscan is licensed under the BSD License. See the LICENSE file in the project repository.

Versioning

The master branch on Github will always contain the most recent release of Hyperscan. Each version released to master goes through QA and testing before it is released; if you're a user, rather than a developer, this is the version you should be using.

Further development towards the next release takes place on the develop branch.

Get Involved

The official homepage for Vectorscan is at www.github.com/VectorCamp/vectorscan.

Original Hyperscan links

The official homepage for Hyperscan is at www.hyperscan.io.

If you have questions or comments, we encourage you to join the mailing list. Bugs can be filed by sending email to the list, or by creating an issue on Github.

If you wish to contact the Hyperscan team at Intel directly, without posting publicly to the mailing list, send email to [email protected].

Issues
  • cannot build on apple silicon

    cannot build on apple silicon

    I am trying to compile the project on my mac (MacBook Pro (14-inch, 2021), Monterey 12.0.1) I did the following steps:

    git clone [email protected]:VectorCamp/vectorscan.git 
    brew install make
    brew install boost
    brew install bagel
    brew install sqlite
    
    dd vectorscan
    mkdir build
    cd build
    cmake ..
    

    and I am getting:

    CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
      Compatibility with CMake < 2.8.12 will be removed from a future version of
      CMake.
    
      Update the VERSION argument <min> value or use a ...<max> suffix to tell
      CMake that the project does not need compatibility with older versions.
    
    
    -- The C compiler identification is AppleClang 13.0.0.13000027
    -- The CXX compiler identification is AppleClang 13.0.0.13000027
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Performing Test ARCH_X86_64
    -- Performing Test ARCH_X86_64 - Failed
    -- Performing Test ARCH_IA32
    -- Performing Test ARCH_IA32 - Failed
    -- Performing Test ARCH_AARCH64
    -- Performing Test ARCH_AARCH64 - Success
    -- Performing Test ARCH_ARM32
    -- Performing Test ARCH_ARM32 - Failed
    -- Default build type 'Release with debug info'
    -- using release build
    -- Boost version: 1.76.0
    -- Found PythonInterp: /usr/bin/python (found version "2.7.18")
    -- Build date: 2022-01-05
    -- Building static libraries
    -- Looking for include file unistd.h
    -- Looking for include file unistd.h - found
    -- Looking for C++ include arm_neon.h
    -- Looking for C++ include arm_neon.h - found
    -- Looking for posix_memalign
    -- Looking for posix_memalign - found
    -- Looking for _aligned_malloc
    -- Looking for _aligned_malloc - not found
    -- Performing Test HAS_C_HIDDEN
    -- Performing Test HAS_C_HIDDEN - Success
    -- Performing Test HAS_CXX_HIDDEN
    -- Performing Test HAS_CXX_HIDDEN - Success
    -- Looking for _LIBCPP_VERSION
    -- Looking for _LIBCPP_VERSION - found
    -- Performing Test HAVE_NEON
    -- Performing Test HAVE_NEON - Success
    -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED
    -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED - Success
    -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED
    -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED - Success
    -- Performing Test HAVE__BUILTIN_CONSTANT_P
    -- Performing Test HAVE__BUILTIN_CONSTANT_P - Success
    -- Performing Test C_FLAG_Wvla
    -- Performing Test C_FLAG_Wvla - Success
    -- Performing Test C_FLAG_Wpointer_arith
    -- Performing Test C_FLAG_Wpointer_arith - Success
    -- Performing Test C_FLAG_Wstrict_prototypes
    -- Performing Test C_FLAG_Wstrict_prototypes - Success
    -- Performing Test C_FLAG_Wmissing_prototypes
    -- Performing Test C_FLAG_Wmissing_prototypes - Success
    -- Performing Test CXX_FLAG_Wvla
    -- Performing Test CXX_FLAG_Wvla - Success
    -- Performing Test CXX_FLAG_Wpointer_arith
    -- Performing Test CXX_FLAG_Wpointer_arith - Success
    -- Performing Test CC_SELF_ASSIGN
    -- Performing Test CC_SELF_ASSIGN - Success
    -- Performing Test CXX_SELF_ASSIGN
    -- Performing Test CXX_SELF_ASSIGN - Success
    -- Performing Test CC_PAREN_EQUALITY
    -- Performing Test CC_PAREN_EQUALITY - Success
    -- Performing Test CXX_UNUSED_CONST_VAR
    -- Performing Test CXX_UNUSED_CONST_VAR - Success
    -- Performing Test CXX_IGNORED_ATTR
    -- Performing Test CXX_IGNORED_ATTR - Success
    -- Performing Test CXX_REDUNDANT_MOVE
    -- Performing Test CXX_REDUNDANT_MOVE - Success
    -- Performing Test CXX_WEAK_VTABLES
    -- Performing Test CXX_WEAK_VTABLES - Success
    -- Performing Test CXX_MISSING_DECLARATIONS
    -- Performing Test CXX_MISSING_DECLARATIONS - Success
    -- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS
    -- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS - Success
    -- Performing Test CXX_WUNUSED_VARIABLE
    -- Performing Test CXX_WUNUSED_VARIABLE - Success
    -- Performing Test CC_STRINGOP_OVERFLOW
    -- Performing Test CC_STRINGOP_OVERFLOW - Failed
    -- Building for current host CPU: -march= -mtune=native
    -- Looking for mmap
    -- Looking for mmap - not found
    -- Doxygen not found, unable to generate API reference
    -- Sphinx not found, unable to generate developer reference
    -- Found PkgConfig: //opt/homebrew/bin/pkg-config (found version "0.29.2")
    -- Checking for module 'libpcre>=8.41'
    --   Found libpcre, version 8.45
    -- PCRE version 8.41 or above
    -- Looking for pthread.h
    -- Looking for pthread.h - not found
    -- Could NOT find Threads (missing: Threads_FOUND)
    -- Checking for module 'sqlite3'
    --   Found sqlite3, version 3.36.0
    -- Performing Test SQLITE_VERSION_OK
    -- Performing Test SQLITE_VERSION_OK - Failed
    CMake Error at cmake/sqlite3.cmake:34 (message):
      sqlite3 is broken from 3.8.7 to 3.8.10 - please find a working version
    Call Stack (most recent call first):
      tools/hsbench/CMakeLists.txt:1 (include)
    

    Any change you know what could be missing so that I can update the doc?

    sqlite is there:

     brew install sqlite
    Running `brew update --preinstall`...
    ==> Auto-updated Homebrew!
    Updated 1 tap (homebrew/core).
    ==> Updated Formulae
    Updated 1 formula.
    
    Warning: sqlite 3.37.1 is already installed and up-to-date.
    To reinstall 3.37.1, run:
      brew reinstall sqlite
    

    and pthread as well:

    ls -la /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread
    total 96
    drwxr-xr-x   10 root  wheel    320 Jul 22 21:03 .
    drwxr-xr-x  295 root  wheel   9440 Jan  5 10:28 ..
    -rw-r--r--    1 root  wheel   5885 Jul 22 20:30 introspection.h
    -rw-r--r--    1 root  wheel  24358 Jul 22 20:30 pthread.h
    -rw-r--r--    1 root  wheel   1977 Jul 18 23:22 pthread_impl.h
    -rw-r--r--    1 root  wheel   3630 Jul 22 20:30 pthread_spis.h
    -rw-r--r--    1 root  wheel  10256 Jul 22 20:30 qos.h
    -rw-r--r--    1 root  wheel   1410 Jul 18 23:22 sched.h
    -rw-r--r--    1 root  wheel   2761 Jul 22 20:30 spawn.h
    -rw-r--r--    1 root  wheel   2062 Jul 22 20:30 stack_np.h
    

    and finally:

    cmake --version
    cmake version 3.22.1
    
    opened by epot 15
  • shuftiExec() and truffleExec() are vectorscan code hot spot for Snort3 on arm

    shuftiExec() and truffleExec() are vectorscan code hot spot for Snort3 on arm

    Vectorscan version: v5.4.2+vectorscan Hyperscan version: v5.4.2

    When Snort3 inspects one .pcap packets file with same configuration, the vectorscan module profiling result on arm (around 2900 milliseconds) is almost twice as much as hyperscan profiling result on x86 (around 1600 milliseconds). From the perf analysis result, shuftiExec() and truffleExec() are vectorscan code hot spot on arm, but this is not the case for x86 hyperscan. Any idea or suggestion for vectorscan here? Thanks.

    Snort 3 with Arm vectorscan perf result:

    arm

    Snort 3 with x86 hyperscan perf result:

    x86
    opened by liangxwa01 14
  • aarch64 and ppc64le: error: narrowing conversion of 'XYZ' from 'int' to 'char' [-Wnarrowing]

    aarch64 and ppc64le: error: narrowing conversion of 'XYZ' from 'int' to 'char' [-Wnarrowing]

    Building vectorscan 5.4.6 fails with GCC11 on openSUSE Tumbleweed aarch64 with:

    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s]   765 |         };
    [  316s]       |         ^
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-64' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-33' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-32' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-17' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-16' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-9' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-8' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-1' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-64' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-33' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-32' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-17' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-16' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-9' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-8' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-1' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-64' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-33' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-32' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-17' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-16' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-9' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-8' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-1' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-64' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-33' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-32' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-17' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-16' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-9' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-8' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-1' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-128' from 'int' to 'char' [-Wnarrowing]
    [  316s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.6/build/src/parser/Parser.cpp:765:9: error: narrowing conversion of '-65' from 'int' to 'char' [-Wnarrowing]
    
    bug 
    opened by ggardet 13
  • legal instruction on startup when running rspamd on aarch on Alpine Linux

    legal instruction on startup when running rspamd on aarch on Alpine Linux

    It ran fine in Alpine 3.14 when it was first released, but some change led to this error. I originally could not stop it.

    It worked on Alpine Edge at that time, now the same issue is happening on Edge.

    It does not create a core when it crashes like this, the log output is:

    2021-10-07 06:43:49 #5913(main) lua; lua_cfg_transform.lua:161: group excessqp has no symbols
    2021-10-07 06:43:49 #5913(main) lua; lua_cfg_transform.lua:161: group excessb64 has no symbols
    2021-10-07 06:43:49 #5913(main) <>; lua; lua_cfg_transform.lua:511: enable `options.check_all_filters` for neural network
    2021-10-07 06:43:49 #5913(main) <>; lua; lua_cfg_transform.lua:569: converted surbl rules to rbl rules
    2021-10-07 06:43:49 #5913(main) <>; lua; lua_cfg_transform.lua:583: converted emails rules to rbl rules
    2021-10-07 06:43:49 #5913(main) cfg; rspamd_rcl_maybe_apply_lua_transform: configuration has been transformed in Lua
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; cfg; rspamd_config_set_action_score: action add header has been already registered with priority 0, override it with new priority: 0, old score: nan
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; cfg; rspamd_config_set_action_score: action reject has been already registered with priority 0, override it with new priority: 0, old score: nan
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; cfg; rspamd_config_set_action_score: action greylist has been already registered with priority 0, override it with new priority: 0, old score: nan
    2021-10-07 06:43:49 #5913(main) rspamd_regexp_library_init: pcre2 is compiled with JIT for ARM-64 64bit (little endian + unaligned)
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; lua; init.lua:58: controller plugin neural; register webui path learn
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; lua; init.lua:58: controller plugin maps; register webui path query
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; lua; init.lua:58: controller plugin maps; register webui path list
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; lua; init.lua:58: controller plugin maps; register webui path query_specific
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; lua; init.lua:58: controller plugin selectors; register webui path list_transforms
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; lua; init.lua:58: controller plugin selectors; register webui path check_message
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; lua; init.lua:58: controller plugin selectors; register webui path list_extractors
    2021-10-07 06:43:49 #5913(main) <1wtzg4>; lua; init.lua:58: controller plugin selectors; register webui path check_selector
    

    There is more info here: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12822

    Now if I disable vectorscan support it runs without error, so it appears to be related to vectorscan even though the version hasn't changed.

    opened by a16bitsysop 12
  • build failure unless build type is release

    build failure unless build type is release

    Building for alpine linux I get this error (as build type has to be None for alpine)

    /hyperscan-5.3.1/unit/internal/masked_move.cpp:35:10: fatal error: util/masked_move.h: No such file or directory
    943   35 | #include "util/masked_move.h"
    

    https://gitlab.alpinelinux.org/a16bitsysop/aports/-/jobs/277952/raw

    If I patch and add "NONE" to the types that sets RELEASE_BUILD

    --- a/CMakeLists.txt
    +++ b/CMakeLists.txt
    @@ -30,7 +30,7 @@
         message(STATUS "Build type ${CMAKE_BUILD_TYPE}")
     endif()
    
    -if(CMAKE_BUILD_TYPE MATCHES RELEASE|RELWITHDEBINFO|MINSIZEREL)
    +if(CMAKE_BUILD_TYPE MATCHES NONE|RELEASE|RELWITHDEBINFO|MINSIZEREL)
         message(STATUS "using release build")
         set(RELEASE_BUILD TRUE)
     else()
    

    It compiles as it misses the problem files, but the tools are not compiled now.

    https://gitlab.alpinelinux.org/a16bitsysop/aports/-/jobs/277980

    I would like to keep the tools, as there is already an hyperscan-tools package on alpine.

    opened by a16bitsysop 11
  • macOS support

    macOS support

    Is there any chance we could support building/installing vectorscan under macOS?

    hyperscan supports building the project under macOS using clang, and many developers use macOS as their dev machine, but for this project, it seems hard coded to use gcc/Linux, which makes it not easy to get people to start using it.

    enhancement 
    opened by niyue 9
  • Different behavior on x64 and aarch64

    Different behavior on x64 and aarch64

    I noticed inconsitent matching behavior between x64 and aarch64 for certain regex and input.

    Setup

    Machine 1 (x64):

    [email protected]:~$ gcc --version
    gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
    Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
    [email protected]:~$ ragel -version
    Ragel State Machine Compiler version 6.10 March 2017
    Copyright (c) 2001-2009 by Adrian Thurston
    [email protected]:~$ uname -a
    Linux ubuntu 5.13.0-39-generic #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
    

    Machine 2 (aarch64) - QEMU Cortex-A72:

    [email protected]:~$ gcc --version
    gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
    Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
    [email protected]:~$ ragel -version
    Ragel State Machine Compiler version 6.10 March 2017
    Copyright (c) 2001-2009 by Adrian Thurston
    [email protected]:~$ uname -a
    Linux aarch64-vm 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:07:22 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
    

    Version

    Tested with vectorscan build from master and from release 5.4.6 - both exhibit same behavior.

    Test program

    I compile it using g++.

    #include <iostream>
    #include <cstring>
    #include "hs/hs.h"
    
    const auto match_cb = [](auto, auto, auto, auto, void* ctx) {
        *((bool*) ctx) = true;
        return 0;
    };
    
    bool vectorscan_scan(char* regex, const char* input, unsigned int id) {
        hs_database_t* db = nullptr;
        hs_scratch_t* scratch = nullptr;
        hs_compile_error_t* error;
        bool is_match = false;
        if (hs_compile(regex, HS_FLAG_DOTALL | HS_FLAG_SINGLEMATCH | HS_FLAG_UTF8 | HS_FLAG_CASELESS, HS_MODE_BLOCK, nullptr, &db, &error) != HS_SUCCESS) {
            printf("ERROR hs_compile(): %s\n", error->message);
            return false;
        }
        if (hs_alloc_scratch(db, &scratch) != HS_SUCCESS) {
            printf("ERROR hs_alloc_scratch()\n");
            return false;
        }
        if (hs_scan(db, input, strlen(input), id, scratch, match_cb, &is_match) != HS_SUCCESS) {
            printf("ERROR hs_compile(): %s\n", error->message);
            return false;
        }
        hs_free_database(db);
        hs_free_scratch(scratch);
        return is_match;
    }
    
    int main() {
        char regex[] = R"(schtasks(\.exe)?\s.*\/create.*cscript.*)";
        char input[] = R"(schtasks.exe /create "cscript abcd")";
        if (!vectorscan_scan(regex, input, 1)) {
            printf("FAIL - SHOULD MATCH\n");
        } else {
            printf("SUCCESS - OK\n");
        }
        return 0;
    }
    

    The problem

    When I run the program on x64 the input is successfully matched, however on ARM machine it isn't. I observed that the issue occurs only when both HS_FLAG_UTF8 | HS_FLAG_CASELESS flags are provided. Sometimes small tweaks in regex make the problem go away, but I'm unable to pinpoint what exactly triggers this inconsistency.

    bug 
    opened by Palkovsky 8
  • [question] install vectorscan package

    [question] install vectorscan package

    @markos do we support install package through apt-get or we have to build the package on local? for hyperscan we run apt-get install -y libhyperscan5 command. do we have something similar for vectorscan? thanks.

    opened by yangzhang21 8
  • Fix all ASAN issues in vectorscan

    Fix all ASAN issues in vectorscan

    Closes https://github.com/VectorCamp/vectorscan/issues/91 Closes https://github.com/VectorCamp/vectorscan/issues/100

    There were lots of problems in small ranges, I tried to fix them all by carefully reading the code.

    Also adopt all changes we made to make it compile like setbit unsetting in tests.

    We tested with all combinations HS_OPTIMIZE/DEBUG/HASWELL/SSE4.2/ARM/ASAN/MSAN/NOSANITIZERS. Haven't tested AVX512, PPC and SVE2

    Resubmitting to a develop branch. I'll take a look at AVX512 failure from FAT_RUNTIME

    opened by danlark1 7
  • build failure on x86

    build failure on x86

    Build fails for x86 when building on Alpine Linux, log here: https://gitlab.alpinelinux.org/a16bitsysop/aports/-/jobs/278030/raw

    407/builds/a16bitsysop/aports/community/hyperscan/src/hyperscan-5.3.1/src/util/arch/x86/simd_utils.h: In function 'movq':
    408/builds/a16bitsysop/aports/community/hyperscan/src/hyperscan-5.3.1/src/util/arch/x86/simd_utils.h:128:12: error: implicit declaration of function '_mm_cvtsi128_si64'; did you mean '_mm_cvtsi128_si32'? [-Werror=implicit-function-declaration]
    409  128 |     return _mm_cvtsi128_si64(in);
    410      |            ^~~~~~~~~~~~~~~~~
    411      |            _mm_cvtsi128_si32
    
    opened by a16bitsysop 7
  • ARM Intrinsics don't build on clang

    ARM Intrinsics don't build on clang

    I'm trying to build vectorscan on an ARM machine using clang (arm64, macOS, M1 chip). Clang compilation works on x86_64, but fails on ARM. The build fails while compiling the NEON intrinsics:

    Version (clang -v):

    Apple clang version 12.0.5 (clang-1205.0.22.9)
    Target: arm64-apple-darwin20.4.0
    Thread model: posix
    InstalledDir: /Library/Developer/CommandLineTools/usr/bin
    

    Compiler output:

    Consolidate compiler generated dependencies of target hs_exec
    [  1%] Building C object CMakeFiles/hs_exec.dir/src/runtime.c.o
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/runtime.c:45:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/nfa_rev_api.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/vermicelli.h:36:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/bitutils.h:51:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/bitutils.h:41:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/common/bitutils.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/simd_utils.h:67:
    /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/simd_utils.h:98:19: error: argument to '__builtin_neon_vshlq_n_v' must be a constant integer
        return (m128) vshlq_n_s32((int64x2_t)a, b);
                      ^                         ~
    /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include/arm_neon.h:24760:23: note: expanded from macro 'vshlq_n_s32'
      __ret = (int32x4_t) __builtin_neon_vshlq_n_v((int8x16_t)__s0, __p1, 34); \
                          ^                                         ~~~~
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/runtime.c:45:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/nfa_rev_api.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/vermicelli.h:36:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/bitutils.h:51:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/bitutils.h:41:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/common/bitutils.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/simd_utils.h:67:
    /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/simd_utils.h:103:19: error: argument to '__builtin_neon_vshrq_n_v' must be a constant integer
        return (m128) vshrq_n_s32((int64x2_t)a, b);
                      ^                         ~
    /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include/arm_neon.h:25168:23: note: expanded from macro 'vshrq_n_s32'
      __ret = (int32x4_t) __builtin_neon_vshrq_n_v((int8x16_t)__s0, __p1, 34); \
                          ^                                         ~~~~
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/runtime.c:45:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/nfa_rev_api.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/vermicelli.h:36:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/bitutils.h:51:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/bitutils.h:41:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/common/bitutils.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/simd_utils.h:67:
    /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/simd_utils.h:108:19: error: argument to '__builtin_neon_vshlq_n_v' must be a constant integer
        return (m128) vshlq_n_s64((int64x2_t)a, b);
                      ^                         ~
    /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include/arm_neon.h:24778:23: note: expanded from macro 'vshlq_n_s64'
      __ret = (int64x2_t) __builtin_neon_vshlq_n_v((int8x16_t)__s0, __p1, 35); \
                          ^                                         ~~~~
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/runtime.c:45:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/nfa_rev_api.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/vermicelli.h:36:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/bitutils.h:51:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/bitutils.h:41:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/common/bitutils.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/simd_utils.h:67:
    /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/simd_utils.h:113:19: error: argument to '__builtin_neon_vshrq_n_v' must be a constant integer
        return (m128) vshrq_n_s64((int64x2_t)a, b);
                      ^                         ~
    /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include/arm_neon.h:25186:23: note: expanded from macro 'vshrq_n_s64'
      __ret = (int64x2_t) __builtin_neon_vshrq_n_v((int8x16_t)__s0, __p1, 35); \
                          ^                                         ~~~~
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/runtime.c:45:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/nfa_rev_api.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/vermicelli.h:36:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/bitutils.h:51:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/bitutils.h:41:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/common/bitutils.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/simd_utils.h:67:
    /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/simd_utils.h:166:12: error: argument to '__builtin_neon_vgetq_lane_i32' must be a constant integer
        return vgetq_lane_u32((uint32x4_t) in, imm);
               ^                               ~~~
    /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include/arm_neon.h:7511:22: note: expanded from macro 'vgetq_lane_u32'
      __ret = (uint32_t) __builtin_neon_vgetq_lane_i32((int32x4_t)__s0, __p1); \
                         ^                                              ~~~~
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/runtime.c:45:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/nfa_rev_api.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/vermicelli.h:36:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/bitutils.h:51:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/bitutils.h:41:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/common/bitutils.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/simd_utils.h:67:
    /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/simd_utils.h:190:12: error: argument to '__builtin_neon_vgetq_lane_i64' must be a constant integer
        return vgetq_lane_u64((uint64x2_t) in, imm);
               ^                               ~~~
    /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include/arm_neon.h:7534:22: note: expanded from macro 'vgetq_lane_u64'
      __ret = (uint64_t) __builtin_neon_vgetq_lane_i64((int64x2_t)__s0, __p1); \
                         ^                                              ~~~~
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/runtime.c:45:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/nfa_rev_api.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/nfa/vermicelli.h:36:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/bitutils.h:51:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/bitutils.h:41:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/common/bitutils.h:38:
    In file included from /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/simd_utils.h:67:
    /Users/mg/projects/hyperscan-java-native/cppbuild/hyperscan-5.4.0/src/util/arch/arm/simd_utils.h:304:18: error: argument to '__builtin_neon_vextq_v' must be a constant integer
        return (m128)vextq_s8((int8x16_t)l, (int8x16_t)r, offset);
                     ^                                    ~~~~~~
    /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include/arm_neon.h:6896:23: note: expanded from macro 'vextq_s8'
      __ret = (int8x16_t) __builtin_neon_vextq_v((int8x16_t)__s0, (int8x16_t)__s1, __p2, 32); \
                          ^                                                        ~~~~
    7 errors generated.
    make[2]: *** [CMakeFiles/hs_exec.dir/src/runtime.c.o] Error 1
    make[1]: *** [CMakeFiles/hs_exec.dir/all] Error 2
    make: *** [all] Error 2
    

    It seems like clang doesn't understand how to inline parameters it expects to be constant.

    Maybe some of the abstraction functions for the intrinsics need to be converted to macros?

    opened by gliwka 5
  • Build fails with glibc >= 2.34

    Build fails with glibc >= 2.34

    Build fails with glibc >= 2.34 with:

    [ 6426s] [ 91%] Building CXX object tools/hscollider/CMakeFiles/hscollider.dir/sig.cpp.o
    [ 6426s] cd /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/build/tools/hscollider && /usr/bin/c++  -I/home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/tools/hscollider -I/home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/build/tools/hscollider -I/home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/tools/hscollider -I/home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/src -I/home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/build -I/home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7 -I/home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/tools/src -I/home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/util -isystem /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/include -mbranch-protection=standard -O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type  -g -march=armv8-a+crypto+crc -mtune=native  -O3 -std=c++17 -Wall -Wextra -Wshadow -Wswitch -Wreturn-type -Wcast-qual -Wno-deprecated -Wnon-virtual-dtor -fno-strict-aliasing -fno-new-ttp-matching -DNDEBUG -Wno-maybe-uninitialized -Wno-abi -fno-omit-frame-pointer -fvisibility=hidden -Wvla -Wpointer-arith -Wno-unused-const-variable -Wno-ignored-attributes -Wno-redundant-move -Wmissing-declarations  -g -DNDEBUG -MD -MT tools/hscollider/CMakeFiles/hscollider.dir/sig.cpp.o -MF CMakeFiles/hscollider.dir/sig.cpp.o.d -o CMakeFiles/hscollider.dir/sig.cpp.o -c /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/tools/hscollider/sig.cpp
    [ 6428s] In file included from /usr/include/signal.h:328,
    [ 6428s]                  from /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/tools/hscollider/sig.cpp:40:
    [ 6428s] /home/abuild/rpmbuild/BUILD/vectorscan-vectorscan-5.4.7/tools/hscollider/sig.cpp:169:40: error: size of array 'alt_stack_loc' is not an integral constant-expression
    [ 6428s]   169 | static TLS_VARIABLE char alt_stack_loc[SIGSTKSZ];
    [ 6428s]       |                                        ^~~~~~~~
    

    This was found on openSUSE Tumbleweed aarch64.

    opened by ggardet 1
  • Building fails on macos/x64 with 'mktemp: illegal option -- p'

    Building fails on macos/x64 with 'mktemp: illegal option -- p'

    Building 73695e419c27af7fe2a099c7aa57931cc02aea5d on x64 macos 12.4 (Monterey) fails with the default build instructions, e.g.

    $ cmake ../vectorscan
    CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
      Compatibility with CMake < 2.8.12 will be removed from a future version of
      CMake.
    
      Update the VERSION argument <min> value or use a ...<max> suffix to tell
      CMake that the project does not need compatibility with older versions.
    
    -- The C compiler identification is AppleClang 13.1.6.13160021
    -- The CXX compiler identification is AppleClang 13.1.6.13160021
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Performing Test ARCH_X86_64
    -- Performing Test ARCH_X86_64 - Success
    -- Performing Test ARCH_IA32
    -- Performing Test ARCH_IA32 - Failed
    -- Performing Test ARCH_AARCH64
    -- Performing Test ARCH_AARCH64 - Failed
    -- Performing Test ARCH_ARM32
    -- Performing Test ARCH_ARM32 - Failed
    -- Performing Test ARCH_PPC64EL
    -- Performing Test ARCH_PPC64EL - Failed
    -- Default build type 'Release with debug info'
    -- using release build
    -- Boost version: 1.78.0
    -- Found Python: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8 (found version "3.8.9") found components: Interpreter 
    -- Build date: 2022-06-19
    -- Building static libraries
    -- clang will tune for native, generic
    -- ARCH_C_FLAGS   : -msse4.2
    -- ARCH_CXX_FLAGS : -msse4.2
    -- Looking for include file unistd.h
    -- Looking for include file unistd.h - found
    -- Looking for include file intrin.h
    -- Looking for include file intrin.h - not found
    -- Looking for C++ include intrin.h
    -- Looking for C++ include intrin.h - not found
    -- Looking for include file x86intrin.h
    -- Looking for include file x86intrin.h - found
    -- Looking for C++ include x86intrin.h
    -- Looking for C++ include x86intrin.h - found
    -- Looking for posix_memalign
    -- Looking for posix_memalign - found
    -- Looking for _aligned_malloc
    -- Looking for _aligned_malloc - not found
    -- Performing Test HAS_C_HIDDEN
    -- Performing Test HAS_C_HIDDEN - Success
    -- Performing Test HAS_CXX_HIDDEN
    -- Performing Test HAS_CXX_HIDDEN - Success
    -- Looking for _LIBCPP_VERSION
    -- Looking for _LIBCPP_VERSION - found
    -- Performing Test HAVE_SSE42
    -- Performing Test HAVE_SSE42 - Success
    -- Performing Test HAVE_AVX2
    -- Performing Test HAVE_AVX2 - Success
    -- Performing Test HAVE_AVX512
    -- Performing Test HAVE_AVX512 - Failed
    -- Performing Test HAVE_AVX512VBMI
    -- Performing Test HAVE_AVX512VBMI - Failed
    -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED
    -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED - Success
    -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED
    -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED - Success
    -- Performing Test C_FLAG_Wvla
    -- Performing Test C_FLAG_Wvla - Success
    -- Performing Test C_FLAG_Wpointer_arith
    -- Performing Test C_FLAG_Wpointer_arith - Success
    -- Performing Test C_FLAG_Wstrict_prototypes
    -- Performing Test C_FLAG_Wstrict_prototypes - Success
    -- Performing Test C_FLAG_Wmissing_prototypes
    -- Performing Test C_FLAG_Wmissing_prototypes - Success
    -- Performing Test CXX_FLAG_Wvla
    -- Performing Test CXX_FLAG_Wvla - Success
    -- Performing Test CXX_FLAG_Wpointer_arith
    -- Performing Test CXX_FLAG_Wpointer_arith - Success
    -- Performing Test CC_SELF_ASSIGN
    -- Performing Test CC_SELF_ASSIGN - Success
    -- Performing Test CXX_SELF_ASSIGN
    -- Performing Test CXX_SELF_ASSIGN - Success
    -- Performing Test CC_PAREN_EQUALITY
    -- Performing Test CC_PAREN_EQUALITY - Success
    -- Performing Test CXX_UNUSED_CONST_VAR
    -- Performing Test CXX_UNUSED_CONST_VAR - Success
    -- Performing Test CXX_IGNORED_ATTR
    -- Performing Test CXX_IGNORED_ATTR - Success
    -- Performing Test CXX_REDUNDANT_MOVE
    -- Performing Test CXX_REDUNDANT_MOVE - Success
    -- Performing Test CXX_WEAK_VTABLES
    -- Performing Test CXX_WEAK_VTABLES - Success
    -- Performing Test CXX_MISSING_DECLARATIONS
    -- Performing Test CXX_MISSING_DECLARATIONS - Success
    -- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS
    -- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS - Success
    -- Performing Test CXX_WUNUSED_VARIABLE
    -- Performing Test CXX_WUNUSED_VARIABLE - Success
    -- Performing Test CC_STRINGOP_OVERFLOW
    -- Performing Test CC_STRINGOP_OVERFLOW - Failed
    -- Building runtime for multiple microarchitectures
    -- Looking for mmap
    -- Looking for mmap - found
    -- Doxygen not found, unable to generate API reference
    -- Sphinx not found, unable to generate developer reference
    -- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2") 
    -- Checking for module 'libpcre>=8.41'
    --   Found libpcre, version 8.45
    -- PCRE version 8.41 or above
    -- Looking for pthread.h
    -- Looking for pthread.h - found
    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
    -- Found Threads: TRUE  
    -- Checking for module 'sqlite3'
    --   Found sqlite3, version 3.37.0
    -- Performing Test SQLITE_VERSION_OK
    -- Performing Test SQLITE_VERSION_OK - Success
    -- Looking for sqlite3_open_v2
    -- Looking for sqlite3_open_v2 - found
    -- Looking for C++ include pthread_np.h
    -- Looking for C++ include pthread_np.h - not found
    -- Looking for pthread_setaffinity_np
    -- Looking for pthread_setaffinity_np - not found
    -- Looking for malloc_info
    -- Looking for malloc_info - not found
    -- Looking for shmget
    -- Looking for shmget - found
    -- Performing Test BACKTRACE_LIBC
    -- Performing Test BACKTRACE_LIBC - Success
    -- Performing Test HAS_RDYNAMIC
    -- Performing Test HAS_RDYNAMIC - Success
    -- Looking for sigaltstack
    -- Looking for sigaltstack - found
    -- Looking for sigaction
    -- Looking for sigaction - found
    -- Looking for setrlimit
    -- Looking for setrlimit - found
    -- Configuring done
    CMake Warning (dev) at CMakeLists.txt:1255 (add_library):
      Policy CMP0115 is not set: Source file extensions must be explicit.  Run
      "cmake --help-policy CMP0115" for policy details.  Use the cmake_policy
      command to set the policy and suppress this warning.
    
      File:
    
        /Users/<user>/<path>/vectorscan/src/hs_version.h.in
    This warning is for project developers.  Use -Wno-dev to suppress it.
    
    -- Generating done
    -- Build files have been written to: /Users/<user>/<path>/vectorscan-build
    $ cmake --build .
    [  0%] Building C object CMakeFiles/hs_exec_core2.dir/src/crc32.c.o
    mktemp: illegal option -- p
    usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
           mktemp [-d] [-q] [-u] -t prefix 
    make[2]: *** [CMakeFiles/hs_exec_core2.dir/src/crc32.c.o] Error 1
    make[1]: *** [CMakeFiles/hs_exec_core2.dir/all] Error 2
    make: *** [all] Error 2
    $ 
    

    macos's mktemp has the following options:

    OPTIONS
         The available options are as follows:
    
         -d      Make a directory instead of a file.
    
         -q      Fail silently if an error occurs.  This is useful if a script does
                 not want error output to go to standard error.
    
         -t prefix
                 Generate a template (using the supplied prefix and TMPDIR if set)
                 to create a filename template.
    
         -u      Operate in “unsafe” mode.  The temp file will be unlinked before
                 mktemp exits.  This is slightly better than mktemp(3) but still
                 introduces a race condition.  Use of this option is not encouraged.
    
    opened by rmharris 2
  • 'Illegal instruction' error on x86_64 CentOS 7.9

    'Illegal instruction' error on x86_64 CentOS 7.9

    I got a error 'Illegal instruction' to run unit test and example on x86_64 CentOS 7.9. It's appreciated to get help from the community.

    build

    wget https://github.com/VectorCamp/vectorscan/archive/refs/tags/vectorscan/5.4.7.zip unzip 5.4.7.zip cd vectorscan-vectorscan-5.4.7 mkdir build cd build PATH=$RAGEL_PATH:$PATH cmake -DBOOST_ROOT=$BOOST_SOURCE .. make

    run

    ./bin/unit-hyperscan Illegal instruction

    ./bin/simplegrep cpp Makefile Illegal instruction

    Env

    • CentOS Linux release 7.9.2009 (Core)
    • Linux hostname 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
    • gcc version 11.1.0

    Using built-in specs. COLLECT_GCC=/home/xiaokang/opt/ldb/bin/gcc-11 COLLECT_LTO_WRAPPER=/mnt/disk3/xiaokang/opt/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/lto-wrapper OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa OFFLOAD_TARGET_DEFAULT=1 Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.1.0-1ubuntu1~18.04.1' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --disable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-YRKbe7/gcc-11-11.1.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-YRKbe7/gcc-11-11.1.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix Supported LTO compression algorithms: zlib zstd gcc version 11.1.0 (Ubuntu 11.1.0-1ubuntu1~18.04.1)

    • cat /proc/cpuinfo

    processor : 0 vendor_id : AuthenticAMD cpu family : 23 model : 49 model name : AMD EPYC 7K62 48-Core Processor stepping : 0 microcode : 0x1000065 cpu MHz : 2595.124 cache size : 512 KB physical id : 0 siblings : 48 core id : 0 cpu cores : 24 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm art rep_good nopl extd_apicid amd_dcm eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext retpoline_amd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat bogomips : 5190.24 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management:

    opened by xiaokang 9
  • FAT_RUNTIME, shared library only builds, and various other distro related requirements.

    FAT_RUNTIME, shared library only builds, and various other distro related requirements.

    I've been looking at packaging this project on fedora/rhel/epel, and it seems many of the x86 options in place for distro's (hyperscan is part of fedora/etc) don't work on aarch64/ppc. Beyond what others have noted about the signed/unsigned char/colm issues:

    For starters the expectation is that something like the FAT_RUNTIME option in the makefile generates binaries with a baseline (say armv-8a, without any optional arch extensions) that runs regardless of the HW, and at runtime selects neon/SVE/SVE2 paths if the HW supports it. Since most distro's also build nativly, the CMake file needs to be aware of this and honor the baseline compiler options (which tend to also include things like -mbranch_protection/etc) only extending them as needed rather than trying to do -march/mtune=native options because the build machine won't be related to the install machine.

    Also, most distro's frown on static libraries, but the unit test won't compile without them (-DBUILD_SHARED_LIB:BOOL=ON). There is also a problem on aarch64 platforms where the exception stack size is a TLS variable but won't be fixed at buildtime (this is generic to hyperscan as well AFAIK). There are various other lower priority issues, like the documentation isn't installed, the project tosses a LOT of warnings building on aarch64, the c++17 requirement means it won't build with the system compiler on older distros, etc.

    Basically, I've spent a couple days trying to create a generic package on aarch64 and it looks like it needs more work beyond the hacky patches I've applied to force it to build in a single case.

    opened by jlinton 4
  • vectorscan/5.4.7 fails to enable SVE2 with gcc 10.3.0

    vectorscan/5.4.7 fails to enable SVE2 with gcc 10.3.0

    The main issue is that CMakeLists.txt is appending flags for march that are already in the flag list (namely +sve2 and +sve2-bitperm). The second issue is the attempt to set mtune to be the same as march which does not work for aarch64.

    git status

    HEAD detached at vectorscan/5.4.7 nothing to commit, working tree clean

    /usr/bin/cmake --prefix=/usr/local -DBUILD_SVE2=1 -DBUILD_SVE2_BITPERM=1 -DCMAKE_INSTALL_PREFIX=/usr/local -DFAT_RUNTIME=off -DBOOST_ROOT=/bmark/snort/0/source/boost/ -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="-mcpu=native" -DCMAKE_CXX_FLAGS="-mcpu=native" /bmark/snort/0/source/hyperscan

    -- The C compiler identification is GNU 10.3.0 -- The CXX compiler identification is GNU 10.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Performing Test ARCH_X86_64 -- Performing Test ARCH_X86_64 - Failed -- Performing Test ARCH_IA32 -- Performing Test ARCH_IA32 - Failed -- Performing Test ARCH_AARCH64 -- Performing Test ARCH_AARCH64 - Success -- Performing Test ARCH_ARM32 -- Performing Test ARCH_ARM32 - Failed -- Performing Test ARCH_PPC64EL -- Performing Test ARCH_PPC64EL - Failed -- Build type RELEASE -- using release build -- Boost version: 1.77.0 -- Found PythonInterp: /usr/bin/python (found version "2.7.18") -- Build date: 2022-05-06 -- Building static libraries -- gcc version 10.3.0 CMake Warning at CMakeLists.txt:187 (message): Something went wrong determining gcc tune: -mtune=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres not valid, falling back to -mtune=native

    -- ARCH_C_FLAGS : -- ARCH_CXX_FLAGS : -- g++ version 10.3.0 -- Looking for include file unistd.h -- Looking for include file unistd.h - found -- Looking for C++ include arm_neon.h -- Looking for C++ include arm_neon.h - found -- Looking for C++ include arm_sve.h -- Looking for C++ include arm_sve.h - found -- Looking for posix_memalign -- Looking for posix_memalign - found -- Looking for _aligned_malloc -- Looking for _aligned_malloc - not found -- Performing Test HAS_C_HIDDEN -- Performing Test HAS_C_HIDDEN - Success -- Performing Test HAS_CXX_HIDDEN -- Performing Test HAS_CXX_HIDDEN - Success -- Looking for _LIBCPP_VERSION -- Looking for _LIBCPP_VERSION - not found -- Performing Test HAVE_NEON -- Performing Test HAVE_NEON - Success -- Performing Test HAVE_SVE2_BITPERM -- Performing Test HAVE_SVE2_BITPERM - Failed -- Performing Test HAVE_SVE2 -- Performing Test HAVE_SVE2 - Failed -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED - Success -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED - Success -- Performing Test HAVE__BUILTIN_CONSTANT_P -- Performing Test HAVE__BUILTIN_CONSTANT_P - Success -- Performing Test C_FLAG_Wvla -- Performing Test C_FLAG_Wvla - Success -- Performing Test C_FLAG_Wpointer_arith -- Performing Test C_FLAG_Wpointer_arith - Success -- Performing Test C_FLAG_Wstrict_prototypes -- Performing Test C_FLAG_Wstrict_prototypes - Success -- Performing Test C_FLAG_Wmissing_prototypes -- Performing Test C_FLAG_Wmissing_prototypes - Success -- Performing Test CXX_FLAG_Wvla -- Performing Test CXX_FLAG_Wvla - Success -- Performing Test CXX_FLAG_Wpointer_arith -- Performing Test CXX_FLAG_Wpointer_arith - Success -- Performing Test CC_SELF_ASSIGN -- Performing Test CC_SELF_ASSIGN - Failed -- Performing Test CXX_SELF_ASSIGN -- Performing Test CXX_SELF_ASSIGN - Failed -- Performing Test CC_PAREN_EQUALITY -- Performing Test CC_PAREN_EQUALITY - Failed -- Performing Test CXX_UNUSED_CONST_VAR -- Performing Test CXX_UNUSED_CONST_VAR - Success -- Performing Test CXX_IGNORED_ATTR -- Performing Test CXX_IGNORED_ATTR - Success -- Performing Test CXX_REDUNDANT_MOVE -- Performing Test CXX_REDUNDANT_MOVE - Success -- Performing Test CXX_WEAK_VTABLES -- Performing Test CXX_WEAK_VTABLES - Failed -- Performing Test CXX_MISSING_DECLARATIONS -- Performing Test CXX_MISSING_DECLARATIONS - Success -- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS -- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS - Success -- Performing Test CXX_WUNUSED_VARIABLE -- Performing Test CXX_WUNUSED_VARIABLE - Success -- Performing Test CC_STRINGOP_OVERFLOW -- Performing Test CC_STRINGOP_OVERFLOW - Success -- Building for current host CPU: -march=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres+sve2-bitperm -mtune=native -- Looking for mmap -- Looking for mmap - not found -- Doxygen not found, unable to generate API reference -- Sphinx not found, unable to generate developer reference -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Checking for module 'libpcre>=8.41' -- Found libpcre, version 8.45 -- PCRE version 8.41 or above -- Looking for pthread.h -- Looking for pthread.h - not found -- Could NOT find Threads (missing: Threads_FOUND) -- Checking for module 'sqlite3' -- Found sqlite3, version 3.31.1 -- Performing Test SQLITE_VERSION_OK -- Performing Test SQLITE_VERSION_OK - Failed CMake Error at cmake/sqlite3.cmake:34 (message): sqlite3 is broken from 3.8.7 to 3.8.10 - please find a working version Call Stack (most recent call first): tools/hsbench/CMakeLists.txt:1 (include)

    -- Configuring incomplete, errors occurred! See also "/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeOutput.log". See also "/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeError.log".

    tail -n 20 CMakeFiles/CMakeError.log

    Change Dir: /bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/bin/make cmTC_4ea79/fast && /usr/bin/make -f CMakeFiles/cmTC_4ea79.dir/build.make CMakeFiles/cmTC_4ea79.dir/build make[1]: Entering directory '/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp' Building C object CMakeFiles/cmTC_4ea79.dir/src.c.o /usr/bin/cc -mcpu=native -march=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres+sve2-bitperm -mtune=native -DSQLITE_VERSION_OK -o CMakeFiles/cmTC_4ea79.dir/src.c.o -c /bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp/src.c Assembler messages: Error: must specify extensions to add before specifying those to remove Error: unrecognized option -march=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres+sve2-bitperm make[1]: *** [CMakeFiles/cmTC_4ea79.dir/build.make:66: CMakeFiles/cmTC_4ea79.dir/src.c.o] Error 1 make[1]: Leaving directory '/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp' make: *** [Makefile:121: cmTC_4ea79/fast] Error 2

    Source file was: #include <sqlite3.h> #if SQLITE_VERSION_NUMBER >= 3008007 && SQLITE_VERSION_NUMBER < 3008010 #error broken sqlite #endif int main() {return 0;}

    opened by mddirba 3
Releases(vectorscan/5.4.7)
  • vectorscan/5.4.7(May 5, 2022)

    What's Changed

    • Fix word boundary assertions under C++20 by @BigRedEye in https://github.com/VectorCamp/vectorscan/pull/90
    • Fix all ASAN issues in vectorscan by @danlark1 in https://github.com/VectorCamp/vectorscan/pull/93
    • change FAT_RUNTIME to a normal option so it can be set to off by @a16bitsysop in https://github.com/VectorCamp/vectorscan/pull/94
    • Optimized and correct version of movemask128 for ARM by @danlark1 in https://github.com/VectorCamp/vectorscan/pull/102
    Source code(tar.gz)
    Source code(zip)
  • vectorscan/5.4.6(Jan 21, 2022)

    New release 5.4.6

    Changelog:

    • Major refactoring of many engines to use internal SuperVector C++ templates library. Code size reduced to 1/3rd with no loss of performance in most cases.
    • Microbenchmarking tool added for performance finetuning
    • Arm Advanced SIMD/NEON fully ported. Initial work on SVE2 for a couple of engines.
    • Power9 VSX ppc64le fully ported. Initial port needs some optimization.
    • Clang compiler support added.
    • Apple M1 support added.
    • CI added, the following configurations are tested on every PR: gcc-debug, gcc-release, clang-debug, clang-release: Linux Intel: SSE4.2, AVX2, AVX512, FAT Linux Arm Linux Power9 clang-debug, clang-release: MacOS Apple M1
    Source code(tar.gz)
    Source code(zip)
Owner
VectorCamp
SIMD and Optimizations Specialists
VectorCamp
Love 6's Regular Expression Engine. Support Concat/Select/Closure Basic function. Hope u can enjoy this tiny engine :)

Regex_Engine Love 6's Blog Website: https://love6.blog.csdn.net/ Love 6's Regular Expression Engine Hope u can love my tiny regex engine :) maybe a fe

Love6 2 May 24, 2022
Flexible, portable, high-performance bit fields C++ library. unsigned a:13 becomes F<13> a;

C-plus-plus-library-bit-fields Flexible, portible, high-performance bit fields C++ library. The bit fields are specified with a dummy structure where

Walt Karas 25 Feb 27, 2022
An eventing framework for building high performance and high scalability systems in C.

NOTE: THIS PROJECT HAS BEEN DEPRECATED AND IS NO LONGER ACTIVELY MAINTAINED As of 2019-03-08, this project will no longer be maintained and will be ar

Meta Archive 1.7k Aug 8, 2022
Fork of junaburg's picom fork with a patch for rounded corners and shadows

picom new! : You'll now also find tryone's dual_kawase blur for the new backend, as well as rounded corners from sdhand if they are so desired, merged

Arian Rezazadeh 42 Jul 20, 2022
Functional programming style pattern-matching library for C++

Mach7: Pattern Matching for C++ by Yuriy Solodkyy, Gabriel Dos Reis, Bjarne Stroustrup Abstract Pattern matching is an abstraction mechanism that can

Yuriy Solodkyy 1.2k Jul 31, 2022
Parsing Expression Grammar Template Library

Welcome to the PEGTL The Parsing Expression Grammar Template Library (PEGTL) is a zero-dependency C++ header-only parser combinator library for creati

The Art of C++ 1.6k Aug 8, 2022
The Pizza Compass will determine your location and direct you to the nearest pizza place. It’s like a regular compass, but better!

Pizza_Compass A Particle project named Pizza_Compass Welcome to your project! Every new Particle project is composed of 3 important elements that you'

Joe Grand 66 Jul 25, 2022
Simple header only pattern matching for c++14

Simple, Extensible C++ Pattern Matching Library I have recently been looking at Haskell and Rust. One of the things I wanted in C++ from those languag

John Bandela 205 Aug 8, 2022
Pan-Genomic Matching Statistics

SPUMONI Pan-genomic Matching Statistics for Targeted Nanopore Sequencing Based on MONI: A MEM-finder with Multi-Genome References. MONI index uses the

Omar Ahmed 23 Jul 26, 2022
fuzzy matching selection gui

fm === fm provides a gui to select an item from a list using a fuzzy matching algorithm. When an item is selected, it is sent to the plumber `send` po

phil9 8 Nov 24, 2021
Distance matching plugin

Distance Matching This plug-in is custom implementation of the Distance Matching technique which was shown by Laurent Delayen at Nucl.ai 2016. In two

Roman Merkushin 46 Jun 25, 2022
An in-progress matching decompilation of Final Fantasy VII For the PSX.

FFVII An in-progress decompilation of the original US release of Final Fantasy VII on the PSX. Building (Linux) Install build dependencies The build p

null 15 Jun 5, 2022
K-Closest Points and Maximum Clique Pruning for Efficient and Effective 3-D Laser Scan Matching (RA-L 2022)

KCP The official implementation of KCP: K-Closest Points and Maximum Clique Pruning for Efficient and Effective 3D Laser Scan Matching, accepted for p

Yu-Kai Lin 97 Aug 1, 2022
Node.js bindings for the Mathematical Expression Toolkit

ExprTk.js This is the Node.js bindings for ExprTk (Github) by @ArashPartow ExprTk.js supports both synchronous and asynchronous background execution o

Momtchil Momtchev 6 Jul 23, 2022
A C++ expression -> x64 JIT

NativeJIT NativeJIT is an open-source cross-platform library for high-performance just-in-time compilation of expressions involving C data structures.

null 1.1k Aug 6, 2022
A C++11 large integer library with effective high performance, simplistic in nature and also clean in the eyes.

BigIntegerCPP BigIntegerCPP is a C++11 port of large integer library used in CryptoLib4Pascal. It allows mostly parsing of numbers as strings in diffe

Telepati 25 Apr 22, 2022
Diff Match Patch is a high-performance library in multiple languages that manipulates plain text.

The Diff Match and Patch libraries offer robust algorithms to perform the operations required for synchronizing plain text. Diff: Compare two blocks o

Google 5.5k Aug 8, 2022
Visualization Library is a C++ middleware for high-performance 2D and 3D graphics applications based on OpenGL 1.x-4.x supporting Windows, Linux and Mac OS X.

Visualization Library 2.2 Gallery About Visualization Library is a C++ middleware for high-performance 2D and 3D graphics applications based on the in

Michele 300 Aug 9, 2022