Unfortunately the CMake is a bit hacky so the fixes are also a bit hacky. In my opinion ExternalProject_Add should eventually be replaced with find_package (in combination with vcpkg), but LIEF/tlsh (and older versions of capstone) don't have proper packages so this is rather tricky to do for now. Another issue is that there is no support for multi-config generators (like Visual Studio or Ninja) so you might end up with errors trying to build anything but Release.

I forked tlsh to https://github.com/mrexodia/tlsh with some changes to the CMakeLists.txt to make it work on Windows. The repo seems rather dead so I don't think trying to get this upstream will be successful. Feel free to fork it to your account.

Closes #126 Closes #38 Closes #128

Hello! I wanted to process an OpenSSL library and noticed that the latest version of binlex recognized only a negligible number of functions - 7 meanwhile IDA recognized 1636. I used command binlex -m pe:x86_64 -i <lib_name> | jq -r 'select(.type == ("function"))', am I doing something wrong or is there a bug please?

Also to build, flags needed to be set:

CFLAGS='-std=c11' CXXFLAGS='-std=c++11' make THREADS=4

Built on M1:

binlex % file ./build/binlex 
./build/binlex: Mach-O 64-bit executable arm64

binlex % ./build/binlex
binlex v1.1.1 - A Binary Genetic Traits Lexer
  -i  --input		input file		(required)
  -m  --mode		set mode		(optional)
  -lm --list-modes	list modes		(optional)
  -c  --corpus		corpus name		(optional)
  -g  --tag		add a tag		(optional)
           		(can be specified multiple times)
  -t  --threads		number of threads	(optional)
  -to --timeout		execution timeout in s	(optional)
  -h  --help		display help		(optional)
  -o  --output		output file		(optional)
  -p  --pretty		pretty output		(optional)
  -d  --debug		print debug info	(optional)
  -v  --version		display version		(optional)

The function ClearTrait overwrites trait->bytes_sha256 with NULL. The memory which may have been allocated is not freed resulting in a memory leak. The same is true for trait->trait.

Dear @c3rb3ru5d3d53c,

this PR adds the necessary packaging sources for building Debian binary packages of the programs/libs

• libbinlex
• binlex
• blyara

in order to ease installation and package removal.

As it is described in the updated README.md-file, users can simply run make deb to create those .deb-files, which uses dpkg-buildpackage under the hood.

Please note, that for blyara, an unnecessary dependency on libcapstone has been removed. Packaging the development-files as libbinlex-dev has not been done yet.

Thanks already in advance for considering this PR.

Best regards, jgru

Description:

When using the Python binding and the Raw disassembler then randomly segment violations occur.

To Reproduce:

Using the following Python script "blfile.py":

#! /usr/bin/env python3

import sys
import pybinlex

def main():
    for fnam in sys.argv[1:]:
        raw = pybinlex.Raw()
        #raw = pybinlex.ELF()
        raw.set_architecture(pybinlex.BINARY_ARCH.BINARY_ARCH_X86, pybinlex.BINARY_MODE.BINARY_MODE_64)
        raw.read_file(fnam)
        disasm = pybinlex.Decompiler(raw)
        disasm.decompile()
        print("# file %s:\n\t%s\n" % (fnam, disasm.get_traits()))

if __name__ == "__main__":
    main()

Then using the command line on binary files segfaults, e.g. python3 blfile.py /bin/a*.

Expected Behavior:

It should produce some traits, replacing the Raw object by an ELF object works as intended.

Affected OS/Version:

Debian Bullseye

In order to work with frequency analysis on traits, we would need to track the file hashes associated with given traits.

To do this we would need the equivalent of a stored procedure in mongodb when documents are posted to keep records of hashes for traits.

This would make the db a little more complex, but it the pay off would be pretty great, as we would be able to search traits by sample hash and more.

Hello, nice project!. I'm wondering whether it wouldn't be useful to make the output more granular - additionally generate traits/bytes for individual blocks/instructions/opcodes/operands. I believe that it would be useful for subsequent processing in some cases, e.g. the first team in the Microsoft Malware Classification Challenge used frequency of opcodes and their N-grams among other things.

This GitHub Actions script is easily extensible to include other operating systems and/or compilers. But it might be a bit difficult to get the tests running on all platforms properly.

Related: #133

When we pass a file object, we set the file's BINARY_ARCH and BINARY_MODE, which are abstractions of cs_arch and cs_mode. This will make the API much simpler to use as we only need to pass the file object in the constructor and not call Decompiler.Setup method.

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like TLSH Bytes and Traits

Describe alternatives you've considered N/A

Additional context

Looks like parsing our traits and bytes based on binary data vs. string of the traits or byte sequences will yield best results.

#!/usr/bin/env python

import tlsh
from hexdump import hexdump

str_0 = b'15 d9 e0 d3 e3 23 f8 74 ba 97 6d f0 ?? ?? ?? ?? 04 1b 12 85 8d 31 4e 6f 22 77 2b 3b 5c 16 07 8d 41 55 3e ea eb ec 66 9d 5f 60 3e cd fa a0 40 c3 01 d4 df 11 e6 1c 7d 15 bb db 46 9e e7 a4 c0 5a 8d 45 20 40 8d d7 14 4e 9b 86 7f a3 6e df c5 e4 76 ea 30 4a 78 ff fd 9e 2a a3 06 6f 7d fb 16 80 8e 4a 1a e8 e4 45 49 9c 39 0b 00 d0 31 d0 0e 07 62 f0 9c 0a cd e5 ba f7 72 ed c8 a2 9c 6c 36 ab'
str_1 = b'15 e9 e0 d3 e3 23 f8 74 ba 97 6d f4 ?? ?? ?? ?? 04 1b 12 85 8d 31 4e 6f 22 77 2b 3b 5c 16 07 8d 9b 55 3e ea eb ec 66 9d 5f 60 3e cd fa a0 40 c3 01 d4 df 11 e6 1c 7d 15 bb db 46 9e e7 a4 c0 5a 8d 45 20 40 8d d7 14 4e 9b 86 7f a3 6e df c5 e4 76 ea 30 4a 78 ff fd 9e 2a a3 06 6f 7d fb 16 80 8e 4a 1a e8 e4 45 49 9c 39 0b 00 d0 31 d0 0e 07 62 f0 9c 0a cd e5 ba f7 72 ed c8 a2 9c 6c 36 ac'

bytes_0 = bytes(bytearray.fromhex(str_0.decode('utf-8').replace(' ', '').replace('?', '')))
bytes_1 = bytes(bytearray.fromhex(str_1.decode('utf-8').replace(' ', '').replace('?', '')))

str_h0 = tlsh.Tlsh()
str_h0.update(str_0)
str_h0.final()

bytes_h0 = tlsh.Tlsh()
bytes_h0.update(bytes_0)
bytes_h0.final()

str_h1 = tlsh.Tlsh()
str_h1.update(str_1)
str_h1.final()

bytes_h1 = tlsh.Tlsh()
bytes_h1.update(bytes_1)
bytes_h1.final()

print('str_h0 : ' + str_0.decode('utf-8'))
print('str_tlsh_h0: ' + str_h0.hexdigest())

print('str_h1: ' + str_1.decode('utf-8'))
print('str_tlsh_h1: ' + str_h1.hexdigest())

print('string_score: ' + str(str_h0.diff(str_h1)))

print('bytes_h0: ' + str_0.decode('utf-8'))
hexdump(bytes_0)
print('bytes_h0: ' + bytes_h0.hexdigest())

print('bytes_h0: ' + str_1.decode('utf-8'))
hexdump(bytes_1)
print('bytes_tlsh_h1: ' + bytes_h1.hexdigest())

print('bytes_score: ' + str(bytes_h0.diff(bytes_h1)))

Description: When submitting a sample in "auto" mode, neither binlex.py nor bldec defines the "mode". bldec then raises an error when parsing the architecture.

To Reproduce: curl -X POST --insecure -H "X-API-Key: " --upload-file file.exe https://127.0.0.1:8443/binlex/api/v1/samples/goodware/auto {'corpus': 'goodware', 'mode': 'auto', 'object_name': 'sha256'}

data = {'corpus': 'goodware', 'mode': 'auto', 'object_name': 'sha256'}
file_type = data['mode'].split(':')[0]      # auto
architecture = data['mode'].split(':')[1]   # IndexError: list index out of range

Expected Behavior:

• binlex.py should determine the mode or
• bldec should handle {'mode': 'auto'}

Affected OS/Version: OS: Ubuntu 20.04.4 LTS version: 1.1.1 branch Master

Additional context: master branch (not release)

Is your feature request related to a problem? Please describe. As mentioned in #127, we should move to using findpackage instead as ExternalProject_Add introduces issues with Windows builds.

Describe the solution you'd like The solution would be to add git submodules for LIEF, tlsh, and capstone and use findpackage instead.

Describe alternatives you've considered N/A

Additional context This was suggested by @mrexodia :smile:

Description: Strange error when processing specific obfuscated .NET binary.

To Reproduce: Download pe.cil.2.zip

Run:

binlex -m auto -i pe.cil.2
Try to read 0x4 bytes from 0x153e00 (153e04) which is bigger than the binary's size
Try to read 0x4 bytes from 0x153e00 (153e04) which is bigger than the binary's size
Try to read 0x4 bytes from 0x153e00 (153e04) which is bigger than the binary's size

Expected Behavior: Output traits

Affected OS/Version: Linux/v1.1.1-rc1