CITL's static analysis engine for native code artifacts

Related tags

citl-static-analysis
Overview

citl-static-analyzer

Fast binary hardening analysis tooling.

Building on Linux

The build process varies by Linux distribution, owing to differences between package names and default environment configuration. Luckily, the build process follows the same template on all supported Linux distributions:

  1. Identify which Linux distribution you're using.
  2. Install dependencies.
  3. Decide between performing a Release or Debug build.
  4. Depending on your Linux distribution, determine any additional flags which will later be passed to cmake (examples provided in the subsections below).
  5. Follow the build instructions at the end of this section.
  6. Depending on your Linux distribution, generate a package (examples provided in the subsections below).

Arch Linux

The build dependencies can be installed with:

pacman -S git cmake clang openssl python

When performing the build, pass these additional flags to cmake:

-DCMAKE_CXX_COMPILER=`which clang++` -DCMAKE_C_COMPILER=`which clang`

Ubuntu 14.04 LTS

The build dependencies can be installed with:

sudo apt-get install git cmake3 clang-3.9 libc++-dev libc++abi-dev libssl-dev

When performing the build, pass these additional flags to cmake:

-DCMAKE_CXX_FLAGS="-Doffsetof=__builtin_offsetof" -DCMAKE_CXX_COMPILER=`which clang++-3.9` -DCMAKE_C_COMPILER=`which clang-3.9`

To create a .deb package, perform the build using the instructions below. Then, within the build directory (i.e. build/Release), execute

cpack -G DEB

Ubuntu 16.04 LTS

The build dependencies can be installed with:

sudo apt-get install git cmake clang-3.9 libc++-dev libc++abi-dev libssl-dev

When performing the build, pass these additional flags to cmake:

-DCMAKE_CXX_FLAGS="-Doffsetof=__builtin_offsetof" -DCMAKE_CXX_COMPILER=`which clang++-3.9` -DCMAKE_C_COMPILER=`which clang-3.9`

To create a .deb package, perform the build using the instructions below. Then, within the build directory (i.e. build/Release), execute

cpack -G DEB

Ubuntu 18.04 LTS

The build dependencies for 18.04 are pretty similar to 14.04 LTS. The cmake package is now cmake v3.x and should be installed as simple cmake.

sudo apt-get install git cmake clang-3.9 libc++-dev libc++abi-dev libssl-dev

When performing the build, pass these additional flags to cmake:

-DCMAKE_CXX_COMPILER=`which clang++-3.9` -DCMAKE_C_COMPILER=`which clang-3.9`

To create a .deb package, perform the build using the instructions below. Then, within the build directory (i.e. build/Release), execute

cpack -G DEB

Centos 7.3

The build dependencies can be installed with:

sudo yum groupinstall 'Development Tools'
sudo yum install git cmake3 clang openssl-dev

When performing the build, use the cmake3 command instead of the cmake command, and pass these additional flags to cmake3:

-DCMAKE_CXX_FLAGS="-Doffsetof=__builtin_offsetof" -DCMAKE_CXX_COMPILER=`which clang++` -DCMAKE_C_COMPILER=`which clang`

To create a .rpm package, perform the build using the instructions below. Then, within the build directory (i.e. build/Release), execute

cpack3 -G RPM

NixOS

The build dependencies can be loaded using nix-shell. Create a file called default.nix and populate it with the following:

with import  {};
libcxxStdenv.mkDerivation rec {
    name = "env";
    env = buildEnv { name = name; paths = buildInputs; };
    buildInputs = [
    git
    cmake
    gdb
    openssl
    gnumake
    python
    ];
}

When performing the build, pass these additional flags to cmake:

-DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang

Building the tool

The build process begins in the same directory in which this README resides:

cd citl-static-analysis
ls README.md

Now we decide between performing a Release or a Debug build. In what follows we will perform a Release build. To that end, we create a build directory and change into it:

mkdir -p build/Release ; cd build/Release
cmake -DCMAKE_BUILD_TYPE=Release ../..
make -j<CPUTHREADS> citl-static-analysis unit-tests

If this succeeds, the tool has been built and should be located at ./citl-static-analysis.

At this point we can run the test suite against the build to ensure basic functionality:

ctest

The test suite verifies the functionality of the tool by performing end-to-end tests against a fixed corpus of binaries and checking the resulting output against known-good values. Thus, if the tests pass, the tool should be ready for use.

Building and using a docker container of static-analysis

docker build -t citl-static-analysis .
docker run --rm -it static-a /bin/sh
citl-static-analysis -logtostderr -nolog_prefix -binfile <target_binary_path>

Running

From within the build directory:

./citl-static-analysis -logtostderr -nolog_prefix -binfile /path/to/binary

A full help output is available by running with (-help) arg.

Running multiple binaries

There is a small helper utility to run a large collection of binaries with the release build. First ensure that build/Release/ has been built. The tool will iterate through all files and directories searching for any file mime type that we support, please run:

pip install --user python-magic
python ./utils/citl-run-directory.py -d /path/to/bins -o /tmp/data

Helpful options

-printcfg       : Pretty prints a complete CFG basic block list.
-all_analyzers  : Toggles analyzers which create large amounts of output (ret distances for example)
-printsyms      : Pretty prints all resolved symbols as well as metadata about them.
--vmodule=CFG=1 : Toggles debugging information about CFG creation.
-addition_funcs : Toggles the Selectable function analyzer to check for call counts of user supplied function names.
                  ex: -addition_funcs "getopt,calloc"

Tests

In order to run all the integration style tests, change directories into the current build directory and run ctest.

Example:

cd build/Debug/
ctest
Issues
  • GCC stack-clash mitigation

    GCC stack-clash mitigation

    GCC (and in the future clang) implement a stack clash mitigation, we should be able to detect it:

    https://blog.qualys.com/securitylabs/2017/06/19/the-stack-clash https://gcc.gnu.org/legacy-ml/gcc-patches/2017-07/msg00556.html https://reviews.llvm.org/D68720

    enhancement 
    opened by mothran 1
  • Auto initialization analyzer

    Auto initialization analyzer

    LLVM/CLANG and MSVC are starting to add auto scalar / POD auto-initialization systems to the compilers, if they are toggleable, this might be a good one to write and analyzer for.

    LLVM: https://reviews.llvm.org/D54604

    MSVC: Currently they have not released the flag to the greater VC codebase https://twitter.com/aionescu/status/1066718242406891520?lang=en

    GCC: Unknown if they have a implementation

    Extra info (MSVC): https://www.youtube.com/watch?v=rQWjF8NvqAU&feature=youtu.be

    enhancement 
    opened by mothran 0
  • PPC64 Support

    PPC64 Support

    PPC64 currently unsupported by the CFG and other components.

    enhancement 
    opened by mothran 0
  • MIPS16 support

    MIPS16 support

    Currently capstone does not support mips16 extensions: https://github.com/aquynh/capstone/issues/241 https://github.com/aquynh/capstone/issues/1319

    So we have to block binaries that have that ELF tag until capstone adds support.

    enhancement 
    opened by mothran 0
  • MIPS64 support

    MIPS64 support

    Implementing MIPS64 support will require an overhaul of the CpuState set of MIPS helpers. Because mips 32/64 works much like x86 with just different modes of capstone. We need to hot swap based on the block mode within the helper functions.

    This also means swapping the return type to int64_t and enforcing correct casting between the 32/64 based values.

    enhancement 
    opened by mothran 0
  • x86-32 thunk + GOT loads

    x86-32 thunk + GOT loads

    On x86-32 binaries sometimes there will be a call to "__x86.get_pc_thunk.bx" or similar function to load up a register with the current address so the binary will do a IP relative style reference into the GOT (in pie binaries).

    This behavior effects the Fuzzer Imm generator because it will load up the register at the head of function and use it throughout for loads of strings from the rodata.

    We need to implement a system to detect that, then cache that register value for the whole function processing cycle.

    bug 
    opened by mothran 0
  • Retguard analyzer

    Retguard analyzer

    The openbsd port of clang includes a -fret-protect (Retguard) extension that injects a cookie check, jmp and int3's into each function's epilogue to prevent ROP gadgets from working on aligned instructions.

    Currently retguard only applies to openbsd's clang built binaries interesting to track because its possible it might be upstreamed at some point. There is both a x86 and aarch64 version of retguard.

    As for x86 I think the best model for an analyzer so far would be to trigger on a per instruction analyzer that walks back up from a ret* instruction then checks for the canary check, the jump and the int3's. We would need to walk past the first int3 and manually check the second because the CFG breaks on interrupts but should be easy enough to check one byte forward without a disasm. I would like a list of protected functions so we can get an idea of coverage when we report on it.

    aarch64 would need a bit more research but should be doable in a similar fashion to x86.

    References: https://undeadly.org/cgi?action=article;sid=20181231111206 https://marc.info/?l=openbsd-tech&m=150317547021396&w=2 https://github.com/openbsd/src/blob/master/gnu/llvm/lib/Target/X86/X86ReturnProtectorLowering.cpp https://github.com/openbsd/src/blob/master/gnu/llvm/lib/Target/AArch64/AArch64ReturnProtectorLowering.cpp

    enhancement 
    opened by mothran 0
  • ROP-ability measurements

    ROP-ability measurements

    We should measure how easy it would be for tools like ROPGadget.py (https://github.com/JonathanSalwan/ROPgadget) to generate valid ROP chains on the binaries we process. An example mitigation can be found in the clang extensions from openbsd:

    https://github.com/openbsd/src/blob/master/gnu/llvm/lib/Target/X86/X86FixupGadgets.cpp

    This attempts to reduce the introduction of 0xc3 from the mod/rm (dst = RBX).

    For a given architecture we should measure how often a byte sequence corresponding to a valid return is found within the executable segments of the binary. We could also extend this a bit by walking back from there doing a little sanity checking for possible chains. I want to avoid re-writing the expensive tooling of ROPGadget but make something that can give us a rough measure of likely chains.

    Finally we should sanity check our final product with ROPGadget's output to ensure we are at least model accurate.

    enhancement 
    opened by mothran 0
  • Cyclomatic complexity Analyzer

    Cyclomatic complexity Analyzer

    Implement a analyzer that generates the cyclomatic complexity for each function in the binary. Because we have both blocks and followers / leaders we should be able to create a mapping for each function of 'linked blocks' as a set of pairs. The pairs might need a custom compare method that ignores ordering.

    Currently we don't have a solid way to run analyzers 'for each function' so that will need to be implemented as its own analyzer class first. This might be worth holding off until all analyzers are in the event model.

    enhancement 
    opened by mothran 0
  • Code Obfuscation Heuristics

    Code Obfuscation Heuristics

    Currently we have no heuristics to make judgments about code obfuscation of our input binaries. We should do a study of possible common metrics of obfuscation tools and codify them.

    enhancement 
    opened by mothran 0
Owner
Cyber Independent Testing Lab
Cyber Independent Testing Lab
Pharos Static Binary Analysis Framework

Automated static analysis tools for binary programs

Software Engineering Institute 1.1k Oct 19, 2021
Tool to check C++ #include dependencies (dependency graphs created in .dot format)

Read Me for Dependency Checker Copyright (C) 2012-2017, TomTom International BV. All rights reserved. The tool cpp-dependencies creates #include depen

TomTom 637 Oct 11, 2021
A static analyzer for Java, C, C++, and Objective-C

Infer Infer is a static analysis tool for Java, C++, Objective-C, and C. Infer is written in OCaml. Installation Read our Getting Started page for det

Facebook 12.7k Oct 18, 2021
A tool for use with clang to analyze #includes in C and C++ source files

Include What You Use For more in-depth documentation, see docs. Instructions for Users "Include what you use" means this: for every symbol (type, func

null 2.5k Oct 15, 2021
Static code checker for C++

cpplint - static code checker for C++ Cpplint is a command-line tool to check C/C++ files for style issues following Google's C++ style guide. Cpplint

null 889 Oct 9, 2021
将 C/C++ 代码转换成流程图 / Turn your C/C++ code into flowchart

将 C/C++ 代码转换成流程图 / Turn your C/C++ code into flowchart

mgt 293 Oct 17, 2021