Practical mutation testing tool for C and C++

Overview

Mull

Mull is a tool for Mutation Testing based on LLVM/Clang with a strong focus on C and C++ languages.

For installation and usage please refer to the latest documentation: https://mull.readthedocs.io/en/latest/

Join us in Discord

Here is the invitation link to the Discord channel: https://discord.gg/Hphp7dW

Contributing

Here is the starting point: CONTRIBUTING.md

Citation

Mull it over: mutation testing based on LLVM (preprint)

@INPROCEEDINGS{8411727, 
author={A. Denisov and S. Pankevich}, 
booktitle={2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)}, 
title={Mull It Over: Mutation Testing Based on LLVM}, 
year={2018}, 
volume={}, 
number={}, 
pages={25-31}, 
keywords={just-in-time;program compilers;program testing;program verification;mutations;Mull;LLVM IR;mutated programs;compiled programming languages;LLVM framework;LLVM JIT;tested program;mutation testing tool;Testing;Tools;Computer languages;Instruments;Runtime;Computer crashes;Open source software;mutation testing;llvm}, 
doi={10.1109/ICSTW.2018.00024}, 
ISSN={}, 
month={April},}

Packages

Hosted By: Cloudsmith

Hosting for precompiled packages is graciously provided by Cloudsmith.

Copyright

Copyright (c) 2016-2020 Alex Denisov [email protected] and Stanislav Pankevich [email protected]. See LICENSE for details.

Comments
  • Skipped mutation after increasing workers

    Skipped mutation after increasing workers

    Hello, I just noticed that more workers I use more mutation mull skips, I use the flag --workers. It is this normal? I was trying to increase the number of threads to make mull run faster.

    opened by AndreiBogdanPislaru 29
  • Mull doesn't generate mutations for lines changed in specified Git diff.

    Mull doesn't generate mutations for lines changed in specified Git diff.

    Description

    Mull doesn't generate mutations for lines changed in specified Git diff.

    [email protected]:~/sources/mull_testing$ git show 580212eee09fdbe89fb00b6d577f026106d0a8aa
    commit 580212eee09fdbe89fb00b6d577f026106d0a8aa
    Author: Sergey Bronnikov <[email protected]>
    Date:   Mon Mar 22 11:15:30 2021 +0300
    
        add different statements to main
    
    diff --git a/main.c b/main.c
    index 67eb934..8405df3 100644
    --- a/main.c
    +++ b/main.c
    @@ -4,6 +4,8 @@ void f1() {
         puts("abc");
         int a = 0;
         a = 2 + 2;
    +    a = a + 2;
    +    a = 2 + a;
     }
     
     int main() {
    
    [email protected]:~/sources/mull_testing$ mull-cxx --git-diff-ref 580212eee09fdbe89fb00b6d577f026106d0a8aa --git-project-root . ./main
    [info] Extracting bitcode from executable (threads: 1)
           [################################] 1/1. Finished in 1ms
    [info] Loading bitcode files (threads: 3)
           [################################] 3/3. Finished in 12ms
    [info] Incremental testing using Git Diff is enabled.
    - Git ref: 580212eee09fdbe89fb00b6d577f026106d0a8aa
    - Git project root: .
    [info] Sanity check run (threads: 1)
           [################################] 1/1. Finished in 2ms
    [info] Gathering functions under test (threads: 1)
           [################################] 1/1. Finished in 0ms
    [info] Applying function filter: no debug info (threads: 8)
           [################################] 10/10. Finished in 1ms
    [info] Applying function filter: file path (threads: 4)
           [################################] 4/4. Finished in 0ms
    [info] Instruction selection (threads: 4)
           [################################] 4/4. Finished in 10ms
    [info] Searching mutants across functions (threads: 4)
           [################################] 4/4. Finished in 1ms
    [info] Deduplicate mutants (threads: 1)
           [################################] 1/1. Finished in 0ms
    [info] No mutants found. Mutation score: infinitely high
    [info] Total execution time: 50ms
    [email protected]:~/sources/mull_testing$ 
    

    How-to reproduce:

    1. git clone https://github.com/ligurio/mull-testing
    2. cd mull-testing
    3. mull-cxx --git-diff-ref 580212eee09fdbe89fb00b6d577f026106d0a8aa --git-project-root . ./main

    Version

    [email protected]:~/sources/mull_testing$ mull-cxx --version Mull: LLVM-based mutation testing https://github.com/mull-project/mull Version: 0.10.0-trunk1 Commit: 296ee8e Date: 10 Mar 2021 LLVM: 11.0.0

    opened by ligurio 17
  • Trouble embedding bitcode into object file

    Trouble embedding bitcode into object file

    Hello,

    I'm trying to embed bitcode into an executable for the MQTT protocol library and I'm running into some problems. I compile a lot of files into .bc files using a line similar to this one:

    clang -I. -I.. -I../lib -I../src/deps -Wall -ggdb -c -O0 -emit-llvm -fno-inline -DVERSION="\"1.6.7\"" -DWITH_BROKER -fembed-bitcode -c mosquitto.c -o mosquitto.bc

    and then I link all the files together using llvm-link-10. Once it links and I try to run mull-cxx on the resulting executable, it says "The file was not recognized as a valid object file" and then no mutants were found. So it looks like it is able to embed the bitcode into the executable, but it isn't creating a valid object file for some reason. It's a lot of files so I'm assuming that some mutant would be generated somewhere, I'm curious why it looks like none can be found. My guess guess is it has to do with some way I'm compiling the executable. Does anyone have any ideas on how I can fix this? Any help would be greatly appreciated!

    opened by hdwhittaker 15
  • Some question about './', 'stddef.h' and GTest

    Some question about './', 'stddef.h' and GTest

    If i use the mull-cxx command like in hello-world example: $ mull-cxx hello-world [info] Extracting bitcode from executable (threads: 1) [################################] 1/1. Finished in 1ms [info] Loading bitcode files (threads: 1) [################################] 1/1. Finished in 0ms [info] Sanity check run (threads: 1) [error] Executable not found: hello-world [error] Error messages are treated as fatal errors. Exiting now.

    I solved this first problem using $ mull-cxx ./hello-world, my doubt is whether or not mull-cxx is supposed to work without './' and I do some configuration wrong.


    If my code include <stddef.h> or other library which contain it, like <stdio.h>, i have a warning: stddef.h not found.

    $ mull-cxx ./hello-world [info] Extracting bitcode from executable (threads: 1) [################################] 1/1. Finished in 10ms [info] Loading bitcode files (threads: 1) [################################] 1/1. Finished in 2ms [info] Sanity check run (threads: 1) [################################] 1/1. Finished in 2ms [info] Gathering functions under test (threads: 1) [################################] 1/1. Finished in 0ms [info] Applying function filter: no debug info (threads: 1) [################################] 4/4. Finished in 1ms [info] Applying function filter: file path (threads: 1) [################################] 2/2. Finished in 0ms [info] Instruction selection (threads: 1) [################################] 2/2. Finished in 0ms [info] Searching mutants across functions (threads: 1) [################################] 2/2. Finished in 0ms [info] Applying filter: no debug info (threads: 1) [################################] 6/6. Finished in 0ms [info] Applying filter: file path (threads: 1) [################################] 6/6. Finished in 0ms [info] Applying filter: junk (threads: 1) In file included from /home/andrei2/MULL/main.cpp:2: /usr/include/stdio.h:33:10: fatal error: 'stddef.h' file not found #include <stddef.h> ^~~~~~~~~~ [warning] Cannot parse file: '/home/andrei2/MULL/main.cpp': mull-cxx /home/andrei2/MULL/main.cpp Make sure that the flags provided to Mull are the same flags that are used for normal compilation. [################################] 6/6. Finished in 18ms [info] Deduplicate mutants (threads: 1) [################################] 1/1. Finished in 0ms [info] Prepare mutations (threads: 1) [################################] 1/1. Finished in 0ms [info] Cloning functions for mutation (threads: 1) [################################] 1/1. Finished in 1ms [info] Removing original functions (threads: 1) [################################] 1/1. Finished in 0ms [info] Redirect mutated functions (threads: 1) [################################] 1/1. Finished in 0ms [info] Applying mutations (threads: 1) [################################] 4/4. Finished in 0ms [info] Compiling original code (threads: 1) [################################] 1/1. Finished in 16ms [info] Link mutated program (threads: 1) [info] Compiled executable: /tmp/mull-8100f7.exe [################################] 1/1. Finished in 226ms [info] Warm up run (threads: 1) [################################] 1/1. Finished in 1ms [info] Baseline run (threads: 1) [################################] 1/1. Finished in 1ms [info] Running mutants (threads: 1) [################################] 4/4. Finished in 12ms [info] Survived mutants (1/4): /home/andrei2/MULL/main.cpp:6:10: warning: Survived: Replaced >= with > [cxx_ge_to_gt] if(age >=21){ ^ [info] Mutation score: 75% [info] Total execution time: 290ms


    If I build a GTest with clang and I input mull-cxx ./gtest, I have an error that says undefined reference to main. If I execute gtest, it run all my tests without problems.

    $ mull-cxx ./gtest [info] Extracting bitcode from executable (threads: 1) [################################] 1/1. Finished in 4ms [info] Loading bitcode files (threads: 1) [################################] 7/7. Finished in 56ms [info] Sanity check run (threads: 1) [################################] 1/1. Finished in 25ms [info] Gathering functions under test (threads: 1) [################################] 1/1. Finished in 0ms [info] Applying function filter: no debug info (threads: 1) [################################] 644/644. Finished in 1ms [info] Applying function filter: file path (threads: 1) [################################] 485/485. Finished in 0ms [info] Instruction selection (threads: 1) [################################] 485/485. Finished in 23ms [info] Searching mutants across functions (threads: 1) [################################] 485/485. Finished in 22ms [info] Applying filter: no debug info (threads: 1) [################################] 606/606. Finished in 0ms [info] Applying filter: file path (threads: 1) [################################] 606/606. Finished in 0ms [info] Applying filter: junk (threads: 1) [------------------------------------------------------------------------------------------------------------------] [----------------a series of warning about stddef.h and stdbool.h not founded------------------] [------------------------------------------------------------------------------------------------------------------] [info] Deduplicate mutants (threads: 1) [################################] 1/1. Finished in 1ms [info] Prepare mutations (threads: 1) [################################] 1/1. Finished in 0ms [info] Cloning functions for mutation (threads: 1) [################################] 7/7. Finished in 195ms [info] Removing original functions (threads: 1) [################################] 7/7. Finished in 56ms [info] Redirect mutated functions (threads: 1) [################################] 7/7. Finished in 1ms [info] Applying mutations (threads: 1) [################################] 169/169. Finished in 1ms [info] Compiling original code (threads: 1) [################################] 7/7. Finished in 1231ms [info] Link mutated program (threads: 1) [error] Cannot link program status: Timedout time: 30202ms exit: -110 command: clang /tmp/mull-7e6cce.o /tmp/mull-fe1e98.o /tmp/mull-69b8ae.o /tmp/mull-a36dd9.o /tmp/mull-25ac3b.o /tmp/mull-9ba076.o /tmp/mull-e65f4e.o -o /tmp/mull-7b7232.exe stdout: stderr: /usr/bin/ld: /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/crt1.o: in function _start': (.text+0x24): undefined reference tomain'

    I set these flags in my script bash i use to switch between clang and gcc compiler

    compiler="-D CMAKE_C_COMPILER=clang -D CMAKE_CXX_COMPILER=clang++" flag="-fembed-bitcode" debug="-g" cmake $compiler -D CMAKE_C_FLAGS=$flag -D CMAKE_C_FLAGS_DEBUG_INIT=$debug -D CMAKE_CXX_FLAGS=$flag -D CMAKE_CXX_FLAGS_DEBUG_INIT=$debug

    opened by AndreiBogdanPislaru 15
  • Mull is not mutating the shared object file

    Mull is not mutating the shared object file

    The tool is not mutating the shared object file. Here are the details to replicate the behavior.

    Details:

    valid_age.h

    #ifndef MULL_EXAMPLE_H
    #define MULL_EXAMPLE_H
    
    int valid_age(int age);
    
    #endif
    

    valid_age.cpp

    #include "valid_age.h"
    
    int valid_age(int age) {
        if (age >= 21) {
            return 1;
        }
        return 0;
    }
    
    

    test_valid_age.cpp

    #include "mull_example_lib.h"
    
    
    int main() {
        int test1 = valid_age(25) == true;
        if (!test1) {
            /// test failed
            return 1;
        }
    
        int test2 = valid_age(20) == false;
        if (!test2) {
            /// test failed
            return 1;
        }
    
        /// success
        return 0;
    }
    

    Commands

    clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c valid_age.cpp -o _objs/valid_age.pic.o
    clang++ -shared -o _objs/libvalid_age.so _objs/valid_age.pic.o -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
    clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c test_valid_age.cpp -o _objs/test_valid_age.pic.o
    clang++ -o test_valid_age.out '-Wl,-rpath,./_objs' -L./_objs _objs/test_valid_age.pic.o -lvalid_age -pie -pthread -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
    mull-cxx \
        --linker="/usr/local/bin/clang++" \
        --mutators="cxx_all" \
        --linker-flags="-Wl,-rpath,./_objs -L./_objs -Wl,-S -fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -lvalid_age -pthread -lstdc++ -lm" \
        --ide-reporter-show-killed \
        --reporters=IDE\
        --print-options \
        --debug \
        test_valid_age.out
    

    Expected Behavior

    The tool should mutate both valid_age.cpp and test_valid_age.cpp.

    Current Behavior

    The tool is only validating test_valid_age.cpp

    [info] Killed mutants (4/6):
    /home/mull_issue_code/test_valid_age.cpp:5:17: warning: Killed: Replaced call to a function with 42 [cxx_replace_scalar_call]
        int test1 = valid_age(25) == true;
                    ^
    /home/mull_issue_code/test_valid_age.cpp:5:31: warning: Killed: Replaced == with != [cxx_eq_to_ne]
        int test1 = valid_age(25) == true;
                                  ^
    /home/mull_issue_code/test_valid_age.cpp:11:17: warning: Killed: Replaced call to a function with 42 [cxx_replace_scalar_call]
        int test2 = valid_age(20) == false;
                    ^
    /home/mull_issue_code/test_valid_age.cpp:11:31: warning: Killed: Replaced == with != [cxx_eq_to_ne]
        int test2 = valid_age(20) == false;
                                  ^
    [info] Survived mutants (2/6):
    /home/mull_issue_code/test_valid_age.cpp:5:9: warning: Survived: Replaced 'T a = b' with 'T a = 42' [cxx_init_const]
        int test1 = valid_age(25) == true;
            ^
    /home/mull_issue_code/test_valid_age.cpp:11:9: warning: Survived: Replaced 'T a = b' with 'T a = 42' [cxx_init_const]
        int test2 = valid_age(20) == false;
            ^
    [info] Mutation score: 66%
    
    

    Versions

    mull-cxx

    Mull: LLVM-based mutation testing
    https://github.com/mull-project/mull
    Version: 0.11.2
    Commit: a9a8ed6c
    Date: 01 Oct 2021
    LLVM: 11.0.1
    
    opened by duthades 13
  • Mull corrupted LLVM module

    Mull corrupted LLVM module

    I tried using mull against our codebase and got this error:

    [error] Uh oh! Mull corrupted LLVM module. 28/56: matrixmultiply, anyhow, proc-macro2, wide, num-traits(build), typenum, cxx(build.rs), link-cplusplus(build.rs) Please, report the following error message here https://github.com/mull-project/mull/issues Underlying error message: Instruction does not dominate all uses! %157 = call zeroext i1 @ZN19kr2rc_api_internals5Timex8soonerEqERK8timespecS3(%struct.timespec* nonnull align 8 dereferenceable(16) %8, %struct.timespec* nonnull align 8 dereferenceable(16) %20), !dbg !17016 %160 = phi i1 [ %157, %158 ], [ %157, %150 ], !dbg !17017

    The module was corrupted by 'cxx_logical_and_to_or' mutator. To disable it, add the following: ignoreMutators:

    • cxx_logical_and_to_or

    build command was: catkin_make -DCMAKE_CXX_FLAGS="-O0 -fexperimental-new-pass-manager -fpass-plugin=/usr/lib/mull-ir-frontend-12 -g -grecord-command-line -fprofile-instr-generate -fcoverage-mapping" install

    opened by JaroslavHavrda 12
  • False unkilled mutant

    False unkilled mutant

    When i run my testsuite, mull tells me that a mutant is not killed, like so:

    Testing/Engine.cpp:53:9: warning: Survived: Replace Call: replaced a call to function _ZNK7Testing6Engine15currentModuleIdEv with 42
        if (currentModuleId() == moduleId)
            ^
    

    I was initially initially missing a test case, but then i added it. Mull keep telling me that this mutant is unkilled. So i decided to mutate my own code manually, and run my testsuite manually. One of the test case fails. So this mutant should really be killed.

    This got me thinking:

    • caching issues? No, the whole process is a script, and the whole project is rebuilt from scratch in a new directory
    • My test is wrong? No, manual mutation yields ftest failure
    • A bug in mull? Maybe...

    tested code:

    void Engine::setCurrentModuleId(Enums::ModuleId moduleId)
    {
        if (currentModuleId() == moduleId)
        {
            return;
        }
    
        m_currentModule.reset(Factory::createModule(moduleId, m_hal));
        emit currentModuleChanged(m_currentModule.get());
        emit currentModuleIdChanged(moduleId);
    }
    

    Test cases:

    TEST_F(EngineTestSuite, ChangingCurrentModuleTakesEffectAndNotifies)
    {
        QSignalSpy idChangedSpy(&m_engine, &Engine::currentModuleIdChanged);
        QSignalSpy moduleChangedSpy(&m_engine, &Engine::currentModuleChanged);
    
        m_engine.setCurrentModuleId(Enums::DisplayModuleId);
    
        EXPECT_EQ(m_engine.currentModuleId(), Enums::DisplayModuleId);
        EXPECT_NE(m_engine.currentModule(), nullptr);
        ASSERT_EQ(idChangedSpy.count(), 1);
        EXPECT_EQ(idChangedSpy.value(0).value(0).value<Enums::ModuleId>(), Enums::DisplayModuleId);
        ASSERT_EQ(moduleChangedSpy.count(), 1);
        EXPECT_NE(moduleChangedSpy.value(0).value(0).value<Module*>(), nullptr);
    }
    
    TEST_F(EngineTestSuite, SettingCurrentModuleTwiceInARawDoesNothing)
    {
        m_engine.setCurrentModuleId(Enums::DisplayModuleId);
    
        QSignalSpy idChangedSpy(&m_engine, &Engine::currentModuleIdChanged);
        QSignalSpy moduleChangedSpy(&m_engine, &Engine::currentModuleChanged);
        const Module* module = m_engine.currentModule();
    
        m_engine.setCurrentModuleId(Enums::DisplayModuleId);
    
        EXPECT_EQ(m_engine.currentModuleId(), Enums::DisplayModuleId);
        EXPECT_EQ(idChangedSpy.count(), 0);
        EXPECT_EQ(m_engine.currentModule(), module);
        EXPECT_EQ(moduleChangedSpy.count(), 0);
    }
    

    Manually modified tested code:

    void Engine::setCurrentModuleId(Enums::ModuleId moduleId)
    {
        if (/*currentModuleId()*/ Enums::ModuleId(42) == moduleId)
        {
            return;
        }
    
        m_currentModule.reset(Factory::createModule(moduleId, m_hal));
        emit currentModuleChanged(m_currentModule.get());
        emit currentModuleIdChanged(moduleId);
    }
    

    Checks that catch the mutation:

        EXPECT_EQ(idChangedSpy.count(), 0);
        EXPECT_EQ(m_engine.currentModule(), module);
        EXPECT_EQ(moduleChangedSpy.count(), 0);
    
    opened by chgans 12
  • Linking problems against LLVM because of C++11 ABI change

    Linking problems against LLVM because of C++11 ABI change

    Linking on my SLES 12 machine against LLVM fails because of ABI changes. I built LLVM myself. I tried using g++ as well as clang++ to build mull, but linking always fails, seemingly with relation to finding string related functions:

    [92/92] Linking CXX executable unittests/MullUnitTests
    FAILED: : && /usr/local/bin/clang++  -D_GLIBCXX_USE_CXX11_ABI=1  -D_GLIBCXX_USE_CXX11_ABI=1   -L/usr/local/lib  -Wl,-undefined,dynamic_lookup -rdynamic -ldl unittests/CMakeFiles/MullUnitTests.dir/CompilerTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/ConfigParserTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/ContextTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/DriverTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/ForkProcessSandboxTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/MutationPointTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/ModuleLoaderTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/DynamicCallTreeTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/MutatorsFactoryTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/TesteesTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/TestRunnersTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/UniqueIdentifierTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/TaskExecutorTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/MutatorsTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/NegateConditionMutatorTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/RemoveVoidFunctionMutatorTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/ScalarValueMutatorTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/ConditionalsBoundaryMutatorTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/JunkDetection/CXXJunkDetectorTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/SimpleTest/SimpleTestFinderTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/GoogleTest/GoogleTestFinderTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/CustomTestFramework/CustomTestRunnerTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/CustomTestFramework/CustomTestFinderTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/SQLiteReporterTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/TestModuleFactory.cpp.o  -o unittests/MullUnitTests  lib/libmull.a googletest/libgoogle-test.a -lLLVMOption -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMBitWriter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMOrcJIT -lLLVMTransformUtils -lLLVMExecutionEngine -lLLVMTarget -lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMMC -lLLVMAsmParser -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -pthread /usr/lib64/libz.so -lncurses /usr/lib64/libsqlite3.so -lLLVMOption -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMBitWriter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMOrcJIT -lLLVMTransformUtils -lLLVMExecutionEngine -lLLVMTarget -lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMMC -lLLVMAsmParser -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -Wl,--start-group /usr/local/lib/libclang.so /usr/local/lib/libclangTooling.a /usr/local/lib/libclangBasic.a /usr/local/lib/libclangFrontend.a /usr/local/lib/libclangAST.a /usr/local/lib/libclangDriver.a /usr/local/lib/libclangLex.a /usr/local/lib/libclangSema.a /usr/local/lib/libclangSerialization.a /usr/local/lib/libclangAnalysis.a /usr/local/lib/libclangParse.a /usr/local/lib/libclangEdit.a -Wl,--end-group && cd /home/def/git/mull/BuildNinja/unittests && /home/def/.HappyMake/cache/dependencies/newdb/cmake-3.11.0-sap2-linuxx86_64-linuxx86_64/bin/cmake -E create_symlink /home/def/git/mull/unittests/fixtures /home/def/git/mull/BuildNinja/unittests/fixtures
    unittests/CMakeFiles/MullUnitTests.dir/CompilerTests.cpp.o: In function `Compiler_CompileModule_Test::TestBody()':
    CompilerTests.cpp:(.text._ZN27Compiler_CompileModule_Test8TestBodyEv+0x164): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
    unittests/CMakeFiles/MullUnitTests.dir/TestRunnersTests.cpp.o: In function `SimpleTestRunner_runTest_Test::TestBody()':
    TestRunnersTests.cpp:(.text._ZN29SimpleTestRunner_runTest_Test8TestBodyEv+0x164): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
    lib/libmull.a(ConfigParser.cpp.o): In function `_ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE':
    ConfigParser.cpp:(.text._ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE[_ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE]+0x88): undefined reference to `llvm::yaml::ScalarTraits<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::output(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void*, llvm::raw_ostream&)'
    ConfigParser.cpp:(.text._ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE[_ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE]+0x208): undefined reference to `llvm::yaml::ScalarTraits<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::input(llvm::StringRef, void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
    lib/libmull.a(Driver.cpp.o): In function `llvm::sys::DynamicLibrary::LoadLibraryPermanently(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
    Driver.cpp:(.text._ZN4llvm3sys14DynamicLibrary22LoadLibraryPermanentlyEPKcPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN4llvm3sys14DynamicLibrary22LoadLibraryPermanentlyEPKcPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x19): undefined reference to `llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
    lib/libmull.a(Toolchain.cpp.o): In function `mull::Toolchain::Toolchain(mull::Config&)':
    Toolchain.cpp:(.text._ZN4mull9ToolchainC2ERNS_6ConfigE+0x16c): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
    lib/libmull.a(CXXJunkDetector.cpp.o): In function `getCompilationDatabase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
    CXXJunkDetector.cpp:(.text._ZL22getCompilationDatabaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x95): undefined reference to `clang::tooling::CompilationDatabase::loadFromDirectory(llvm::StringRef, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
    lib/libmull.a(InstrumentedCompilationTask.cpp.o): In function `mull::InstrumentedCompilationTask::operator()(__gnu_cxx::__normal_iterator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > const*, std::vector<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> >, std::allocator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > > > >, __gnu_cxx::__normal_iterator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > const*, std::vector<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> >, std::allocator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > > > >, std::vector<llvm::object::OwningBinary<llvm::object::ObjectFile>, std::allocator<llvm::object::OwningBinary<llvm::object::ObjectFile> > >&, mull::progress_counter&)':
    InstrumentedCompilationTask.cpp:(.text._ZN4mull27InstrumentedCompilationTaskclEN9__gnu_cxx17__normal_iteratorIPKSt10unique_ptrINS_10MullModuleESt14default_deleteIS4_EESt6vectorIS7_SaIS7_EEEESD_RSA_IN4llvm6object12OwningBinaryINSF_10ObjectFileEEESaISI_EERNS_16progress_counterE+0x161): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
    lib/libmull.a(MutantExecutionTask.cpp.o): In function `mull::MutantExecutionTask::operator()(__gnu_cxx::__normal_iterator<mull::MutationPoint* const*, std::vector<mull::MutationPoint*, std::allocator<mull::MutationPoint*> > >, __gnu_cxx::__normal_iterator<mull::MutationPoint* const*, std::vector<mull::MutationPoint*, std::allocator<mull::MutationPoint*> > >, std::vector<std::unique_ptr<mull::MutationResult, std::default_delete<mull::MutationResult> >, std::allocator<std::unique_ptr<mull::MutationResult, std::default_delete<mull::MutationResult> > > >&, mull::progress_counter&)':
    MutantExecutionTask.cpp:(.text._ZN4mull19MutantExecutionTaskclEN9__gnu_cxx17__normal_iteratorIPKPNS_13MutationPointESt6vectorIS4_SaIS4_EEEESA_RS7_ISt10unique_ptrINS_14MutationResultESt14default_deleteISC_EESaISF_EERNS_16progress_counterE+0x161): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
    lib/libmull.a(OriginalCompilationTask.cpp.o): In function `mull::OriginalCompilationTask::operator()(__gnu_cxx::__normal_iterator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > const*, std::vector<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> >, std::allocator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > > > >, __gnu_cxx::__normal_iterator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > const*, std::vector<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> >, std::allocator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > > > >, std::vector<llvm::object::OwningBinary<llvm::object::ObjectFile>, std::allocator<llvm::object::OwningBinary<llvm::object::ObjectFile> > >&, mull::progress_counter&)':
    OriginalCompilationTask.cpp:(.text._ZN4mull23OriginalCompilationTaskclEN9__gnu_cxx17__normal_iteratorIPKSt10unique_ptrINS_10MullModuleESt14default_deleteIS4_EESt6vectorIS7_SaIS7_EEEESD_RSA_IN4llvm6object12OwningBinaryINSF_10ObjectFileEEESaISI_EERNS_16progress_counterE+0x161): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
    lib/libmull.a(InstrumentationResolver.cpp.o): In function `mull::InstrumentationResolver::findSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
    InstrumentationResolver.cpp:(.text._ZN4mull23InstrumentationResolver10findSymbolERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xa7): undefined reference to `llvm::RTDyldMemoryManager::getSymbolAddressInProcess(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
    lib/libmull.a(NativeResolver.cpp.o): In function `mull::NativeResolver::findSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
    NativeResolver.cpp:(.text._ZN4mull14NativeResolver10findSymbolERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x94): undefined reference to `llvm::RTDyldMemoryManager::getSymbolAddressInProcess(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
    clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.
    Makefile:67: recipe for target 'ninja.build.unit-tests' failed
    make: *** [ninja.build.unit-tests] Error 1
    

    I played around with -D_GLIBCXX_USE_CXX11_ABI=0/1 to try and fix it but some part is always unlinkable. Is there a way to ensure the ABI is the same as used in building LLVM?

    opened by def- 12
  • The Makefile does not work on Ubuntu.

    The Makefile does not work on Ubuntu.

    I don't know what is going on with BuildNinja but it cannot find the file and it's infuriating, as I have installed it and followed the instructions to a tee. I've tried finding more information on ninja not being included, but there's really not a lot of documentation out there. I really want to use Mull to mutate a project I'm working on, as there are virtually no other mutation tools for C. Here is the error after running "make install":

    cd /opt/mull/BuildNinja && ninja mull-driver /bin/sh: 1: cd: can't cd to /opt/mull/BuildNinja Makefile:63: recipe for target 'ninja.build.mull-driver' failed make: *** [ninja.build.mull-driver] Error 2

    opened by Brick7Face 12
  • mull-cxx produces inconsistent mutation scores on multiple runs

    mull-cxx produces inconsistent mutation scores on multiple runs

    While running mull-cxx on the example code snippet found in the mull tutorial at step 3, we find inconsistent mutation scores from run to run.

    This shell script run_mull.txt runs mull on the above mentioned tutorial example for 1000 runs and counts incorrect (unexpected according to the tutorial) scores produced at any iteration.

    Here are the reports from the above experiment for two different versions of mull-cxx, versions 0.10.0 and 0.11.0: report_0.10.0.txt report_0.11.0.txt

    From the reports we gather:

    • version 0.10.0 produces 228 unexpected scores from 1000 runs
    • version 0.11.0 produces 159 unexpected scores from 1000 runs

    Please help confirm if this is an issue in mull-cxx or something we are doing wrong. If the problem is with how we are using mull-cxx, please help us identify where we are going wrong.

    opened by pravinblaze 11
  • Add a feature with mutating code changed by a patch

    Add a feature with mutating code changed by a patch

    Currently there is only a one supported scenario to run mutation testing with Mull -- run it on a whole source codebase. It's nice for case when we want to do it once to know how good code coverage in a project. For regular use it is not practical and would be better to support mutations limited by source code in patch. Imagine you are a developer in a project and you made a patch with some functionality and tests that covers that functionality. You want to know how good these tests covers your patch.

    opened by ligurio 11
  • Miscompilation of variadic functions (partially solved)

    Miscompilation of variadic functions (partially solved)

    Mull miscompiles functions with variadic functions if they need to be mutated. As a workaround, Mull skips mutation of such functions: https://github.com/mull-project/mull/pull/977, but it should be solved eventually.


    This issue is similar to #961

    Bug 
    opened by AlexDenisov 0
  • Mull miscompiles code with jump tables (partially solved)

    Mull miscompiles code with jump tables (partially solved)

    When the following code compiles:

    int printf(const char *, ...);
    int main(int argc, char **argv) {
      static const void *jump_table[] = {&&label, &&label2};
      goto *jump_table[argc - 1];
    label:
      printf("label1\n");
      goto out;
    label2:
      printf("label2\n");
      goto out;
    out:
      return 0;
    }
    

    Clang generates a global jump table containing blockaddresses.

    E.g.:

    @main.jump_table = internal global [2 x i8*] [i8* blockaddress(@main, %11), i8* blockaddress(@main, %13)], align 16
    

    Mull corrupts the blockaddresses when it replaces the original function with an indirect stub.

    We should be also duplicating and fixing all the blockaddresses before destructive actions.

    Bug 
    opened by AlexDenisov 1
  • Get source code of the mutated version of the original program

    Get source code of the mutated version of the original program

    Hello.

    I went through "Two-step analysis process" in the tutorial and get an executable file (hello-world-mutated) of the Hello World example by running: $ mull-cxx -mutate-only -output=hello-world-mutated hello-world

    I am wondering if it is possible to get the source of the mutants instead of executable file. Is there any command to do so or any approach?

    Any hint is appreciated.

    opened by RainyTong 3
  • SARIF reporter

    SARIF reporter

    We should add SARIF reporter so that we can get better GitHub integration https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning

    opened by AlexDenisov 1
Releases(0.19.0)
Owner
Mull Project
LLVM-based Mutation Testing
Mull Project
C++ Unit Testing Easier: A Header-only C++ unit testing framework

CUTE C++ Unit Testing Easier: A Header-only C++ unit testing framework usually available as part of the Cevelop C++ IDE (http://cevelop.com) Dependenc

Peter Sommerlad 36 Dec 26, 2022
A tool to help in testing client/server robustness in the presence of malformed data.

Tool to assist in testing robustness of network-attached services in the presence of malformed data.

Peter Farley 1 Aug 27, 2022
C++ Testing using spies and fakes for isolation and simulation

ELFspy is a library for linux for writing tests using fakes and spies in C++. For very large call graphs, the testing of the higher nodes can be diffi

Robin Nicholson 49 Dec 9, 2022
Googletest - Google Testing and Mocking Framework

GoogleTest OSS Builds Status Announcements Release 1.10.x Release 1.10.x is now available. Coming Soon Post 1.10.x googletest will follow Abseil Live

Google 28.8k Jan 9, 2023
Modern c++17 unit testing framework on Microsoft Windows, Apple macOS, Linux, iOS and android.

tunit Modern c++17 unit testing framework on Windows, macOS, Linux, iOS and android. Continuous Integration build status Operating system Status Windo

Gammasoft 8 Apr 5, 2022
testing joystick under Linux environment, support monitoring disconnection state and auto recovery mode

qjoystick This qjoystick class is rewritten based on the library: https://github.com/drewnoakes/joystick. Please look at this library if you want to s

ibov 1 Oct 30, 2021
Header-only C++11 library for property-based testing.

autocheck Header-only C++11 library for QuickCheck (and later, SmallCheck) testing. Please consult the wiki for documentation. Install conan remote ad

John Freeman 120 Dec 22, 2022
The fastest feature-rich C++11/14/17/20 single-header testing framework

master branch Windows All dev branch Windows All doctest is a new C++ testing framework but is by far the fastest both in compile times (by orders of

Viktor Kirilov 4.5k Jan 4, 2023
C++ xUnit-like testing framework without macros

tst C++ testing framework. Installation, documentation, tutorials See WiKi. Features xUnit-like concepts minimal use of preprocessor macros declarativ

cppfw 9 Sep 26, 2022
proftest is a C application for testing the quality of different operating system APIs for profiling.

proftest is a C application for testing the quality of different operating system APIs for profiling.

Felix Geisendörfer 5 Jul 23, 2021
c++ testing framework

iutest iutest - iris unit test framework Welcome to the iutest iutest is framework for writing C++ tests. Features An XUnit test framework. Header onl

srz_zumix 60 Sep 12, 2022
UT: C++20 μ(micro)/Unit Testing Framework

"If you liked it then you "should have put a"_test on it", Beyonce rule UT / μt | Motivation | Quick Start | Overview | Tutorial | Examples | User Gui

boost::ext 956 Jan 3, 2023
A micro unit-testing library for C/C++

µ-test A micro unit testing framework for C/C++ to get you up and running with unit-testing ASAP (even without libc). Usage Simply include the C and h

Trevor McKay 1 Dec 8, 2021
Bayesian A/B testing calculations for C++

BayesTest C++ Bayesian A/B testing calculations for C++ Based on this post by Evan Miller Also available in Rust Installation Include the header in yo

Andrew Kane 2 Nov 14, 2022
5G core testing solution

CoreScope CoreScope combines gNodeB and UE components without any radio transmission. It behaves like a UE and exposes an IP interface, but to the cor

srsRAN 37 Oct 21, 2022
A complete unit testing framework in a header

liblittletest A complete unit testing framework in a header liblittletest is an easy to use all-in-an-header testing framework; all you have to do in

Sebastiano Merlino 13 Nov 11, 2021
Simple C testing framework

MrTest Simple C testing framework Usage Copy the mrtest.c and mrtest.h file into your project. In order to use the mrtest main: create a .c file that

Maarten Raasveldt 2 Jul 20, 2022
Testing memleaks with /usr/share/bcc/tools/memleak

Testing memleaks with /usr/share/bcc/tools/memleak

Doug Horner 1 Dec 13, 2021
xtest is a C++ testing framework inspired by googletest.

xtest C++ testing framework inspired by googletest Explore the docs » Wiki · Report Bug · Request Feature Contents xtest Commence Prerequisites Ubuntu

Ayush Joshi 2 Dec 13, 2022