Hello, I just noticed that more workers I use more mutation mull skips, I use the flag --workers. It is this normal? I was trying to increase the number of threads to make mull run faster.

Description

Mull doesn't generate mutations for lines changed in specified Git diff.

sergeyb@pony:~/sources/mull_testing$ git show 580212eee09fdbe89fb00b6d577f026106d0a8aa
commit 580212eee09fdbe89fb00b6d577f026106d0a8aa
Author: Sergey Bronnikov <[email protected]>
Date:   Mon Mar 22 11:15:30 2021 +0300

    add different statements to main

diff --git a/main.c b/main.c
index 67eb934..8405df3 100644
--- a/main.c
+++ b/main.c
@@ -4,6 +4,8 @@ void f1() {
     puts("abc");
     int a = 0;
     a = 2 + 2;
+    a = a + 2;
+    a = 2 + a;
 }
 
 int main() {

sergeyb@pony:~/sources/mull_testing$ mull-cxx --git-diff-ref 580212eee09fdbe89fb00b6d577f026106d0a8aa --git-project-root . ./main
[info] Extracting bitcode from executable (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Loading bitcode files (threads: 3)
       [################################] 3/3. Finished in 12ms
[info] Incremental testing using Git Diff is enabled.
- Git ref: 580212eee09fdbe89fb00b6d577f026106d0a8aa
- Git project root: .
[info] Sanity check run (threads: 1)
       [################################] 1/1. Finished in 2ms
[info] Gathering functions under test (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Applying function filter: no debug info (threads: 8)
       [################################] 10/10. Finished in 1ms
[info] Applying function filter: file path (threads: 4)
       [################################] 4/4. Finished in 0ms
[info] Instruction selection (threads: 4)
       [################################] 4/4. Finished in 10ms
[info] Searching mutants across functions (threads: 4)
       [################################] 4/4. Finished in 1ms
[info] Deduplicate mutants (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] No mutants found. Mutation score: infinitely high
[info] Total execution time: 50ms
sergeyb@pony:~/sources/mull_testing$ 

How-to reproduce:

1. git clone https://github.com/ligurio/mull-testing
2. cd mull-testing
3. mull-cxx --git-diff-ref 580212eee09fdbe89fb00b6d577f026106d0a8aa --git-project-root . ./main

Version

sergeyb@pony:~/sources/mull_testing$ mull-cxx --version Mull: LLVM-based mutation testing https://github.com/mull-project/mull Version: 0.10.0-trunk1 Commit: 296ee8e Date: 10 Mar 2021 LLVM: 11.0.0

Hello,

I'm trying to embed bitcode into an executable for the MQTT protocol library and I'm running into some problems. I compile a lot of files into .bc files using a line similar to this one:

clang -I. -I.. -I../lib -I../src/deps -Wall -ggdb -c -O0 -emit-llvm -fno-inline -DVERSION="\"1.6.7\"" -DWITH_BROKER -fembed-bitcode -c mosquitto.c -o mosquitto.bc

and then I link all the files together using llvm-link-10. Once it links and I try to run mull-cxx on the resulting executable, it says "The file was not recognized as a valid object file" and then no mutants were found. So it looks like it is able to embed the bitcode into the executable, but it isn't creating a valid object file for some reason. It's a lot of files so I'm assuming that some mutant would be generated somewhere, I'm curious why it looks like none can be found. My guess guess is it has to do with some way I'm compiling the executable. Does anyone have any ideas on how I can fix this? Any help would be greatly appreciated!

If i use the mull-cxx command like in hello-world example: $ mull-cxx hello-world [info] Extracting bitcode from executable (threads: 1) [################################] 1/1. Finished in 1ms [info] Loading bitcode files (threads: 1) [################################] 1/1. Finished in 0ms [info] Sanity check run (threads: 1) [error] Executable not found: hello-world [error] Error messages are treated as fatal errors. Exiting now.

I solved this first problem using $ mull-cxx ./hello-world, my doubt is whether or not mull-cxx is supposed to work without './' and I do some configuration wrong.

If my code include <stddef.h> or other library which contain it, like <stdio.h>, i have a warning: stddef.h not found.

$ mull-cxx ./hello-world [info] Extracting bitcode from executable (threads: 1) [################################] 1/1. Finished in 10ms [info] Loading bitcode files (threads: 1) [################################] 1/1. Finished in 2ms [info] Sanity check run (threads: 1) [################################] 1/1. Finished in 2ms [info] Gathering functions under test (threads: 1) [################################] 1/1. Finished in 0ms [info] Applying function filter: no debug info (threads: 1) [################################] 4/4. Finished in 1ms [info] Applying function filter: file path (threads: 1) [################################] 2/2. Finished in 0ms [info] Instruction selection (threads: 1) [################################] 2/2. Finished in 0ms [info] Searching mutants across functions (threads: 1) [################################] 2/2. Finished in 0ms [info] Applying filter: no debug info (threads: 1) [################################] 6/6. Finished in 0ms [info] Applying filter: file path (threads: 1) [################################] 6/6. Finished in 0ms [info] Applying filter: junk (threads: 1) In file included from /home/andrei2/MULL/main.cpp:2: /usr/include/stdio.h:33:10: fatal error: 'stddef.h' file not found #include <stddef.h> ^~~~~~~~~~ [warning] Cannot parse file: '/home/andrei2/MULL/main.cpp': mull-cxx /home/andrei2/MULL/main.cpp Make sure that the flags provided to Mull are the same flags that are used for normal compilation. [################################] 6/6. Finished in 18ms [info] Deduplicate mutants (threads: 1) [################################] 1/1. Finished in 0ms [info] Prepare mutations (threads: 1) [################################] 1/1. Finished in 0ms [info] Cloning functions for mutation (threads: 1) [################################] 1/1. Finished in 1ms [info] Removing original functions (threads: 1) [################################] 1/1. Finished in 0ms [info] Redirect mutated functions (threads: 1) [################################] 1/1. Finished in 0ms [info] Applying mutations (threads: 1) [################################] 4/4. Finished in 0ms [info] Compiling original code (threads: 1) [################################] 1/1. Finished in 16ms [info] Link mutated program (threads: 1) [info] Compiled executable: /tmp/mull-8100f7.exe [################################] 1/1. Finished in 226ms [info] Warm up run (threads: 1) [################################] 1/1. Finished in 1ms [info] Baseline run (threads: 1) [################################] 1/1. Finished in 1ms [info] Running mutants (threads: 1) [################################] 4/4. Finished in 12ms [info] Survived mutants (1/4): /home/andrei2/MULL/main.cpp:6:10: warning: Survived: Replaced >= with > [cxx_ge_to_gt] if(age >=21){ ^ [info] Mutation score: 75% [info] Total execution time: 290ms

If I build a GTest with clang and I input mull-cxx ./gtest, I have an error that says undefined reference to main. If I execute gtest, it run all my tests without problems.

$ mull-cxx ./gtest [info] Extracting bitcode from executable (threads: 1) [################################] 1/1. Finished in 4ms [info] Loading bitcode files (threads: 1) [################################] 7/7. Finished in 56ms [info] Sanity check run (threads: 1) [################################] 1/1. Finished in 25ms [info] Gathering functions under test (threads: 1) [################################] 1/1. Finished in 0ms [info] Applying function filter: no debug info (threads: 1) [################################] 644/644. Finished in 1ms [info] Applying function filter: file path (threads: 1) [################################] 485/485. Finished in 0ms [info] Instruction selection (threads: 1) [################################] 485/485. Finished in 23ms [info] Searching mutants across functions (threads: 1) [################################] 485/485. Finished in 22ms [info] Applying filter: no debug info (threads: 1) [################################] 606/606. Finished in 0ms [info] Applying filter: file path (threads: 1) [################################] 606/606. Finished in 0ms [info] Applying filter: junk (threads: 1) [------------------------------------------------------------------------------------------------------------------] [----------------a series of warning about stddef.h and stdbool.h not founded------------------] [------------------------------------------------------------------------------------------------------------------] [info] Deduplicate mutants (threads: 1) [################################] 1/1. Finished in 1ms [info] Prepare mutations (threads: 1) [################################] 1/1. Finished in 0ms [info] Cloning functions for mutation (threads: 1) [################################] 7/7. Finished in 195ms [info] Removing original functions (threads: 1) [################################] 7/7. Finished in 56ms [info] Redirect mutated functions (threads: 1) [################################] 7/7. Finished in 1ms [info] Applying mutations (threads: 1) [################################] 169/169. Finished in 1ms [info] Compiling original code (threads: 1) [################################] 7/7. Finished in 1231ms [info] Link mutated program (threads: 1) [error] Cannot link program status: Timedout time: 30202ms exit: -110 command: clang /tmp/mull-7e6cce.o /tmp/mull-fe1e98.o /tmp/mull-69b8ae.o /tmp/mull-a36dd9.o /tmp/mull-25ac3b.o /tmp/mull-9ba076.o /tmp/mull-e65f4e.o -o /tmp/mull-7b7232.exe stdout: stderr: /usr/bin/ld: /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/crt1.o: in function _start': (.text+0x24): undefined reference tomain'

I set these flags in my script bash i use to switch between clang and gcc compiler

compiler="-D CMAKE_C_COMPILER=clang -D CMAKE_CXX_COMPILER=clang++" flag="-fembed-bitcode" debug="-g" cmake $compiler -D CMAKE_C_FLAGS=$flag -D CMAKE_C_FLAGS_DEBUG_INIT=$debug -D CMAKE_CXX_FLAGS=$flag -D CMAKE_CXX_FLAGS_DEBUG_INIT=$debug

The tool is not mutating the shared object file. Here are the details to replicate the behavior.

Details:

valid_age.h

#ifndef MULL_EXAMPLE_H
#define MULL_EXAMPLE_H

int valid_age(int age);

#endif

valid_age.cpp

#include "valid_age.h"

int valid_age(int age) {
    if (age >= 21) {
        return 1;
    }
    return 0;
}

test_valid_age.cpp

#include "mull_example_lib.h"


int main() {
    int test1 = valid_age(25) == true;
    if (!test1) {
        /// test failed
        return 1;
    }

    int test2 = valid_age(20) == false;
    if (!test2) {
        /// test failed
        return 1;
    }

    /// success
    return 0;
}

Commands

clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c valid_age.cpp -o _objs/valid_age.pic.o
clang++ -shared -o _objs/libvalid_age.so _objs/valid_age.pic.o -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c test_valid_age.cpp -o _objs/test_valid_age.pic.o
clang++ -o test_valid_age.out '-Wl,-rpath,./_objs' -L./_objs _objs/test_valid_age.pic.o -lvalid_age -pie -pthread -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
mull-cxx \
    --linker="/usr/local/bin/clang++" \
    --mutators="cxx_all" \
    --linker-flags="-Wl,-rpath,./_objs -L./_objs -Wl,-S -fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -lvalid_age -pthread -lstdc++ -lm" \
    --ide-reporter-show-killed \
    --reporters=IDE\
    --print-options \
    --debug \
    test_valid_age.out

Expected Behavior

The tool should mutate both valid_age.cpp and test_valid_age.cpp.

Current Behavior

The tool is only validating test_valid_age.cpp

[info] Killed mutants (4/6):
/home/mull_issue_code/test_valid_age.cpp:5:17: warning: Killed: Replaced call to a function with 42 [cxx_replace_scalar_call]
    int test1 = valid_age(25) == true;
                ^
/home/mull_issue_code/test_valid_age.cpp:5:31: warning: Killed: Replaced == with != [cxx_eq_to_ne]
    int test1 = valid_age(25) == true;
                              ^
/home/mull_issue_code/test_valid_age.cpp:11:17: warning: Killed: Replaced call to a function with 42 [cxx_replace_scalar_call]
    int test2 = valid_age(20) == false;
                ^
/home/mull_issue_code/test_valid_age.cpp:11:31: warning: Killed: Replaced == with != [cxx_eq_to_ne]
    int test2 = valid_age(20) == false;
                              ^
[info] Survived mutants (2/6):
/home/mull_issue_code/test_valid_age.cpp:5:9: warning: Survived: Replaced 'T a = b' with 'T a = 42' [cxx_init_const]
    int test1 = valid_age(25) == true;
        ^
/home/mull_issue_code/test_valid_age.cpp:11:9: warning: Survived: Replaced 'T a = b' with 'T a = 42' [cxx_init_const]
    int test2 = valid_age(20) == false;
        ^
[info] Mutation score: 66%

Versions

mull-cxx

Mull: LLVM-based mutation testing
https://github.com/mull-project/mull
Version: 0.11.2
Commit: a9a8ed6c
Date: 01 Oct 2021
LLVM: 11.0.1

I tried using mull against our codebase and got this error:

[error] Uh oh! Mull corrupted LLVM module. 28/56: matrixmultiply, anyhow, proc-macro2, wide, num-traits(build), typenum, cxx(build.rs), link-cplusplus(build.rs) Please, report the following error message here https://github.com/mull-project/mull/issues Underlying error message: Instruction does not dominate all uses! %157 = call zeroext i1 @ZN19kr2rc_api_internals5Timex8soonerEqERK8timespecS3(%struct.timespec* nonnull align 8 dereferenceable(16) %8, %struct.timespec* nonnull align 8 dereferenceable(16) %20), !dbg !17016 %160 = phi i1 [ %157, %158 ], [ %157, %150 ], !dbg !17017

The module was corrupted by 'cxx_logical_and_to_or' mutator. To disable it, add the following: ignoreMutators:

• cxx_logical_and_to_or

build command was: catkin_make -DCMAKE_CXX_FLAGS="-O0 -fexperimental-new-pass-manager -fpass-plugin=/usr/lib/mull-ir-frontend-12 -g -grecord-command-line -fprofile-instr-generate -fcoverage-mapping" install

When i run my testsuite, mull tells me that a mutant is not killed, like so:

Testing/Engine.cpp:53:9: warning: Survived: Replace Call: replaced a call to function _ZNK7Testing6Engine15currentModuleIdEv with 42
    if (currentModuleId() == moduleId)
        ^

I was initially initially missing a test case, but then i added it. Mull keep telling me that this mutant is unkilled. So i decided to mutate my own code manually, and run my testsuite manually. One of the test case fails. So this mutant should really be killed.

This got me thinking:

• caching issues? No, the whole process is a script, and the whole project is rebuilt from scratch in a new directory
• My test is wrong? No, manual mutation yields ftest failure
• A bug in mull? Maybe...

tested code:

void Engine::setCurrentModuleId(Enums::ModuleId moduleId)
{
    if (currentModuleId() == moduleId)
    {
        return;
    }

    m_currentModule.reset(Factory::createModule(moduleId, m_hal));
    emit currentModuleChanged(m_currentModule.get());
    emit currentModuleIdChanged(moduleId);
}

Test cases:

TEST_F(EngineTestSuite, ChangingCurrentModuleTakesEffectAndNotifies)
{
    QSignalSpy idChangedSpy(&m_engine, &Engine::currentModuleIdChanged);
    QSignalSpy moduleChangedSpy(&m_engine, &Engine::currentModuleChanged);

    m_engine.setCurrentModuleId(Enums::DisplayModuleId);

    EXPECT_EQ(m_engine.currentModuleId(), Enums::DisplayModuleId);
    EXPECT_NE(m_engine.currentModule(), nullptr);
    ASSERT_EQ(idChangedSpy.count(), 1);
    EXPECT_EQ(idChangedSpy.value(0).value(0).value<Enums::ModuleId>(), Enums::DisplayModuleId);
    ASSERT_EQ(moduleChangedSpy.count(), 1);
    EXPECT_NE(moduleChangedSpy.value(0).value(0).value<Module*>(), nullptr);
}

TEST_F(EngineTestSuite, SettingCurrentModuleTwiceInARawDoesNothing)
{
    m_engine.setCurrentModuleId(Enums::DisplayModuleId);

    QSignalSpy idChangedSpy(&m_engine, &Engine::currentModuleIdChanged);
    QSignalSpy moduleChangedSpy(&m_engine, &Engine::currentModuleChanged);
    const Module* module = m_engine.currentModule();

    m_engine.setCurrentModuleId(Enums::DisplayModuleId);

    EXPECT_EQ(m_engine.currentModuleId(), Enums::DisplayModuleId);
    EXPECT_EQ(idChangedSpy.count(), 0);
    EXPECT_EQ(m_engine.currentModule(), module);
    EXPECT_EQ(moduleChangedSpy.count(), 0);
}

Manually modified tested code:

void Engine::setCurrentModuleId(Enums::ModuleId moduleId)
{
    if (/*currentModuleId()*/ Enums::ModuleId(42) == moduleId)
    {
        return;
    }

    m_currentModule.reset(Factory::createModule(moduleId, m_hal));
    emit currentModuleChanged(m_currentModule.get());
    emit currentModuleIdChanged(moduleId);
}

Checks that catch the mutation:

    EXPECT_EQ(idChangedSpy.count(), 0);
    EXPECT_EQ(m_engine.currentModule(), module);
    EXPECT_EQ(moduleChangedSpy.count(), 0);

Linking on my SLES 12 machine against LLVM fails because of ABI changes. I built LLVM myself. I tried using g++ as well as clang++ to build mull, but linking always fails, seemingly with relation to finding string related functions:

[92/92] Linking CXX executable unittests/MullUnitTests
FAILED: : && /usr/local/bin/clang++  -D_GLIBCXX_USE_CXX11_ABI=1  -D_GLIBCXX_USE_CXX11_ABI=1   -L/usr/local/lib  -Wl,-undefined,dynamic_lookup -rdynamic -ldl unittests/CMakeFiles/MullUnitTests.dir/CompilerTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/ConfigParserTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/ContextTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/DriverTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/ForkProcessSandboxTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/MutationPointTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/ModuleLoaderTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/DynamicCallTreeTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/MutatorsFactoryTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/TesteesTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/TestRunnersTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/UniqueIdentifierTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/TaskExecutorTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/MutatorsTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/NegateConditionMutatorTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/RemoveVoidFunctionMutatorTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/ScalarValueMutatorTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/Mutators/ConditionalsBoundaryMutatorTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/JunkDetection/CXXJunkDetectorTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/SimpleTest/SimpleTestFinderTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/GoogleTest/GoogleTestFinderTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/CustomTestFramework/CustomTestRunnerTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/CustomTestFramework/CustomTestFinderTests.cpp.o unittests/CMakeFiles/MullUnitTests.dir/SQLiteReporterTest.cpp.o unittests/CMakeFiles/MullUnitTests.dir/TestModuleFactory.cpp.o  -o unittests/MullUnitTests  lib/libmull.a googletest/libgoogle-test.a -lLLVMOption -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMBitWriter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMOrcJIT -lLLVMTransformUtils -lLLVMExecutionEngine -lLLVMTarget -lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMMC -lLLVMAsmParser -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -pthread /usr/lib64/libz.so -lncurses /usr/lib64/libsqlite3.so -lLLVMOption -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMBitWriter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMOrcJIT -lLLVMTransformUtils -lLLVMExecutionEngine -lLLVMTarget -lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMMC -lLLVMAsmParser -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -Wl,--start-group /usr/local/lib/libclang.so /usr/local/lib/libclangTooling.a /usr/local/lib/libclangBasic.a /usr/local/lib/libclangFrontend.a /usr/local/lib/libclangAST.a /usr/local/lib/libclangDriver.a /usr/local/lib/libclangLex.a /usr/local/lib/libclangSema.a /usr/local/lib/libclangSerialization.a /usr/local/lib/libclangAnalysis.a /usr/local/lib/libclangParse.a /usr/local/lib/libclangEdit.a -Wl,--end-group && cd /home/def/git/mull/BuildNinja/unittests && /home/def/.HappyMake/cache/dependencies/newdb/cmake-3.11.0-sap2-linuxx86_64-linuxx86_64/bin/cmake -E create_symlink /home/def/git/mull/unittests/fixtures /home/def/git/mull/BuildNinja/unittests/fixtures
unittests/CMakeFiles/MullUnitTests.dir/CompilerTests.cpp.o: In function `Compiler_CompileModule_Test::TestBody()':
CompilerTests.cpp:(.text._ZN27Compiler_CompileModule_Test8TestBodyEv+0x164): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
unittests/CMakeFiles/MullUnitTests.dir/TestRunnersTests.cpp.o: In function `SimpleTestRunner_runTest_Test::TestBody()':
TestRunnersTests.cpp:(.text._ZN29SimpleTestRunner_runTest_Test8TestBodyEv+0x164): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
lib/libmull.a(ConfigParser.cpp.o): In function `_ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE':
ConfigParser.cpp:(.text._ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE[_ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE]+0x88): undefined reference to `llvm::yaml::ScalarTraits<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::output(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void*, llvm::raw_ostream&)'
ConfigParser.cpp:(.text._ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE[_ZN4llvm4yaml7yamlizeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr16has_ScalarTraitsIT_EE5valueEvE4typeERNS0_2IOERS9_bRNS0_12EmptyContextE]+0x208): undefined reference to `llvm::yaml::ScalarTraits<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::input(llvm::StringRef, void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
lib/libmull.a(Driver.cpp.o): In function `llvm::sys::DynamicLibrary::LoadLibraryPermanently(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
Driver.cpp:(.text._ZN4llvm3sys14DynamicLibrary22LoadLibraryPermanentlyEPKcPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN4llvm3sys14DynamicLibrary22LoadLibraryPermanentlyEPKcPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x19): undefined reference to `llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
lib/libmull.a(Toolchain.cpp.o): In function `mull::Toolchain::Toolchain(mull::Config&)':
Toolchain.cpp:(.text._ZN4mull9ToolchainC2ERNS_6ConfigE+0x16c): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
lib/libmull.a(CXXJunkDetector.cpp.o): In function `getCompilationDatabase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
CXXJunkDetector.cpp:(.text._ZL22getCompilationDatabaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x95): undefined reference to `clang::tooling::CompilationDatabase::loadFromDirectory(llvm::StringRef, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
lib/libmull.a(InstrumentedCompilationTask.cpp.o): In function `mull::InstrumentedCompilationTask::operator()(__gnu_cxx::__normal_iterator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > const*, std::vector<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> >, std::allocator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > > > >, __gnu_cxx::__normal_iterator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > const*, std::vector<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> >, std::allocator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > > > >, std::vector<llvm::object::OwningBinary<llvm::object::ObjectFile>, std::allocator<llvm::object::OwningBinary<llvm::object::ObjectFile> > >&, mull::progress_counter&)':
InstrumentedCompilationTask.cpp:(.text._ZN4mull27InstrumentedCompilationTaskclEN9__gnu_cxx17__normal_iteratorIPKSt10unique_ptrINS_10MullModuleESt14default_deleteIS4_EESt6vectorIS7_SaIS7_EEEESD_RSA_IN4llvm6object12OwningBinaryINSF_10ObjectFileEEESaISI_EERNS_16progress_counterE+0x161): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
lib/libmull.a(MutantExecutionTask.cpp.o): In function `mull::MutantExecutionTask::operator()(__gnu_cxx::__normal_iterator<mull::MutationPoint* const*, std::vector<mull::MutationPoint*, std::allocator<mull::MutationPoint*> > >, __gnu_cxx::__normal_iterator<mull::MutationPoint* const*, std::vector<mull::MutationPoint*, std::allocator<mull::MutationPoint*> > >, std::vector<std::unique_ptr<mull::MutationResult, std::default_delete<mull::MutationResult> >, std::allocator<std::unique_ptr<mull::MutationResult, std::default_delete<mull::MutationResult> > > >&, mull::progress_counter&)':
MutantExecutionTask.cpp:(.text._ZN4mull19MutantExecutionTaskclEN9__gnu_cxx17__normal_iteratorIPKPNS_13MutationPointESt6vectorIS4_SaIS4_EEEESA_RS7_ISt10unique_ptrINS_14MutationResultESt14default_deleteISC_EESaISF_EERNS_16progress_counterE+0x161): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
lib/libmull.a(OriginalCompilationTask.cpp.o): In function `mull::OriginalCompilationTask::operator()(__gnu_cxx::__normal_iterator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > const*, std::vector<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> >, std::allocator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > > > >, __gnu_cxx::__normal_iterator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > const*, std::vector<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> >, std::allocator<std::unique_ptr<mull::MullModule, std::default_delete<mull::MullModule> > > > >, std::vector<llvm::object::OwningBinary<llvm::object::ObjectFile>, std::allocator<llvm::object::OwningBinary<llvm::object::ObjectFile> > >&, mull::progress_counter&)':
OriginalCompilationTask.cpp:(.text._ZN4mull23OriginalCompilationTaskclEN9__gnu_cxx17__normal_iteratorIPKSt10unique_ptrINS_10MullModuleESt14default_deleteIS4_EESt6vectorIS7_SaIS7_EEEESD_RSA_IN4llvm6object12OwningBinaryINSF_10ObjectFileEEESaISI_EERNS_16progress_counterE+0x161): undefined reference to `llvm::EngineBuilder::selectTarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
lib/libmull.a(InstrumentationResolver.cpp.o): In function `mull::InstrumentationResolver::findSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
InstrumentationResolver.cpp:(.text._ZN4mull23InstrumentationResolver10findSymbolERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xa7): undefined reference to `llvm::RTDyldMemoryManager::getSymbolAddressInProcess(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
lib/libmull.a(NativeResolver.cpp.o): In function `mull::NativeResolver::findSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
NativeResolver.cpp:(.text._ZN4mull14NativeResolver10findSymbolERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x94): undefined reference to `llvm::RTDyldMemoryManager::getSymbolAddressInProcess(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
Makefile:67: recipe for target 'ninja.build.unit-tests' failed
make: *** [ninja.build.unit-tests] Error 1

I played around with -D_GLIBCXX_USE_CXX11_ABI=0/1 to try and fix it but some part is always unlinkable. Is there a way to ensure the ABI is the same as used in building LLVM?

I don't know what is going on with BuildNinja but it cannot find the file and it's infuriating, as I have installed it and followed the instructions to a tee. I've tried finding more information on ninja not being included, but there's really not a lot of documentation out there. I really want to use Mull to mutate a project I'm working on, as there are virtually no other mutation tools for C. Here is the error after running "make install":

cd /opt/mull/BuildNinja && ninja mull-driver /bin/sh: 1: cd: can't cd to /opt/mull/BuildNinja Makefile:63: recipe for target 'ninja.build.mull-driver' failed make: *** [ninja.build.mull-driver] Error 2

While running mull-cxx on the example code snippet found in the mull tutorial at step 3, we find inconsistent mutation scores from run to run.

This shell script run_mull.txt runs mull on the above mentioned tutorial example for 1000 runs and counts incorrect (unexpected according to the tutorial) scores produced at any iteration.

Here are the reports from the above experiment for two different versions of mull-cxx, versions 0.10.0 and 0.11.0: report_0.10.0.txt report_0.11.0.txt

From the reports we gather:

• version 0.10.0 produces 228 unexpected scores from 1000 runs
• version 0.11.0 produces 159 unexpected scores from 1000 runs

Please help confirm if this is an issue in mull-cxx or something we are doing wrong. If the problem is with how we are using mull-cxx, please help us identify where we are going wrong.

Currently there is only a one supported scenario to run mutation testing with Mull -- run it on a whole source codebase. It's nice for case when we want to do it once to know how good code coverage in a project. For regular use it is not practical and would be better to support mutations limited by source code in patch. Imagine you are a developer in a project and you made a patch with some functionality and tests that covers that functionality. You want to know how good these tests covers your patch.

Mull miscompiles functions with variadic functions if they need to be mutated. As a workaround, Mull skips mutation of such functions: https://github.com/mull-project/mull/pull/977, but it should be solved eventually.

This issue is similar to #961

When the following code compiles:

int printf(const char *, ...);
int main(int argc, char **argv) {
  static const void *jump_table[] = {&&label, &&label2};
  goto *jump_table[argc - 1];
label:
  printf("label1\n");
  goto out;
label2:
  printf("label2\n");
  goto out;
out:
  return 0;
}

Clang generates a global jump table containing blockaddresses.

E.g.:

@main.jump_table = internal global [2 x i8*] [i8* blockaddress(@main, %11), i8* blockaddress(@main, %13)], align 16

Mull corrupts the blockaddresses when it replaces the original function with an indirect stub.

We should be also duplicating and fixing all the blockaddresses before destructive actions.

Hello.

I went through "Two-step analysis process" in the tutorial and get an executable file (hello-world-mutated) of the Hello World example by running: $ mull-cxx -mutate-only -output=hello-world-mutated hello-world

I am wondering if it is possible to get the source of the mutants instead of executable file. Is there any command to do so or any approach?

Any hint is appreciated.

We should add SARIF reporter so that we can get better GitHub integration https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning