Experimental mutation testing tool for Swift and XCTest powered by mull

Overview

mull-xctest

Experimental mutation testing tool for Swift and XCTest powered by mull.

⚠️ This tool is still experimental and under development.

Installation

This tool is distributed through Homebrew. Run the following command to install:

brew install kateinoigakukun/tap/mull-xctest

Usage

SwiftPM

TODO: Add more docs

# Compile with bitcode and link by mull-ld in mull-tools directory.
$ swift build --build-tests \
    -Xswiftc -embed-bitcode \
    -Xswiftc -tools-directory -Xswiftc /usr/local/opt/mull-xctest/bin/mull-tools

# Run mutation tests and measure mutation score.
$ mull-xctest .build/debug/StubKitPackageTests.xctest

Xcode

TODO: Add more docs

$ xcodebuild build-for-testing \
  -project StubKit.xcodeproj -scheme StubKitTests \
  -destination "platform=iOS Simulator,name=iPhone 8" \
  OTHER_SWIFT_FLAGS="\$(inherited) -embed-bitcode" \
  OTHER_CFLAGS="\$(inherited) -fembed-bitcode" \
  LD="$(which mull-ld)" \
  OTHER_LDFLAGS="\$(inherited) \
    -Xmull --linker=clang \
    -Xmull --linker-flavor=clang"

$ mull-xctestrun \
  -Xxcodebuild -destination \
  -Xxcodebuild "platform=iOS Simulator,name=iPhone 8" \
  --test-target StubKitTests \
  path/to/Build/Products/StubKitTests_iphonesimulator14.4-x86_64.xctestrun

You might also like...
A YAML parser and emitter in C++

yaml-cpp yaml-cpp is a YAML parser and emitter in C++ matching the YAML 1.2 spec. To get a feel for how it can be used, see the Tutorial or How to Emi

Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications

Zmeya Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serializ

Use to copy a file from an NTFS partitioned volume by reading the raw volume and parsing the NTFS structures.

ntfsDump Use to copy a file from an NTFS partitioned volume by reading the raw volume and parsing the NTFS structures. Similar to https://github.com/P

A C++11 ASN.1 BER Encoding and Decoding Library

fast_ber A performant ASN.1 BER encoding and decoding library written in C++11 Introduction fast_ber is a small, lightweight library for BER encoding

A high performance C++14 library for effortlessly reading and writing UBJSON

UbjsonCpp A high performance C++14 library for effortlessly reading and writing UBJSON This library implements UBJSON Draft 12 and Value semmantics Ab

High-performance specialized replacements for PHP's pack() and unpack() functions

ext-encoding High-performance specialized replacements for PHP's pack() and unpack() functions Under a profiler, it becomes obvious that PHP's pack()

International obfuscated contest: Small C program to minify HTML sources and generate a minified HTML output.
International obfuscated contest: Small C program to minify HTML sources and generate a minified HTML output.

HTML Minifier C International obfuscated contest: Just a small C program to minify HTML sources and generate a minified HTML output. Using $ gcc html-

Decode and Print Kadena Chainweb Block Headers

decode-header - Decode and Print Kadena Chainweb BlockHeaders The tool reads Kadena Chainweb Headers in various formats from stdin and prints a JSON r

Comments
  • `No mutants found. Mutation score: infinitely high` error

    `No mutants found. Mutation score: infinitely high` error

    I have a swift packet with UIKit import. Unfortunately there is no way to build it using swift build. That's why tried 3 different ways to use mull-xctests.

    1. I generate a project from a SPM with swift package generate-xcodeproj. Then I build it with xcodebuild build-for-testing -project PSBCore.xcodeproj -scheme PSBCore-Package -destination "platform=iOS Simulator,name=iPhone 11 Pro" OTHER_SWIFT_FLAGS="\$(inherited) -embed-bitcode" OTHER_CFLAGS="\$(inherited) -fembed-bitcode" LD="$(which mull-ld)" OTHER_LDFLAGS="\$(inherited) -Xmull --linker=clang -Xmull --linker-flavor=clang". It succeed. Then when I run mull-xctestrun -Xxcodebuild -destination -Xxcodebuild "platform=iOS Simulator,name=iPhone 11 Pro" --test-target PSBCoreTests /path/to/Library/Developer/Xcode/DerivedData/PSBCore-btiytljcqmsisacjtlzuldouejsz/Build/Products/PSBCore-Package_iphonesimulator15.0-x86_64.xctestrun and get [info] No mutants found. Mutation score: infinitely high
    2. I even tried to test the project if the command similar to above, but if test instead of building-for-testing. After that I tried mull-xctest /path/to/Library/Developer/Xcode/DerivedData/PSBCore-btiytljcqmsisacjtlzuldouejsz/Build/Products/Debug-iphonesimulator/PSBCoreTests.xctest. It also gives me following:
    [info] Warm up run (threads: 1)
           [################################] 1/1. Finished in 45ms
    [info] Baseline run (threads: 1)
           [################################] 1/1. Finished in 41ms
    [info] No mutants found. Mutation score: infinitely high
    
    1. I've also tried to build a SPM with the command from the 1st, but without generating a Project file. This gives me an error:
    x86_64-apple-ios11.0-simulator: No such file or directory
    Command Ld failed with a nonzero exit code
    

    why is there x86_64-apple-ios11.0-simulator? I'm using Xcode 13.0, which doesn't seem to have ios11 simulator.

    What I'm doing wrong? Pls, help me.

    opened by ws233 7
  • Library not loaded: /usr/local/opt/llvm/lib/libLLVM.dylib

    Library not loaded: /usr/local/opt/llvm/lib/libLLVM.dylib

    Getting error on mull-ld launch and during xcodebuild setup:

    dyld: Library not loaded: /usr/local/opt/llvm/lib/libLLVM.dylib
      Referenced from: /usr/local/bin/mull-ld
      Reason: image not found
    Command Ld failed with a nonzero exit code
    
    opened by OdNairy 4
  • Final Release date?

    Final Release date?

    We are looking for a tool to implement Mutation Tests on iOS side in huge project. Do you guys have any ETA about releasing this library? Is this already usable?

    opened by felipecp 0
  • Different result depending on number of workers

    Different result depending on number of workers

    I build your test project StubKit with command

    xcodebuild build-for-testing \
      -project StubKit.xcodeproj -scheme StubKit-Package \
      -destination "platform=iOS Simulator,name=iPhone 8" \
      OTHER_SWIFT_FLAGS="\$(inherited) -embed-bitcode" \
      OTHER_CFLAGS="\$(inherited) -fembed-bitcode" \
      LD="$(which mull-ld)" \
      OTHER_LDFLAGS="\$(inherited) \
        -Xmull --linker=clang \
        -Xmull --linker-flavor=clang"
    

    Then I try to find mutation with the following different --workers= values: 1, 2,4, 8, 9, 10. and finally, no parameter at all. Like follows:

    mull-xctestrun \              
      -Xxcodebuild -destination \
      -Xxcodebuild "platform=iOS Simulator,name=iPhone 8" \
      --test-target StubKitTests /Path/to/Library/Developer/Xcode/DerivedData/StubKit-errbmrafikpnlmgukqsogexfeeyk/Build/Products/StubKit-Package_iphonesimulator14.1-x86_64.xctestrun 
     --workers=1
    

    all of them produces different Mutation scores corresponding: 0%, 55%, 66%, 77%, 44%, 77%.

    It seems like workers don't continue to get the scheduled jobs. And the correct result is only available, when the number of workers are equal to number of mutants found. That become an issue for projects having more than 16 mutants ^.^

    It seems like there is a bug. Could you fix it or explain, how I could fix it by myself?

    opened by ws233 5
Owner
Yuta Saito
UTC+9
Yuta Saito
Header-only TOML config file parser and serializer for C++17 (and later!).

toml++ homepage ✨ This README is fine, but the toml++ homepage is better. ✨ Library features Header-only Supports the latest TOML release (v1.0.0), pl

Mark Gillard 970 Jan 9, 2023
A C++11 or library for parsing and serializing JSON to and from a DOM container in memory.

Branch master develop Azure Docs Drone Matrix Fuzzing --- Appveyor codecov.io Boost.JSON Overview Boost.JSON is a portable C++ library which provides

Boost.org 333 Dec 29, 2022
libcluon is a small and efficient, single-file and header-only library written in modern C++ to power microservices.

libcluon Linux & OSX Build (TravisCI) Win64 Build (AppVeyor) Test Coverage Coverity Analysis CII Best Practices libcluon is a small single-file, heade

Christian Berger 81 Nov 30, 2022
Microsoft 2.5k Dec 31, 2022
Cap'n Proto serialization/RPC system - core tools and C++ library

Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except

Cap'n Proto 9.5k Jan 1, 2023
Header-only C++11 library to encode/decode base64, base64url, base32, base32hex and hex (a.k.a. base16) as specified in RFC 4648, plus Crockford's base32. MIT licensed with consistent, flexible API.

cppcodec Header-only C++11 library to encode/decode base64, base64url, base32, base32hex and hex (a.k.a. base16) as specified in RFC 4648, plus Crockf

Topology 491 Dec 28, 2022
MessagePack implementation for C and C++ / msgpack.org[C/C++]

msgpack for C/C++ It's like JSON but smaller and faster. Overview MessagePack is an efficient binary serialization format, which lets you exchange dat

MessagePack 2.6k Dec 31, 2022
FlatBuffers Compiler and Library in C for C

OS-X & Ubuntu: Windows: The JSON parser may change the interface for parsing union vectors in a future release which requires code generation to match

null 550 Dec 25, 2022
Utility to convert any binary file into C source that can be compiled and linked to the executable.

bin2c Utility to convert any binary file into C source that can be compiled and linked to the executable. bin2o Utility to convert any binary file int

Vadim A. Anisimov 16 Jul 14, 2021