A C++ micro-benchmarking framework

Overview

Nonius

What is nonius?

Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a similar Haskell-based tool. It runs your code, measures the time it takes to run, and then performs some statistical analysis on those measurements.

The library is header-only so you don’t have to build it. It comes as a single header that you can drop somewhere and #include it in your code. Grab that header from the releases page.

Comments
  • Benchmark parameters I

    Benchmark parameters I

    Hi!

    I managed to get started with the feature. There is work to be done, but I open the pull request in case you want to do an early review so I can change my approach if needed. Note that this branch depends on the changes in #54

    Here's the progress:

    • [x] Benchmark writers can declare one or more parameters of any streamable type.
    • [x] Benchmark parameters can be set to a value from the command line.
    • [x] The benchmark runner can be told from the command line try a whole range (linear or exponential) of range values for one (and only one) parameter.
    • [x] Improve the handling of parameter types.
    • [x] Testing (I know, one should do that first, but the areas of the code I needed to touch did not have any tests yet so I just hacked my way through.)
    • [x] Update the HTML reporter to show a graph with the different means and variances for the test run with different parameters.

    Nice to have that I might not do:

    • [ ] Update documentation mentioning this feature.
    • [ ] Do something special for other reporters when trying a parameter range.
    • [ ] Improve parameter parsing on the command line so one can escape the separator : in string parameters.
    • [ ] Add runs through ranges of values over multiple parameters, using support for 3D and/or contour plots from Plotly.
    • [ ] Explicitly defining benchmark suites (capturing parameters at suite level for brevity)

    RFC @rmartinho

    feature runner interface 
    opened by arximboldi 10
  • Don't define main() unless NONIUS_RUNNER is defined

    Don't define main() unless NONIUS_RUNNER is defined

    This allows a custom main() function to call nonius::main(), which is useful if some initialization code needs to run before the benchmarks. Previously, main.h++ couldn't be included because main() would then be defined twice.

    enhancement interface 
    opened by ryb-ableton 9
  • How to ignore destruction of object?

    How to ignore destruction of object?

    I want to ignore in the measurement the destruction of an object. I know the special objects: nonius::storage_for & nonius::destructable_object

    But I think it will not help me, I have following case:

    MyObject obj = method_generate_myobject()
    

    The method method_generate_myobject() will generate an instance of MyObject with automatica storage. The method has access to a private ctor which I cannot invoke directly. How can I ignore with nonius the destruction of MyObject during the benchmark run?

    enhancement wontfix 
    opened by acki-m 9
  • example compilation fail with g++ 5

    example compilation fail with g++ 5

    On ubuntu 16:

    cd examples g++ -Wall -std=c++11 example1.cpp -I../include/nonius/ -I../include/ -o example1 -pthread /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crt1.o: In function _start': (.text+0x20): undefined reference tomain' collect2: error: ld returned 1 exit status

    Adding #include <nonius/main.h++> to the file include/nonius.h++ does fix that but well....

    duplicate invalid docs 
    opened by WilliamTambellini 6
  • Benchmark parameters II (API improvements)

    Benchmark parameters II (API improvements)

    This pull request tries to improve the parameters API. It is a continuation of #56. Following up on the discussion on that pull-request, I try here to make the parameters API more terse and robust. The last commit includes more details.

    This branch also has a commit addressing #10. It could be extracted, but I just happened to do it in this branch.

    enhancement interface 
    opened by arximboldi 6
  • Conan package for Nonius

    Conan package for Nonius

    I've created a Conan-package for Nonius here. It works for me on Mac OS X, but I have no way of testing it on Windows.

    Also, in order to configure the Boost dependency better, it would be useful to know for which versions of Visual Studio specifically it is required.

    I'm happy for any feedback. If you have no objections, once it's confirmed working on Windows, I'd publish it in the Conan central repository.

    feature build 
    opened by bjoernpollex 6
  • Replace Highcharts with Plotly

    Replace Highcharts with Plotly

    Plotly (https://plot.ly/javascript/) does not have the licensing problems discussed in #53. Furthermore, it has more features and plot styles, which will be useful later, like in #36.

    In this branch I tried to keep the styling similar to the one with Highlights. I made small changes, that I consider improvements, like making the plot adapt automatically to the window size -- including on resize.

    An example of the new look of the reports can be seen here:

    https://sinusoid.es/misc/nonius-example2-plotly.html

    enhancement interface 
    opened by arximboldi 4
  • undefined reference to `main'

    undefined reference to `main'

    Hey,

    I've tried to compile the examples but I'm getting undefined reference to main git log shows : 78df23a7ed94a7311dfa10db531fe2a4c810f479

    g++ -o example2 example2.c++ --std=c++14 -I/home/rsoeldner/custom/nonius/include -I/home/rsoeldner/custom/nonius/include/nonius -lpthread
    /usr/lib/../lib64/crt1.o: In function `_start':
    /home/abuild/rpmbuild/BUILD/glibc-2.19/csu/../sysdeps/x86_64/start.S:118: undefined reference to `main'
    collect2: error: ld returned 1 exit status
    
    
    
    bug duplicate invalid 
    opened by ghost 3
  • filter benchmarks to execute at runtime

    filter benchmarks to execute at runtime

    I've added the ability to filter which benchmarks you want to run as a command line option.

    the --filter (and -f) option allows you to define a regex pattern that benchmark names must match to be executed. The default pattern is ".*" which will run all tests.

    feature runner 
    opened by S4H 3
  • fix include for examples that do not provide main

    fix include for examples that do not provide main

    examples where not linking due to missing main since they probably should only demonstrate the most simple solution, it seems good to use the provided main routine.

    invalid 
    opened by marcmo 3
  • Remove boost::variant

    Remove boost::variant

    Replace's boost::variant with a std::function removing the dependency on Boost.Variant.

    Caveat: using a std::shared_ptr is not the same as using an std::unique_ptr but the overhead is on construction only. Since std::function requires CopyConstructible, this is the simplest thing I found that works.

    I also updated the header of some examples again to include nonius_single.h++ since they were missing the main function.

    enhancement invalid 
    opened by gnzlbg 3
  • Broken Link makes it hard to find single-header version

    Broken Link makes it hard to find single-header version

    If you follow the releases link in the readme at https://github.com/libnonius/nonius/ it leads to https://github.com/rmartinho/nonius/releases . All of the releases there are present as full source only with no single-header versions. The correct link that has releases with single-header versions is https://github.com/libnonius/nonius/releases

    I think this is causing some confusion among potential users, as evidenced by other bug reports.

    opened by granthusbands 0
  • disable boost

    disable boost

    is it possible to have a feature to disable boost with a macro?

    #include <boost/math/distributions/normal.hpp>

    for a header only library , having a boost dependency seems counter intuitive

    opened by pedro-vicente 0
  • 2 header locations are needed ?

    2 header locations are needed ?

    to include in a cmake project, it seems 2 header locations are needed ?

    include_directories(nonius-1.1.2/include/nonius)
    include_directories(nonius-1.1.2/include)
    
    opened by pedro-vicente 1
  • Fix bad data on html report summary page

    Fix bad data on html report summary page

    The HTML report's Javascript incorrectly assumes that the benchmarks in each run are sorted in the same order. That's not the case (maybe it should be?). Instead, you can often have something like this:

    var data = { runs: [
        { benchmarks: [ { name: 'a', mean: 1 }, { name: 'b', mean: 2 } ] },
        { benchmarks: [ { name: 'b', mean: 3 }, { name: 'a', mean: 4 } ] },
    ]};
    

    In the above example, the summary plot would assume that the first benchmark is named 'a', and it would draw a line for series 'a' connecting points [1, 3]. It should instead draw a line connecting points [1, 4].

    This change updates the javascript to search all the runs in the benchmark for one with the target name.

    Fixes #108.

    opened by jtippet 0
  • HTML report shows wrong data with multiple param values

    HTML report shows wrong data with multiple param values

    Create a param named X and run a test with multiple param values, like benchmark --reporter=html --param=X:+:0:1:10. The HTML report will show a nice summary page that shows a line plot of how the benchmarks fare on each param value.

    Unfortunately, the data is just wrong. The data is correct only for the first (leftmost) column; all other columns have the data mixed up between the various series.

    Here's an example: image

    Note how the orange and blue series seem to trade off on looking bad. That's not backed up by the raw data -- in the raw data, the orange series should always be bad, and the blue should always be okay.

    A corrected graph would look like this: image

    opened by jtippet 0
Releases(v1.2.0-beta.1)
  • v1.2.0-beta.1(Sep 1, 2016)

    This is a beta for a feature release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release includes several new features:

    • #26 you can now call nonius::main from your main function, allowing you to do any sort of setup you want directly;
    • #54 a better charts library is used for HTML reports;
    • #56, #63 benchmarks can now be run with varying parameters;
    • #57 there are two basic primitives to control the optimizer;
    Source code(tar.gz)
    Source code(zip)
    nonius-1.2.0-beta.1.zip(295.55 KB)
  • v1.1.2(Oct 14, 2015)

    This is a patch release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release updates the HighCharts JS library and reduces some of the third-party dependencies.

    Source code(tar.gz)
    Source code(zip)
    nonius-1.1.2.zip(192.71 KB)
  • v1.1.1(Jun 5, 2015)

    This is a patch release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release removes a warning generated when compiling with 64-bit MSVC.

    Source code(tar.gz)
    Source code(zip)
    nonius-1.1.1.zip(158.73 KB)
  • v1.1.0(Jun 4, 2015)

    This is a feature release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release includes two new features:

    • the usage of Boost.Chrono can be forced by defining a macro, even in environments that don't need it;
    • all of the non-default reporters can be disabled together or individually, by defining specific macros.

    This release also includes several bug fixes.

    Source code(tar.gz)
    Source code(zip)
    nonius-1.1.0.zip(158.36 KB)
  • v1.0.0(Mar 28, 2015)

    This is the first production release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release includes the following features.

    • simple declarative interface for specifying benchmarks;
    • support for testing constructors and destructors;
    • textual, CSV, JUnit, and interactive chart reports;
    • mean and variance estimation with statistical bootstrapping;
    • outlier analysis.
    Source code(tar.gz)
    Source code(zip)
    nonius-1.0.0.zip(159.71 KB)
  • v0.3(Feb 15, 2015)

  • v0.2(Feb 17, 2014)

    Improving upon the previous release, this provides a simpler way of defining benchmarks and features a default runner with various features. It also provides CSV output, and makes MSVC a supported target.

    Coming soon: charts, HTML report, and more data analysis.

    Source code(tar.gz)
    Source code(zip)
    libnonius-v0.2.zip(30.39 KB)
  • v0.1(Jan 25, 2014)

    This is the first beta release of Nonius.

    Right now it's a very rough port of basic Criterion functionality. It can run benchmarks with the right granularity to minimise clock resolution artifacts (so you can measure very small amounts of code), and perform a bootstrap analysis on the measurements. It displays this information in a manner very similar to Criterion's normal textual output.

    Next releases will include CSV exports, graphs, and perhaps other analyses of the data.

    Source code(tar.gz)
    Source code(zip)
    libnonius-v0.1-Linux-amd64-r22.zip(100.59 KB)
Owner
Nonius
A C++ micro-benchmarking framework
Nonius
A testing micro framework for creating function test doubles

Fake Function Framework (fff) A Fake Function Framework for C Hello Fake World! Capturing Arguments Return Values Resetting a Fake Call History Defaul

Mike Long 535 Nov 16, 2022
UT: C++20 μ(micro)/Unit Testing Framework

"If you liked it then you "should have put a"_test on it", Beyonce rule [Boost::ext].UT / μt | Motivation | Quick Start | Overview | Tutorial | Exampl

boost::ext 929 Nov 26, 2022
A unit testing framework for C

Check Table of Contents About Installing Linking Packaging About Check is a unit testing framework for C. It features a simple interface for defining

null 923 Nov 27, 2022
The fastest feature-rich C++11/14/17/20 single-header testing framework

master branch Windows All dev branch Windows All doctest is a new C++ testing framework but is by far the fastest both in compile times (by orders of

Viktor Kirilov 4.4k Nov 23, 2022
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)

Catch2 v3 is being developed! You are on the devel branch, where the next major version, v3, of Catch2 is being developed. As it is a significant rewo

Catch Org 15.9k Nov 25, 2022
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)

Catch2 v3 is being developed! You are on the devel branch, where the next major version, v3, of Catch2 is being developed. As it is a significant rewo

Catch Org 15.9k Nov 24, 2022
C++ Benchmark Authoring Library/Framework

Celero C++ Benchmarking Library Copyright 2017-2019 John Farrier Apache 2.0 License Community Support A Special Thanks to the following corporations f

John Farrier 723 Nov 24, 2022
CppUTest unit testing and mocking framework for C/C++

CppUTest CppUTest unit testing and mocking framework for C/C++ More information on the project page Slack channel: Join if link not expired Getting St

CppUTest 1.1k Nov 26, 2022
Googletest - Google Testing and Mocking Framework

GoogleTest OSS Builds Status Announcements Release 1.10.x Release 1.10.x is now available. Coming Soon Post 1.10.x googletest will follow Abseil Live

Google 28.5k Nov 25, 2022
Minimal unit testing framework for C

MinUnit Minunit is a minimal unit testing framework for C/C++ self-contained in a single header file. It provides a way to define and configure test s

David Siñuela Pastor 451 Nov 18, 2022
A lightweight unit testing framework for C++

Maintenance of UnitTest++, recently sporadic, is officially on hiatus until 26 November 2020. Subscribe to https://github.com/unittest-cpp/unittest-cp

UnitTest++ 510 Nov 16, 2022
🧪 single header unit testing framework for C and C++

?? utest.h A simple one header solution to unit testing for C/C++. Usage Just #include "utest.h" in your code! The current supported platforms are Lin

Neil Henning 541 Nov 17, 2022
test framework

Photesthesis This is a small, experimental parameterized-testing tool. It is intended to be used in concert with another unit-testing framework (eg. C

Graydon Hoare 11 Jun 2, 2021
A simple framework for compile-time benchmarks

Metabench A simple framework for compile-time microbenchmarks Overview Metabench is a single, self-contained CMake module making it easy to create com

Louis Dionne 162 Oct 23, 2022
A C++ micro-benchmarking framework

Nonius What is nonius? Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a sim

Nonius 337 Nov 19, 2022
A C++ micro-benchmarking framework

Nonius What is nonius? Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a sim

R. Martinho Fernandes 92 Nov 6, 2022
C++ benchmarking framework

hayai - the C++ benchmarking framework. hayai is a C++ framework for writing benchmarks for pieces of code along the lines of the normal approach for

Nick Bruun 362 Oct 17, 2022
A simple C++ 03/11/etc timer class for ~microsecond-precision cross-platform benchmarking. The implementation is as limited and as simple as possible to create the lowest amount of overhead.

plf_nanotimer A simple C++ 03/11/etc timer class for ~microsecond-precision cross-platform benchmarking. The implementation is as limited and simple a

Matt Bentley 100 Nov 16, 2022
Benchmarking algebraic effect handler implementations

Benchmarking Effect handlers. Currently supported libraries and languages are: kk (extension .kk), Koka. ml (extension .ml), multi-core OCaml.

Daan 21 Dec 18, 2021
Benchmarking a trivial replacement for std::vector

std::vector replacement benchmark Dependencies You'll need gnuplot and bash to run ./bench.sh. In addition to that, you'll need to have gcc and clang

Dale Weiler 9 Aug 27, 2022