A C++ micro-benchmarking framework

Overview

Nonius

What is nonius?

Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a similar Haskell-based tool. It runs your code, measures the time it takes to run, and then performs some statistical analysis on those measurements.

The library is header-only so you don’t have to build it. It comes as a single header that you can drop somewhere and #include it in your code. Grab that header from the releases page.

Issues
  • Benchmark parameters I

    Benchmark parameters I

    Hi!

    I managed to get started with the feature. There is work to be done, but I open the pull request in case you want to do an early review so I can change my approach if needed. Note that this branch depends on the changes in #54

    Here's the progress:

    • [x] Benchmark writers can declare one or more parameters of any streamable type.
    • [x] Benchmark parameters can be set to a value from the command line.
    • [x] The benchmark runner can be told from the command line try a whole range (linear or exponential) of range values for one (and only one) parameter.
    • [x] Improve the handling of parameter types.
    • [x] Testing (I know, one should do that first, but the areas of the code I needed to touch did not have any tests yet so I just hacked my way through.)
    • [x] Update the HTML reporter to show a graph with the different means and variances for the test run with different parameters.

    Nice to have that I might not do:

    • [ ] Update documentation mentioning this feature.
    • [ ] Do something special for other reporters when trying a parameter range.
    • [ ] Improve parameter parsing on the command line so one can escape the separator : in string parameters.
    • [ ] Add runs through ranges of values over multiple parameters, using support for 3D and/or contour plots from Plotly.
    • [ ] Explicitly defining benchmark suites (capturing parameters at suite level for brevity)

    RFC @rmartinho

    feature runner interface 
    opened by arximboldi 10
  • Don't define main() unless NONIUS_RUNNER is defined

    Don't define main() unless NONIUS_RUNNER is defined

    This allows a custom main() function to call nonius::main(), which is useful if some initialization code needs to run before the benchmarks. Previously, main.h++ couldn't be included because main() would then be defined twice.

    enhancement interface 
    opened by ryb-ableton 9
  • How to ignore destruction of object?

    How to ignore destruction of object?

    I want to ignore in the measurement the destruction of an object. I know the special objects: nonius::storage_for & nonius::destructable_object

    But I think it will not help me, I have following case:

    MyObject obj = method_generate_myobject()
    

    The method method_generate_myobject() will generate an instance of MyObject with automatica storage. The method has access to a private ctor which I cannot invoke directly. How can I ignore with nonius the destruction of MyObject during the benchmark run?

    enhancement wontfix 
    opened by acki-m 9
  • example compilation fail with g++ 5

    example compilation fail with g++ 5

    On ubuntu 16:

    cd examples g++ -Wall -std=c++11 example1.cpp -I../include/nonius/ -I../include/ -o example1 -pthread /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crt1.o: In function _start': (.text+0x20): undefined reference tomain' collect2: error: ld returned 1 exit status

    Adding #include <nonius/main.h++> to the file include/nonius.h++ does fix that but well....

    duplicate invalid docs 
    opened by WilliamTambellini 6
  • Benchmark parameters II (API improvements)

    Benchmark parameters II (API improvements)

    This pull request tries to improve the parameters API. It is a continuation of #56. Following up on the discussion on that pull-request, I try here to make the parameters API more terse and robust. The last commit includes more details.

    This branch also has a commit addressing #10. It could be extracted, but I just happened to do it in this branch.

    enhancement interface 
    opened by arximboldi 6
  • Conan package for Nonius

    Conan package for Nonius

    I've created a Conan-package for Nonius here. It works for me on Mac OS X, but I have no way of testing it on Windows.

    Also, in order to configure the Boost dependency better, it would be useful to know for which versions of Visual Studio specifically it is required.

    I'm happy for any feedback. If you have no objections, once it's confirmed working on Windows, I'd publish it in the Conan central repository.

    feature build 
    opened by bjoernpollex 6
  • Replace Highcharts with Plotly

    Replace Highcharts with Plotly

    Plotly (https://plot.ly/javascript/) does not have the licensing problems discussed in #53. Furthermore, it has more features and plot styles, which will be useful later, like in #36.

    In this branch I tried to keep the styling similar to the one with Highlights. I made small changes, that I consider improvements, like making the plot adapt automatically to the window size -- including on resize.

    An example of the new look of the reports can be seen here:

    https://sinusoid.es/misc/nonius-example2-plotly.html

    enhancement interface 
    opened by arximboldi 4
  • undefined reference to `main'

    undefined reference to `main'

    Hey,

    I've tried to compile the examples but I'm getting undefined reference to main git log shows : 78df23a7ed94a7311dfa10db531fe2a4c810f479

    g++ -o example2 example2.c++ --std=c++14 -I/home/rsoeldner/custom/nonius/include -I/home/rsoeldner/custom/nonius/include/nonius -lpthread
    /usr/lib/../lib64/crt1.o: In function `_start':
    /home/abuild/rpmbuild/BUILD/glibc-2.19/csu/../sysdeps/x86_64/start.S:118: undefined reference to `main'
    collect2: error: ld returned 1 exit status
    
    
    
    bug duplicate invalid 
    opened by ghost 3
  • filter benchmarks to execute at runtime

    filter benchmarks to execute at runtime

    I've added the ability to filter which benchmarks you want to run as a command line option.

    the --filter (and -f) option allows you to define a regex pattern that benchmark names must match to be executed. The default pattern is ".*" which will run all tests.

    feature runner 
    opened by S4H 3
  • fix include for examples that do not provide main

    fix include for examples that do not provide main

    examples where not linking due to missing main since they probably should only demonstrate the most simple solution, it seems good to use the provided main routine.

    invalid 
    opened by marcmo 3
  • Remove boost::variant

    Remove boost::variant

    Replace's boost::variant with a std::function removing the dependency on Boost.Variant.

    Caveat: using a std::shared_ptr is not the same as using an std::unique_ptr but the overhead is on construction only. Since std::function requires CopyConstructible, this is the simplest thing I found that works.

    I also updated the header of some examples again to include nonius_single.h++ since they were missing the main function.

    enhancement invalid 
    opened by gnzlbg 3
  • Broken Link makes it hard to find single-header version

    Broken Link makes it hard to find single-header version

    If you follow the releases link in the readme at https://github.com/libnonius/nonius/ it leads to https://github.com/rmartinho/nonius/releases . All of the releases there are present as full source only with no single-header versions. The correct link that has releases with single-header versions is https://github.com/libnonius/nonius/releases

    I think this is causing some confusion among potential users, as evidenced by other bug reports.

    opened by granthusbands 0
  • disable boost

    disable boost

    is it possible to have a feature to disable boost with a macro?

    #include <boost/math/distributions/normal.hpp>

    for a header only library , having a boost dependency seems counter intuitive

    opened by pedro-vicente 0
  • 2 header locations are needed ?

    2 header locations are needed ?

    to include in a cmake project, it seems 2 header locations are needed ?

    include_directories(nonius-1.1.2/include/nonius)
    include_directories(nonius-1.1.2/include)
    
    opened by pedro-vicente 1
  • Fix bad data on html report summary page

    Fix bad data on html report summary page

    The HTML report's Javascript incorrectly assumes that the benchmarks in each run are sorted in the same order. That's not the case (maybe it should be?). Instead, you can often have something like this:

    var data = { runs: [
        { benchmarks: [ { name: 'a', mean: 1 }, { name: 'b', mean: 2 } ] },
        { benchmarks: [ { name: 'b', mean: 3 }, { name: 'a', mean: 4 } ] },
    ]};
    

    In the above example, the summary plot would assume that the first benchmark is named 'a', and it would draw a line for series 'a' connecting points [1, 3]. It should instead draw a line connecting points [1, 4].

    This change updates the javascript to search all the runs in the benchmark for one with the target name.

    Fixes #108.

    opened by jtippet 0
  • HTML report shows wrong data with multiple param values

    HTML report shows wrong data with multiple param values

    Create a param named X and run a test with multiple param values, like benchmark --reporter=html --param=X:+:0:1:10. The HTML report will show a nice summary page that shows a line plot of how the benchmarks fare on each param value.

    Unfortunately, the data is just wrong. The data is correct only for the first (leftmost) column; all other columns have the data mixed up between the various series.

    Here's an example: image

    Note how the orange and blue series seem to trade off on looking bad. That's not backed up by the raw data -- in the raw data, the orange series should always be bad, and the blue should always be okay.

    A corrected graph would look like this: image

    opened by jtippet 0
Releases(v1.2.0-beta.1)
  • v1.2.0-beta.1(Sep 1, 2016)

    This is a beta for a feature release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release includes several new features:

    • #26 you can now call nonius::main from your main function, allowing you to do any sort of setup you want directly;
    • #54 a better charts library is used for HTML reports;
    • #56, #63 benchmarks can now be run with varying parameters;
    • #57 there are two basic primitives to control the optimizer;
    Source code(tar.gz)
    Source code(zip)
    nonius-1.2.0-beta.1.zip(295.55 KB)
  • v1.1.2(Oct 14, 2015)

    This is a patch release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release updates the HighCharts JS library and reduces some of the third-party dependencies.

    Source code(tar.gz)
    Source code(zip)
    nonius-1.1.2.zip(192.71 KB)
  • v1.1.1(Jun 5, 2015)

    This is a patch release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release removes a warning generated when compiling with 64-bit MSVC.

    Source code(tar.gz)
    Source code(zip)
    nonius-1.1.1.zip(158.73 KB)
  • v1.1.0(Jun 4, 2015)

    This is a feature release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release includes two new features:

    • the usage of Boost.Chrono can be forced by defining a macro, even in environments that don't need it;
    • all of the non-default reporters can be disabled together or individually, by defining specific macros.

    This release also includes several bug fixes.

    Source code(tar.gz)
    Source code(zip)
    nonius-1.1.0.zip(158.36 KB)
  • v1.0.0(Mar 28, 2015)

    This is the first production release of nonius.

    Nonius is an open-source framework for benchmarking small snippets of C++ code, inspired by Criterion. Nonius runs your code, measures the time it takes to run, and then performs some statistical analyses on those measurements. Check out the documentation for more information.

    This release includes the following features.

    • simple declarative interface for specifying benchmarks;
    • support for testing constructors and destructors;
    • textual, CSV, JUnit, and interactive chart reports;
    • mean and variance estimation with statistical bootstrapping;
    • outlier analysis.
    Source code(tar.gz)
    Source code(zip)
    nonius-1.0.0.zip(159.71 KB)
  • v0.3(Feb 15, 2015)

  • v0.2(Feb 17, 2014)

    Improving upon the previous release, this provides a simpler way of defining benchmarks and features a default runner with various features. It also provides CSV output, and makes MSVC a supported target.

    Coming soon: charts, HTML report, and more data analysis.

    Source code(tar.gz)
    Source code(zip)
    libnonius-v0.2.zip(30.39 KB)
  • v0.1(Jan 25, 2014)

    This is the first beta release of Nonius.

    Right now it's a very rough port of basic Criterion functionality. It can run benchmarks with the right granularity to minimise clock resolution artifacts (so you can measure very small amounts of code), and perform a bootstrap analysis on the measurements. It displays this information in a manner very similar to Criterion's normal textual output.

    Next releases will include CSV exports, graphs, and perhaps other analyses of the data.

    Source code(tar.gz)
    Source code(zip)
    libnonius-v0.1-Linux-amd64-r22.zip(100.59 KB)
Owner
Nonius
A C++ micro-benchmarking framework
Nonius
C++ Benchmark Authoring Library/Framework

Celero C++ Benchmarking Library Copyright 2017-2021 John Farrier Apache 2.0 License Community Support A Special Thanks to the following corporations f

John Farrier 704 Jun 27, 2022
A C++ micro-benchmarking framework

Nonius What is nonius? Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a sim

Nonius 334 Jun 27, 2022
A C++ micro-benchmarking framework

Nonius What is nonius? Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a sim

R. Martinho Fernandes 90 Apr 13, 2022
C++ benchmarking framework

hayai - the C++ benchmarking framework. hayai is a C++ framework for writing benchmarks for pieces of code along the lines of the normal approach for

Nick Bruun 360 Jun 15, 2022
A simple C++ 03/11/etc timer class for ~microsecond-precision cross-platform benchmarking. The implementation is as limited and as simple as possible to create the lowest amount of overhead.

plf_nanotimer A simple C++ 03/11/etc timer class for ~microsecond-precision cross-platform benchmarking. The implementation is as limited and simple a

Matt Bentley 88 Jun 20, 2022
Benchmarking algebraic effect handler implementations

Benchmarking Effect handlers. Currently supported libraries and languages are: kk (extension .kk), Koka. ml (extension .ml), multi-core OCaml.

Daan 21 Dec 18, 2021
Benchmarking a trivial replacement for std::vector

std::vector replacement benchmark Dependencies You'll need gnuplot and bash to run ./bench.sh. In addition to that, you'll need to have gcc and clang

Dale Weiler 7 May 23, 2022
Colorful Logging is a simple and efficient library allowing for logging and benchmarking.

Colorful-Logging "Colorful Logging" is a library allowing for simple and efficient logging as well for benchmarking. What can you use it for? -Obvious

Mateusz Antkiewicz 1 Feb 17, 2022
CPU Performance Evaluation and Execution Time Prediction Using Narrow Spectrum Benchmarking

This is a simple implementation of Saavedra-Barrera's paper SAAVEDRA-BARRERA R H. CPU Performance Evaluation and Execution Time Prediction Using Narrow Spectrum Benchmarking[D/OL]. UCB/CSD92-684. EECS Department, University of California, Berkeley, 1992.

null 9 Jan 27, 2022
Alloc-test - Cross-platform benchmarking for memory allocators, aiming to be as close to real world as it is practical

Alloc-test - Cross-platform benchmarking for memory allocators, aiming to be as close to real world as it is practical

null 36 Mar 17, 2022
Mimalloc-bench - Suite for benchmarking malloc implementations.

Mimalloc-bench Suite for benchmarking malloc implementations, originally developed for benchmarking mimalloc. Collection of various benchmarks from th

Daan 154 Jun 25, 2022
A testing micro framework for creating function test doubles

Fake Function Framework (fff) A Fake Function Framework for C Hello Fake World! Capturing Arguments Return Values Resetting a Fake Call History Defaul

Mike Long 490 Jun 22, 2022
UT: C++20 μ(micro)/Unit Testing Framework

"If you liked it then you "should have put a"_test on it", Beyonce rule [Boost::ext].UT / μt | Motivation | Quick Start | Overview | Tutorial | Exampl

boost::ext 867 Jun 28, 2022
Crow is very fast and easy to use C++ micro web framework (inspired by Python Flask)

Crow is C++ microframework for web. (inspired by Python Flask) #include "crow.h" int main() { crow::SimpleApp app; CROW_ROUTE(app, "/")([]()

Jaeseung Ha 6.8k Jun 22, 2022
UT: C++20 μ(micro)/Unit Testing Framework

"If you liked it then you "should have put a"_test on it", Beyonce rule UT / μt | Motivation | Quick Start | Overview | Tutorial | Examples | User Gui

boost::ext 864 Jun 25, 2022
The c++ micro framework for building web applications based on workflow

wfrest The c++ micro framework for building web applications based on workflow ⌛️ Build Step 1 : install workflow git clne [email protected]:sogou/workfl

Shiyu Yi 374 Jun 21, 2022
A customizable hardware prefetching framework using online reinforcement learning as described in the MICRO 2021 paper by Bera and Kanellopoulos et al.

A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning Table of Contents What is Pythia? About the Framework Prerequisites

SAFARI Research Group at ETH Zurich and Carnegie Mellon University 37 Jun 17, 2022
Tau - A Micro (1k lines of code) Unit Test Framework for C/C++

Tau τ A Micro Unit Testing Framework for >C11/C++11 projects, with the promise of always being tiny - about 1k lines of code. This framework is a much

Jason Dsouza 93 Jun 5, 2022
Event loop friendly C++ actor micro-framework

Rotor rotor is event loop friendly C++ actor micro framework, github gitee features minimalistic loop agnostic core erlang-like hierarchical superviso

Ivan Baidakou 218 Jun 23, 2022
Raspberry Pi Pico (RP2040) and Micro-ROS (ROS 2) Integration

The Pico is an amazing microcontroller and I couldn't wait for ROS 2 support or Arduino Core, so here is my approach. Once the Arduino Core for RP2040 is out it will be easier to use micro_ros_arduino.

Darko Lukić 19 Jun 19, 2022
split89 keyboard - a 3d printed 89 key split TKL keyboard base powered by ATmega32U4 Pro Micro controllers with QMK Configurator support.

split89 keyboard - a 3d printed 89 key split TKL keyboard base powered by ATmega32U4 Pro Micro controllers with QMK Configurator support. This keyboar

null 42 May 10, 2022
Arduino Interface for cheap 2.4ghz RF enabled Solar Micro Inverters

NETSGPClient Arduino Interface for cheap 2.4ghz RF enabled Solar Micro Inverters using the so-called NETSGP protocol for communication. Here is a YouT

null 27 Jun 18, 2022
Timed reminder lights using Adafruit NeoPixels and Arduino Micro (3.3v ATMEGA 32U4)

SmartCoaster When I'm working, I get so focused that I forget to drink water. It's not that I'm not thirsty—oh, I am—but I'll get a glass of water, pu

Steven M. Caruso 17 Dec 21, 2021
Zerobug is a 3D-printed micro-servo hexapod robot powered by a Raspberry Pi Zero and a STM32F103

ZeroBug - DIY Hexapod Robot Zerobug is a 3D-printed micro-servo hexapod robot powered by a Raspberry Pi Zero and a STM32F103. The Pi Zero is in charge

Max K 136 Jun 19, 2022
Macro keypad and rotary input based on the ATmega32U4 Pro Micro board

JC-Pro-Macro Macro keypad and rotary input based on the ATmega32U4 Pro Micro Videos: Development: https://www.youtube.com/watch?v=g-XJLiv03rI Assembly

Jeremy S Cook 12 Jun 8, 2022
LAppS - Lua Application Server for micro-services with default communication over WebSockets. The fastest and most vertically scalable WebSockets server implementation ever. Low latency C++ <-> Lua stack roundtrip.

LAppS - Lua Application Server This is an attempt to provide very easy to use Lua Application Server working over WebSockets protocol (RFC 6455). LApp

null 47 Apr 25, 2022
Micro http server and client written in C++

httpp Micro http server and client written in C++ The motivation behind this little piece of code is to provide a really simple, yet efficient HTTP se

Thomas Sanchez 154 Jun 5, 2022
A simple example for 'Arduino' compatible boards to interface with I2C to the MPU-6050, a 6-axis micro-electromechanical IC

Arduino-MPU-6050 A simple example for 'Arduino' compatible boards to interface with I2C to the MPU-6050, a 6-axis micro-electromechanical IC ==About==

Ivan 1 Oct 19, 2021
A micro unit-testing library for C/C++

µ-test A micro unit testing framework for C/C++ to get you up and running with unit-testing ASAP (even without libc). Usage Simply include the C and h

Trevor McKay 1 Dec 8, 2021