C++ benchmarking framework

Related tags

Benchmarking hayai
Overview

hayai - the C++ benchmarking framework.

Build Status

hayai is a C++ framework for writing benchmarks for pieces of code along the lines of the normal approach for unit testing using frameworks such as googletest. For information on the origin of and how to use hayai, please read the introductory blog post until I have the time to write a proper README.

Installing hayai

hayai releases are made available for Mac OS X in Homebrew and for Ubuntu LTS releases in an Ubuntu PPA.

To install from Homebrew on Mac OS X:

$ brew install hayai

To install from the Ubuntu PPA:

$ apt-add-repository ppa:bruun/hayai
$ apt-get update
$ apt-get install libhayai-dev

Building hayai

hayai is fully header based, but to use the simple main function as described in the introducty blog post, the source code must be compiled to provide libhayai_main. Compiling the source code requires CMake to be present on the system:

$ cd hayai
$ cmake .
$ make

This will also build the sample available in the sample/ directory of the repository.

Developing hayai

hayai includes a (currently not at all very comprehensive) test suite for development. To use the test suite, pass a boolean true value as the BUILD_HAYAI_TESTS CMake variable:

$ cd hayai
$ cmake -DBUILD_HAYAI_TESTS=true .
$ make
$ make test
Comments
  • Fix #56: Default to highest available Clock Resolution on POSIX

    Fix #56: Default to highest available Clock Resolution on POSIX

    • fixes issue #56

    • #define _POSIX_TIMERS on POSIX systems resides in posix_opt.h, which is included by unistd.h

    • including unistd.h before hayai.hpp in hayai_main.hpp allows thecorrespoinding libhayai_main.a to built with clock implementation clock_gettime(CLOCK_MONOTONIC_RAW) in time.h instead of gettimeofday() in sys/time.h which, should be a last resort in the case that high resolution clock implementations don't exist.

    opened by conz27 11
  • Several hayai enhancements

    Several hayai enhancements

    • Parameterized benchmark support.
    • More clear output highlighting.
    • Basic support for run-time test filtering rules with "include" logic (by name).
    opened by nevkontakte 7
  • First stab at hayai-config.cmake setup

    First stab at hayai-config.cmake setup

    This is attempt to provide CMake find_package config mode for hayai project The configuration is based on the official tutorials: http://www.cmake.org/Wiki/CMake/Tutorials/How_to_create_a_ProjectConfig.cmake_file http://www.cmake.org/Wiki/CMake/Tutorials/Exporting_and_Importing_Targets

    There is one little details that may need to be improved: either move public headers to ${PROJECT_SOURCE_DIR}include/hayai or tweak ${PROJECT_SOURCE_DIR}/src in CONF_INCLUDE_DIRS variable, so the include paths are set properly while importing targets directly from the build directory (a nice CMake feature, by the way).

    Generally, I think it would be good to install and use buried headers in hayai sources as well as in client programs, like this:

    #include <hayai/hayai.hpp>
    
    opened by mloskot 6
  • Overflow

    Overflow

    It would be nice if you could detect overflows and you are doing your calculations, I ended up having negative value and even very small ones in cases where the test lasts for a while (on low performance hardware). I suggest starting by using uint64_t instead of signed values, and detect overflows when computing the average times. I also think you should keep us and s seperated.

    opened by Lectem 5
  • XML support

    XML support

    This PR adds XML support fo hayai with libxml.

    Additional CMake routines are added.

    Valgrind report:

    ==18702== HEAP SUMMARY:
    ==18702==     in use at exit: 0 bytes in 0 blocks
    ==18702==   total heap usage: 1,276 allocs, 1,276 frees, 153,116 bytes allocated
    
    opened by lisitsyn 4
  • Release hayai-v1.0.2 displays version number as 1.0.1

    Release hayai-v1.0.2 displays version number as 1.0.1

    [email protected]******:~/hayai-1.0.2$ ./build/sample/sample --help
    Usage: sample [OPTIONS]
    
      Runs the benchmarks for this project.
    
    Benchmark selection options:
      -l, --list
        List the names of all benchmarks instead of running them.
      -f, --filter <pattern>
        Run only the tests whose name matches one of the positive patterns but
        none of the negative patterns. '?' matches any single character; '*'
        matches any substring; ':' separates two patterns.
    Benchmark execution options:
      -s, --shuffle
        Randomize benchmark execution order.
    
    Benchmark output options:
      -o, --output <format>[:<path>]
        Output results in a specific format. If no path is specified, the output
        will be presented on stdout. Can be specified multiple times to get output
        in different formats. The supported formats are:
    
        console
          Standard console output.
        json
          JSON.
        junit
          JUnit-compatible XML (very restrictive.)
    
        If multiple output formats are provided without a path, only the last
        provided format will be output to stdout.
      --c, --color (yes|no)
        Enable colored output when available. Default yes.
    
    Miscellaneous options:
      -?, -h, --help
        Show this help information.
    
    hayai version: 1.0.1
    Clock implementation: gettimeofday
    

    Steps to Reproduce:

    1. wget https://github.com/nickbruun/hayai/archive/v1.0.2.tar.gz
    2. cd hayai-1.0.2
    3. cmake -H. -Bbuild
    4. make -C build
    5. ./build/sample/sample --help

    Expected hayai version: 1.0.2 Actual hayai version: 1.0.1

    opened by conz27 3
  • Fixed the tests with C++11 or newer, and extended the travis config

    Fixed the tests with C++11 or newer, and extended the travis config

    Currently the build with clang, std=c++17 and libcxx fails, which is addressed by my other PR.

    I also disabled two warnings in these builds, one is the c++11+ specific null-nullptr warning, the other is about marco names strarting with two underscores.

    opened by dutow 3
  • Move the logic for the default main executable into a Runner class.

    Move the logic for the default main executable into a Runner class.

    Allows those who need to use a secondary flag parser to do something along the lines of:

    #include <hayai/hayai_main.hpp>
    
    
    int main(int argc, char** argv)
    {
        // Set up the main runtime.
        ::hayai::MainRunner runner;
    
        // Parse the arguments.
        std::vector<char*> residualArgs;
        int result = runner.ParseArgs(argc, argv, &residualArgs);
        if (result)
        {
            return result;
        }
    
        // Parse the residual arguments using some other kind of parser.
        // [..]
    
        // Execute based on the selected mode.
        return runner.Run();
    }
    

    I'm unsure of the naming of Runner.

    opened by nickbruun 3
  • Compatitble with other flags parser

    Compatitble with other flags parser

    At first, Thanks for your great project.

    In my work situation, I use hayai and something like gflags in our project. and When I run our benchmark, for example:

    ./projectx --verbose=true --color no
    

    hayai will parse --verbose=true and exit with error. because --verbose is defined in our flags, not for hayai.

    And in the meantime, Our parser will parse --color no will exit with error too.

    So I suggest each flags parser should parse and remove the flags for them. So those flags not for them can be pass to next flagparser. So each parser will work fine together.

    Have a nice day!

    opened by yanyiwu 3
  • Inclusion filters

    Inclusion filters

    Is there an example of the inclusion filters? I'd love an automatic --hayai_filter=... commandline option feature, which is parsed by a standard provided hayai::InitHayai(&argc, argv) function, much like the googletest filters.

    enhancement 
    opened by ngaloppo 3
  • Windows compatibility - New Unix high-res clocks - Fix some header guards - Fix IterationTimeAverage display

    Windows compatibility - New Unix high-res clocks - Fix some header guards - Fix IterationTimeAverage display

    • Windows compatiblity, clock and display
    • New Unix clocks (HP-UX, Solaris, ...)
    • Fix some header guards
    • Fix iteration time average display
    • Add rt library for the compilation of the sample program
    • Add new sample delivery_man_sleep
    opened by nicopauss 3
  • Specifying number of runs/iterations at runtime

    Specifying number of runs/iterations at runtime

    The only way I was able to do it is by hacking and removing const here: https://github.com/nickbruun/hayai/blob/9bdca1c693e950ab2cb17c4ef885792c3c530234/src/hayai_benchmarker.hpp#L287 Otherwise, those numbers are set at startup time and can't be changed later. Or am I missing something?

    opened by aleks-f 1
  • fixture: SetUpBefore()/TearDownAfter()

    fixture: SetUpBefore()/TearDownAfter()

    Some of my benchmarks need to load a huge amount of data and it takes very long time. Current SetUp() and even the constructor are executed before every run which makes those tests painfuly long.

    The downside of this patch is that it is not fully backward compatible because it constructs the test instance only once for each BENCHMARK section.

    I'd like to know your thoughts.

    opened by dsiroky 3
  • How to Stop/Fail a Benchmark if fixture SetUp() fails?

    How to Stop/Fail a Benchmark if fixture SetUp() fails?

    Hi,

    It seems there's not a way to stop/fail a benchmark whenever it is not possible to continue during SetUp().

    class TestFixture : public ::hayai::Fixture
    {
     public:
        virtual void SetUp() {
            test = new Test();
            if (!test->canContinue()) {
                // SHOULD STOP HERE!
            }
        }
        virtual void TearDown() {
            delete(test);
        }
        Test *test;
    };
    

    BENCHMARK_F()'s connected to the fixture above still continues to execute.

    Cheers!

    opened by roniemartinez 3
  • Added HTML Outputter

    Added HTML Outputter

    To keep things simple, this outputter class just puts the JsonOutputter output into a string variable and Javascript handles the rest. Currently Javascript generates an interactive box and whisker plot of the data with plotly. By hovering over the elements you can get all the data your would normally see from console output (and then some), so I did not deem it necessary to add a text based output to the webpage.

    Closes #21

    opened by scribblemaniac 6
  • Fixture setup with arguments

    Fixture setup with arguments

    I was trying to create a fixture so that the size of the input of the test would be different among benchmarks, but with the same fixture.

    Hawai doesn't do well with template fixtures, I expected as much. However I think it would be interesting to give SetUp, TearDown etc a void* parameter so that we can configure those steps per benchmark.

    enhancement 
    opened by Lectem 6
Owner
Nick Bruun
Former CTO of @iconfinder.
Nick Bruun
C++ Benchmark Authoring Library/Framework

Celero C++ Benchmarking Library Copyright 2017-2021 John Farrier Apache 2.0 License Community Support A Special Thanks to the following corporations f

John Farrier 725 Dec 30, 2022
C++ benchmarking framework

hayai - the C++ benchmarking framework. hayai is a C++ framework for writing benchmarks for pieces of code along the lines of the normal approach for

Nick Bruun 361 Dec 22, 2022
A C++ micro-benchmarking framework

Nonius What is nonius? Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a sim

Nonius 339 Dec 19, 2022
A C++ micro-benchmarking framework

Nonius What is nonius? Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a sim

R. Martinho Fernandes 92 Dec 19, 2022
A simple C++ 03/11/etc timer class for ~microsecond-precision cross-platform benchmarking. The implementation is as limited and as simple as possible to create the lowest amount of overhead.

plf_nanotimer A simple C++ 03/11/etc timer class for ~microsecond-precision cross-platform benchmarking. The implementation is as limited and simple a

Matt Bentley 102 Dec 4, 2022
Benchmarking algebraic effect handler implementations

Benchmarking Effect handlers. Currently supported libraries and languages are: kk (extension .kk), Koka. ml (extension .ml), multi-core OCaml.

Daan 21 Dec 18, 2021
Benchmarking a trivial replacement for std::vector

std::vector replacement benchmark Dependencies You'll need gnuplot and bash to run ./bench.sh. In addition to that, you'll need to have gcc and clang

Dale Weiler 9 Aug 27, 2022
Colorful Logging is a simple and efficient library allowing for logging and benchmarking.

Colorful-Logging "Colorful Logging" is a library allowing for simple and efficient logging as well for benchmarking. What can you use it for? -Obvious

Mateusz Antkiewicz 1 Feb 17, 2022
CPU Performance Evaluation and Execution Time Prediction Using Narrow Spectrum Benchmarking

This is a simple implementation of Saavedra-Barrera's paper SAAVEDRA-BARRERA R H. CPU Performance Evaluation and Execution Time Prediction Using Narrow Spectrum Benchmarking[D/OL]. UCB/CSD92-684. EECS Department, University of California, Berkeley, 1992.

null 9 Jan 27, 2022
Alloc-test - Cross-platform benchmarking for memory allocators, aiming to be as close to real world as it is practical

Alloc-test - Cross-platform benchmarking for memory allocators, aiming to be as close to real world as it is practical

null 37 Aug 23, 2022
Mimalloc-bench - Suite for benchmarking malloc implementations.

Mimalloc-bench Suite for benchmarking malloc implementations, originally developed for benchmarking mimalloc. Collection of various benchmarks from th

Daan 186 Dec 24, 2022
Kigs framework is a C++ modular multipurpose cross platform framework.

Kigs framework is a C++ modular multi-purpose cross-platform framework. It was used as a basis for many professionnal projects. The main goal was to b

null 74 Nov 28, 2022
A C++ Web Framework built on top of Qt, using the simple approach of Catalyst (Perl) framework.

Cutelyst - The Qt Web Framework A Web Framework built on top of Qt, using the simple and elegant approach of Catalyst (Perl) framework. Qt's meta obje

Cutelyst 809 Dec 19, 2022
TreeFrog Framework : High-speed C++ MVC Framework for Web Application

Small but Powerful and Efficient TreeFrog Framework is a high-speed and full-stack web application framework based on C++ and Qt, which supports HTTP

TreeFrog Framework 1.1k Dec 22, 2022
Cute Framework (CF for short) is the cutest framework available for making 2D games in C/C++

Cute Framework (CF for short) is the cutest framework available for making 2D games in C/C++. CF comprises of different features, where the various features avoid inter-dependencies. In this way using CF is about picking and choosing which pieces are needed for your game

null 314 Dec 26, 2022
TengineGst is a streaming media analytics framework, based on GStreamer multimedia framework, for creating varied complex media analytics pipelines.

TengineGst is a streaming media analytics framework, based on GStreamer multimedia framework, for creating varied complex media analytics pipelines. It ensures pipeline interoperability and provides optimized media, and inference operations using Tengine Toolkit Inference Engine backend, across varied architecture - CPU, iGPU and VPU.

OAID 69 Dec 17, 2022
Framework for Enterprise Application Development in c++, HTTP1/HTTP2/HTTP3 compliant, Supports multiple server backends

The ffead-cpp Framework ffead-cpp is a web-framework, application framework, utilities all bundled into one. It also provides an embedded HTTP/Web-Soc

Sumeet Chhetri 540 Dec 19, 2022
JUCE is an open-source cross-platform C++ application framework for desktop and mobile applications, including VST, VST3, AU, AUv3, RTAS and AAX audio plug-ins.

JUCE is an open-source cross-platform C++ application framework used for rapidly developing high quality desktop and mobile applications, including VS

JUCE 4.7k Jan 1, 2023
An eventing framework for building high performance and high scalability systems in C.

NOTE: THIS PROJECT HAS BEEN DEPRECATED AND IS NO LONGER ACTIVELY MAINTAINED As of 2019-03-08, this project will no longer be maintained and will be ar

Facebook Archive 1.7k Dec 14, 2022
An Open Source Machine Learning Framework for Everyone

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

null 170.2k Jan 7, 2023