Reckless logging. Low-latency, high-throughput, asynchronous logging library for C++.

Overview

Performance chart for a quad-core CPU

Introduction

Reckless is an extremely low-latency, high-throughput logging library. It was created because I needed to perform extensive diagnostic logging without worrying about performance. Other logging libraries boast the ability to throw log messages away very quickly. Reckless boasts the ability to keep them all, without worrying about the performance impact. Filtering can and should wait until you want to read the log, or need to clean up disk space.

How it works

By low latency I mean that the time from calling the library and returning to the caller is as short as I could make it. The code generated at the call site consists only of pushing the arguments on a shared, lockless queue. In the non-contended case this has roughly the same cost as a making a function call. The actual message formatting and writing is performed asynchronously by a separate thread. This removes or hides several costs:

  • No transition to the kernel at the call site. The kernel is an easily overlooked but important cost, not only because the transition costs time, but because it pollutes the CPU cache. In other words, avoiding this makes your non-logging code run faster than if you were using a library that has to enter the kernel to perform its work.
  • No locks need to be taken for synchronization between threads (unless the queue fills up; see the performance article for more information about the implications of this).
  • No text formatting needs to be performed before resuming the main task of the program.
  • It doesn't have to wait for the actual I/O operation to complete.
  • If there are bursts of log calls, multiple items on the queue can be batched into a single I/O operation, improving throughput without sacrificing write latency.

For a more detailed performance discussion and statistics, see the performance article.

What's the catch?

As all string formatting and I/O is done asynchronously and in a single thread, there are a few caveats you need to be aware of:

  • If you choose to pass log arguments by reference or pointer, then you must ensure that the referenced data remains valid at least until the log has been flushed or closed (unless you're only interested in logging the value of the pointer itself). The best option for dynamically allocated data is typically std::string, std::shared_ptr or std::unique_ptr.
  • You must take special care to handle crashes if you want to make sure that all log data prior to the crash is saved. This is not unique to asynchronous logging—for example fprintf will buffer data until you flush it—but asynchronous logging arguably makes the issue worse. The library provides convenience functions to aid with this.
  • As all string formatting is done in a single thread, it could theoretically limit the scalability of your application if formatting is expensive or your program generates a high volume of log entries in parallel.
  • Performance becomes somewhat less predictable and harder to measure. Rather than putting the cost of the logging on the thread that calls the logging library, the OS may suspend some other thread to make room for the logging thread to run.

Basic use

#include <reckless/severity_log.hpp>
#include <reckless/file_writer.hpp>

// It is possible to build custom loggers for various ways of formatting the
// log. The severity log is a stock policy-based logger that allows you to
// configure fields that should be put on each line, including a severity
// marker for debug/info/warning/error.
using log_t = reckless::severity_log<
    reckless::indent<4>,       // 4 spaces of indent
    ' ',                       // Field separator
    reckless::severity_field,  // Show severity marker (D/I/W/E) first
    reckless::timestamp_field  // Then timestamp field
    >;

reckless::file_writer writer("log.txt");
log_t g_log(&writer);

int main()
{
    std::string s("Hello World!");

    // You can use ordinary printf-style syntax, but unlike stdio this
    // is type-safe and extensible.
    g_log.debug("Pointer: %p", s.c_str());
    g_log.info("Info line: %s", s);

    for(int i=0; i!=4; ++i) {
        reckless::scoped_indent indent;  // The indent object causes the lines
        g_log.warn("Warning: %d", i);    // within this scope to be indented.
    }

    g_log.error("Error: %f", 3.14);

    return 0;
}

This would give the following output:

D 2019-03-09 12:53:54.280 Pointer: 0x7fff58378850
I 2019-03-09 12:53:54.280 Info line: Hello World!
W 2019-03-09 12:53:54.280     Warning: 0
W 2019-03-09 12:53:54.280     Warning: 1
W 2019-03-09 12:53:54.280     Warning: 2
W 2019-03-09 12:53:54.280     Warning: 3
E 2019-03-09 12:53:54.280 Error: 3.140000

Platforms

The library works on Windows and Linux. BSD is on the roadmap. I don't own any Apple computers, so OS X won't happen unless someone sends me a patch or buys me hardware.

Building

Alternative 1: Using Visual Studio

On Windows it is recommended to use Visual Studio for building the library. Simply open reckless.sln, choose "batch build" and "select all". Then press Build. The library files will be placed in the build subdirectory.

To build a program against the library you need to set your library path to point to the appropriate library build for your configuration, and set the include path to $(RECKLESS)/reckless/include, where RECKLESS, given that RECKLESS is a property that points to the reckless source directory.

Alternative 2: using Make

To build the library using GNU Make, clone the git repository and run make. This only works with GCC-compatible compilers.

To build a program against the library, given the variable RECKLESS pointing to the reckless root directory, use e.g.:

g++ -std=c++11 myprogram.cpp -I$(RECKLESS)/reckless/include -L$(RECKLESS)/reckless/lib -lreckless -lpthread

Alternative 3: using CMake

To build the library using CMake, clone the git repository and run the following commands:

mkdir build; cd build
cmake ..
make

To build a program against this library using CMake, add the following line to your program's CMakeLists.txt:

add_subdirectory(path/to/reckless)

Subsequently, to link this library to a program (e.g. your_executable), add the following to your program's CMakeLists.txt:

target_link_libraries(your_executable reckless pthread)

More information

For more details, see the manual.

Alternatives

Other logging libraries with similar, asynchronous design are

Issues
  • Probably problem with SSO for std::string.

    Probably problem with SSO for std::string.

    Code:

        std::string str("~~~~~~~~~~~~~~~");
    
        for (int i = 0; i != 1000000; ++i)
          g_log.info("Info line: %s @%d", str, i);
    

    Result in:

    munmap_chunk(): invalid pointer
    Aborted
    

    Randomly I saw also:

    munmap_chunk(): invalid size
    

    If str has more than 15 characters problem doesn't appear. @mattiasflodin Could you take a look at that? Probably it is similar to #35 (also short string).

    bug 
    opened by itwasntme-mk 6
  • Use canonical boost includes style

    Use canonical boost includes style

    Now I see some boost libs are located in boost/boost_1_56_0 and boost is included like #include <boost_1_56_0/lockfree/queue.hpp> https://github.com/mattiasflodin/reckless/blob/master/reckless/include/reckless/basic_log.hpp#L31 . But common practice is to include boost like #include <boost/lockfree/queue.hpp>.

    So, I suggest to move all boost stuff to boost/boost_1_56_0/boost, add include paths to boost/boost_1_56_0 and all includes will be just like #include <boost/lockfree/queue.hpp>.

    This purpose of this is to use other (system, newer) versions of boost rather than 1.56.0.

    opened by egorpugin 5
  • Corrupted double-linked list

    Corrupted double-linked list

    Hi Mattias,

    I build a benchmark to compare some Logger in Java, these are Log4J2, logback, spdlog, glog, g3log and reckless.

    What I have seen, that spdlog, reckless and g3log are nearly insane, but g3log and reckless have some issues.

    The issue I have is a corrupted double-linked list, my problem is, that I have no clou why and when this appears, but it appears sometimes.

    *** Error in `java': corrupted double-linked list: 0x00007f56c4046530 ***

    Therefore the results I gain from reckless are unusable, like g3log. But in terms of speed it is similar to spdlog.

    As I described, that I use Java I need to mention here, that I use JNI to load it into Java. This is how I create the logger for reckless, maybe it can be done better:

    using log_t = reckless::severity_log< reckless::indent<4>, // 4 spaces of indent ' ', // Field separator reckless::severity_field, // Show severity marker (D/I/W/E) first reckless::timestamp_field // Then timestamp field >;

    reckless::file_writer writer("logs/reckless_async.log"); log_t logger(&writer);

    Cheers Mader102

    wontfix 
    opened by mader102 4
  • Consistent crash with example from README

    Consistent crash with example from README

    The library consistently crashes when example from README is built with it. All you need to change is print the s string in the loop (in addition to integer i) and increase the counter:

        for(int i=0; i!=10000000; ++i) { // <- Increase the counter to crash (change 1 of 2)
            reckless::scoped_indent indent;
            g_log.warn("Warning: %s, %d", s, i);    // <- Print s string to crash (change 2 of 2)
        }
    
    • Verified on Ubuntu 18.04 / gcc 9.5.0, also with gcc 8.3.0
    • The crash usually occurs at random counter number.
    • The crash seems to occur sooner when stdout_writer is used, later with file_writer.
    • Crashing stops as soon as string s gets longer than 16 characters.
    duplicate 
    opened by Kiwonik 3
  • Crash while running Benchmark

    Crash while running Benchmark

    After running the benchmark as specified by

    https://github.com/Iyengar111/NanoLog/blob/master/nano_vs_spdlog_vs_g3log_vs_reckless.cpp

    The code is running successfully with -o0 and -o2 but crashes with -o3.

    I'm getting the following crash: crash

    I'm unable to solve the issue for a while.

    Please do let me know how to solve this.

    opened by AnkitAggarwalPEC 3
  • Add install target to cmake build

    Add install target to cmake build

    Try to add install target, but boost is a huge dependency. If replace its lockfree queue with something else we can get rid of it. Up to you what to do with this MR.

    enhancement 
    opened by ayum 3
  • Custom fields in JSON logger

    Custom fields in JSON logger

    Hi! Thank you for sharing this awesome library!

    I need to create a JSON logger that could accept custom JSON fields for context logging. So apart from the message, severity etc there would be other, custom fields.

    Is this possible with reckless?

    Thanks!

    question 
    opened by janekolszak 3
  • Conan package

    Conan package

    Hello, Do you know about Conan? Conan is modern dependency manager for C++. And will be great if your library will be available via package manager for other developers.

    Here you can find example, how you can create package for the library.

    If you have any questions, just ask :-)

    enhancement help wanted 
    opened by zamazan4ik 3
  • Error building a program against reckless

    Error building a program against reckless

    I was able to build the library with the make file, however when you try to compile the examples or include reckless in another project the following error is thrown:

    g++ -std=c++11 severity_log.cpp -I/home/n/git/cpp/reckless/reckless/include -L/home/n/git/cpp/reckless/reckless/lib -lreckless
    In file included from /home/n/git/cpp/reckless/reckless/include/reckless/policy_log.hpp:25:0,
                     from /home/n/git/cpp/reckless/reckless/include/reckless/severity_log.hpp:25,
                     from severity_log.cpp:22:
    /home/n/git/cpp/reckless/reckless/include/reckless/basic_log.hpp:31:43: fatal error: boost_1_56_0/lockfree/queue.hpp: No such file or directory
    compilation terminated.
    
    bug 
    opened by npezolano 3
  • [QUESTION] Claryfying the confusion in MPSC ring buffer initialization code

    [QUESTION] Claryfying the confusion in MPSC ring buffer initialization code

    Hello, Mattias!

    Thanks for your library, it is a great piece of work one could learn a lot from by reading the source code. I am studying the implementation of MPSC ring buffer now, and I am not sure I got everything right about mpsc_ring_buffer::init(std::size_t). Would you mind clarifying the confusion and also correct me if I got something wrong about the general idea of the following code?

    capacity = round_capacity(capacity);
    
    #if defined(__linux__)
        // allocate shared memory for buffer data
        int shm = shmget(IPC_PRIVATE, capacity, IPC_CREAT | S_IRUSR | S_IWUSR);
        if(shm == -1)
            throw std::bad_alloc();
    
        void* pbase = nullptr;
        while(!pbase) {
            // ensure we can acquire a set of consecutive pages twice as big as required capacity located
            // in process address space;
            pbase = mmap(nullptr, 2*capacity, PROT_NONE,
                    MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
            if(MAP_FAILED == pbase)
                throw std::bad_alloc();
            // unmap it - we are not going to use it anyway
            munmap(pbase, 2*capacity);
            // attach shared segment to the location we got from mmap()
            pbase = shmat(shm, pbase, 0);
            // THAT IS WHERE IT FEELS OFF
            // shmat() returns either a pointer to attachment address or ((void *) -1), 
            // but does not return NULL in any case
            // Let's say someone had a chance to occupy the memory location we got from mmap() before we get here
            // (freshly loaded shared library etc.) so shmat() returned ((void *) -1)
            // SHOULDN'T IT BE if (pbase != (void*) -1) ?
            if(pbase) {
                // if first shmat() call succeded , map the same shared segment to the second half of page set
                if((void*)-1 == shmat(shm, static_cast<char*>(pbase) + capacity, 0))
                {
                    shmdt(pbase);
                    pbase = nullptr;
                }
            }
        }
        // mark segment for removal
        shmctl(shm, IPC_RMID, nullptr);
    #elif defined(_WIN32)
    // ...
    #endif
    

    Thank you again.

    bug 
    opened by nvevg 2
  • error build with clang

    error build with clang

    we use cmake to build the lib

    cmake -DCMAKE_C_COMPILER=clang ../ 
    

    and run make

    it throws out

    clang: error: linker command failed with exit code 1 (use -v to see invocation)  
    
    invalid 
    opened by JsBlueCat 2
  • Thread Sanitizer reports multiple data races

    Thread Sanitizer reports multiple data races

    Hi Mattias!

    Thanks for the great library!

    Are you aware of the warnings that thread sanitizer reports? They're probably false positives (I hope :smile:) but nevertheless they would hide real warnings. Could you please have a look?

    Small example: reckless_tsan.cpp.txt

    Build and run like this:

    g++ reckless_tsan.cpp -O2 -fsanitize=thread -lreckless && ./a.out
    

    Getting these warnings: tsan_warnings.txt

    enhancement help wanted 
    opened by mopleen 3
  • Improvements to crash handler for unix

    Improvements to crash handler for unix

    The Unix version of the crash handler should be improved so that it invokes the previous signal handler after cleaning up, similarly to how the win32 version does it.

    enhancement 
    opened by mattiasflodin 0
Releases(v3.0.3)
  • v3.0.3(Apr 10, 2021)

    This release fixes a crash bug that occurred when outputting some non-POD objects (such as std::string in some versions of libstdc++), and makes building examples optional in the CMake build.

    Source code(tar.gz)
    Source code(zip)
  • v3.0.2(Jun 20, 2020)

    This release adds an example for using crash handler and makes sure it builds in CMake, which was missing the crash_handler files in its project file. It also includes an improvement to crash handling when multiple exception filters have been registered.

    Source code(tar.gz)
    Source code(zip)
  • v3.0.1(May 3, 2020)

  • v3.0.0(Apr 2, 2020)

    • Support for Windows operating systems
    • Entirely new and improved implementation of internal lockless queue, removing dependency on Boost
    • Updated benchmarks
    • New performance counters to detect buffer overflow or underuse (resolves issue #19)
    • No more use of TLS variables simplifies use in shared libraries / DLLs.
    • Fixes bug #4
    Source code(tar.gz)
    Source code(zip)
  • v2.2.1(Dec 31, 2018)

  • v2.2.0(Aug 31, 2016)

  • v2.1.0(May 22, 2016)

  • v2.0.0(May 3, 2016)

  • v1.0.0(May 15, 2015)

Owner
Mattias Flodin
Mattias Flodin
A Fast and Convenient C++ Logging Library for Low-latency or Real-time Environments

xtr What is it? XTR is a C++ logging library aimed at applications with low-latency or real-time requirements. The cost of log statements is minimised

null 10 Jul 17, 2022
fmtlog is a performant fmtlib-style logging library with latency in nanoseconds.

fmtlog fmtlog is a performant asynchronous logging library using fmt library format. Features Faster - lower runtime latency than NanoLog and higher t

Meng Rao 376 Aug 7, 2022
Colorful Logging is a simple and efficient library allowing for logging and benchmarking.

Colorful-Logging "Colorful Logging" is a library allowing for simple and efficient logging as well for benchmarking. What can you use it for? -Obvious

Mateusz Antkiewicz 1 Feb 17, 2022
Yet another logging library.

Blackhole - eating your logs with pleasure Blackhole is an attribute-based logger with strong focus on gaining maximum performance as possible for suc

Evgeny Safronov 186 Jun 10, 2022
A lightweight C++ logging library

Loguru: a lightweight and flexible C++ logging library. At a glance Documentation Documentation can be found at https://emilk.github.io/loguru/index.h

Emil Ernerfeldt 1.4k Aug 12, 2022
Portable, simple and extensible C++ logging library

Plog - portable, simple and extensible C++ logging library Pretty powerful logging library in about 1000 lines of code Introduction Hello log! Feature

Sergey Podobry 1.5k Aug 11, 2022
Fast C++ logging library.

spdlog Very fast, header-only/compiled, C++ logging library. Install Header only version Copy the source folder to your build tree and use a C++11 com

Gabi Melman 15.2k Aug 8, 2022
Cute Log is a C++ Library that competes to be a unique logging tool.

Cute Log Cute Log is a C++ Library that competes to be a unique logging tool. Version: 2 Installation Click "Code" on the main repo page (This one.).

null 4 Aug 12, 2021
Minimalistic logging library with threads and manual callstacks

Minimalistic logging library with threads and manual callstacks

Sergey Kosarevsky 21 Jun 24, 2022
logog is a portable C++ library to facilitate logging of real-time events in performance-oriented applications

logog is a portable C++ library to facilitate logging of real-time events in performance-oriented applications, such as games. It is especially appropriate for projects that have constrained memory and constrained CPU requirements.

John Byrd 46 Oct 21, 2020
Boost Logging library

Boost.Log, part of collection of the Boost C++ Libraries, provides tools for adding logging to libraries and applications. Directories build - Boost.L

Boost.org 151 Aug 4, 2022
C++ implementation of the Google logging module

Google Logging Library The Google Logging Library (glog) implements application-level logging. The library provides logging APIs based on C++-style st

Google 5.5k Aug 4, 2022
log4cplus is a simple to use C++ logging API providing thread-safe, flexible, and arbitrarily granular control over log management and configuration. It is modelled after the Java log4j API.

% log4cplus README Short Description log4cplus is a simple to use C++17 logging API providing thread--safe, flexible, and arbitrarily granular control

null 1.3k Jul 31, 2022
Uberlog - Cross platform multi-process C++ logging system

uberlog uberlog is a cross platform C++ logging system that is: Small Fast Robust Runs on Linux, Windows, OSX MIT License Small Two headers, and three

IMQS Software 13 May 18, 2022
An Ultra Low Power temperature logger based on the ESP8266 MCU.

Temperature logging IoT node Overview: The real node wired on a breadboard This is an ultra low power (ULP) temperature logging IoT node based on the

Radhi SGHAIER 11 Feb 22, 2022
Sagan - a multi-threads, high performance log analysis engine

Sagan - Sagan is a multi-threads, high performance log analysis engine. At it's core, Sagan similar to Suricata/Snort but with logs rather than network packets.

Quadrant Information Security 67 Jul 21, 2022