Non bloated asynchronous logger

Overview

Minimal Asynchronous Logger (MAL)

A performant asynchronous data logger with acceptable feature-bloat.

Credit

  • To my former employer Diadrom AB. for allowing me to share this code with a BSD license. They funded most of the development of this project.
  • To Dmitry Vjukov for all the knowledge he has spread on the internets, including the algorithms for the two queues on this project.
  • To my girlfriend for coexisting with me when I become temporarily autistic after having been "in the zone" for too long.

Motivation

This started with the intention to just develop an asynchronous logger that could be used from different dynamically loaded libraries on the same binary without doing link-time hacks like being forced to link static, hiding symbols and some other "niceties".

Then at some point way after the requirements were met I just improved it for fun.

Design rationale

To be:

  • Simple. Not over abstracted and feature bloated, explicit, easy to figure out what the code is doing, easy to modify (2017 self-comment: not that easy after having switched to raw C coding for a while :D).
  • Very low latency. Fast for the caller.
  • Asynchronous (synchronous calls can be made on top for special messages, but they are way slower than using a synchronous logger in the first place).
  • Have minimum string formatting on the calling thread for the most common use cases.
  • Don't use thread-local-storage, the user threads are assumed as external and no extra info is attached to them.
  • Have termination functions to allow blocking until all the logs have been written to disk in program exit (or signal/fault) scenarios.

Various features

  • Targeting g++4.7 and VS 2010(can use incomplete or broken C+11 features from boost).
  • Nanosecond performance.
  • No singleton by design, usable from dynamically loaded libraries. The user provides the logger instance either explicitly or by a global function (Koenig lookup).
  • Suitable for soft-realtime work. Once it's initialized the fast-path can be clear from heap allocations (if properly configured to).
  • File rotation-slicing.
  • One conditional call overhead for inactive logging levels.
  • Able to strip log levels at compile time (for Release builds).
  • Lazy parameter evaluation (as usual with most logging libraries).
  • No ostreams(*) (a very ugly part of C++: dynamically allocated, verbose and stateful), just format strings checked at compile time (if the compiler supports it) with type safe values.
  • The logger severity threshold can be externally changed outside of the process. The IPC mechanism is the simplest, the log worker periodically polls some files when idling (if configured to).
  • Fair blocking behaviour (configurable by severity) when the bounded queue is full and the heap queue is disabled. The logger smoothly starts to act as a synchronous logger. If non-blocking behavior is desired an error is returned instead.
  • Small, you can actually compile it as a part of your application.

(*)An on-stack ostream adapter is available as a last resort, but its use is more verbose and has more overhead than the format literals.

see this example that more or less showcases all available features.

How does it work

It just borrows ideas from many of the loggers out there.

As an asynchronous logger its main objetive is to be as fast and to have as low latency as possible for the caller thread.

When the user is to write a log message, the producer task is to serialize the data to a memory chunk which then the logging backend (consumer) can decode, format and write. No expensive operations occur on the consumer, and if they do it's when using secondary features.

The format string is required to be a literal (compile time constant), so when encoding something like the entry below...

log_error ("File:a.cpp line:8 This is a string that just displays the next number {}, int32_val);

...the memory requirements are just a pointer to the format string and a deep copy of the integer value. The job of the caller is just to serialize some bytes to a memory chunk and to insert the chunk into a queue.

The queue is a mix of two famous lockfree queues of Dmitry Vyukov (kudos to this genious) for this particular MPSC case. The queue is a blend of a fixed capacity and fixed element size array based preallocated queue and an intrusive node based dynamically allocated queue. The resulting queue is still linearizable.

The format string is type-safe and validated at compile time for compilers that support "constexpr" and "variadic template parameters". Otherwise the errors are caught at run time on the logged output (Visual Studio 2010 mostly).

There are other features: you can block the caller thread until some message has been dequeued by the logger thread, to do C++ stream formatting on the caller thread, etc.

File rotation

The library can rotate log files.

Using the current C++11 standard files can just be created, modified and deleted. There is no way to list a directory, so the user is required to pass the list of files generated by previous runs.at startup time.

There is an example here.

Initialization

The library isn't a singleton, so the user should provide a reference to the logger instance on each call.

There are two methods to pass the instance to the logging macros, one is to provide it explicitly and the other one is by providing it on a global function.

If no instance is provided, the global function "get_mal_logger_instance()" will be called without being namespace qualified, so you can use Koenig lookup/ADL. This happens when the user calls the macros with no explicit instance suffix, as e.g. "log_error(fmt string, ...)".

To provide the instance explictly the macros with the "_i" suffix need to be called, e.g. "log_error_i(instance, fmt_string, ...)"

The name of the function can be changed at compile time, by defining MAL_GET_LOGGER_INSTANCE_FUNCNAME.

Termination

The worker blocks on its destructor until its work queue is empty when normally exiting a program.

When a signal is caught you can call the frontend function on termination in your signal handler. This will flush the logger queue and early abort any synchronous calls.

Errors

As of now, every log call returns a boolean to indicate success.

The only possible failures are either to be unable to allocate memory for a log entry or an asynchronous call that was interrupted by "on_termination". A filtered out call returns true".

The logging functions never throw.

Compiler macros

Those that are self-explanatory won't be explained.

  • MAL_GET_LOGGER_INSTANCE_FUNC: See the "Initialization" chapter above.
  • MAL_STRIP_LOG_SEVERITY: Removes the entries of this severity and below at compile time, so they are not evaluated and don't take code space. 0 is the "debug" severity, 5 is the "critical" severity. Stripping at level 5 leaves no log entries at all. Yo can define e.g. MAL_STRIP_LOG_DEBUG, MAL_STRIP_LOG_TRACE, etc. instead. If you define MAL_STRIP_LOG_TRACE all the severities below will be automatically defined for you (in this case MAL_STRIP_LOG_DEBUG).
  • MAL_DYNLIB_COMPILE: Define it when compiling MAL as a dynamic library/shared object.
  • MAL_DYNLIB: Define it when using MAL as a dynamic library. Don't define it if you are static linking or compiling the library with your project.
  • MAL_CACHE_LINE_SIZE: The cache line size of the machine you are compiling for. This is just used for data structure padding. 64 is defaulted when undefined.
  • MAL_USE_BOOST_CSTDINT: If your compiler doesn't have use boost.
  • MAL_USE_BOOST_ATOMIC
  • MAL_USE_BOOST_CHRONO
  • MAL_USE_BOOST_THREAD
  • MAL_NO_VARIABLE_INTEGER_WIDTH: Integers are encoded ignoring the number trailing bytes set to zero, not based on its data type size. So when this isn't defined e.g. encoding an uint64 with a value up to 255 takes one byte (plus 1 byte header). Otherwise all uint64 values will take 8 bytes (plus header), so encoding is less space efficient in this way but it frees the CPU and allows the compiler to inline more.

compilation

You can compile the files in the "src" folder and make a library or just compile everything under /src in your project.

Otherwise you can use cmake.

On Linux there are Legacy GNU makefiles in the "/build/linux" folder too. They respect the GNU makefile conventions. "DESTDIR", "prefix", "includedir" and "libdir" can be used. These are mainly left there because the examples were not ported to CMake.

REMEMBER (for legacy users that use boost): If the library is compiled with e.g. the "MAL_USE_BOOST_THREAD" and "MAL_USE_BOOST_CHRONO" preprocessor variables the client code must define them too. TODO: config.h.in

Windows compilation

There is a Visual Studio 2010 Solution in the "/build/windows" folder, but you need to do a step before opening:

If you don't need the Boost libraries you should run the "build\windows\mal-log\props\from_empty.bat" script. If you need them you should run the "build\windows\mal-log\props\from_non_empty.bat" script.

If you don't need the Boost libraries you can open and compile the solution, otherwise you need to edit (with a text editor) the newly generated file ""build\windows\mal-log\props\mal_dependencies.props" before and to update the paths in the file. You can do this through the Visual Studio Property Manager too.

Performace

It used to be some benchmark code here but the results were reported as being off compared with what some users where getting (out of date versions?).

The tests also had the problem that they were assuming a normal distribution for the latencies.

It was not very nice to add a lot of submodules to this project just for building the benchmarks.

Because all of these reasons I have created a separate repository with some benchmark code. It uses CMake to compile everything and build a static linked executable, so it's very easy to build and run.

If you are writing a logger and want to add it or have some suggestions about how to improve the bencmark code Pull Requests are welcome.

Here is the benchmark project..

Written with StackEdit.

Comments
  • silently refusing to log additional arguments

    silently refusing to log additional arguments

    In some cases the compile fails due to hitting a maximum number of arguments, which tells you there is a problem.

    However in this example (v1 is an int, v4 is a double) the first two log statements silently fail to log, which is a bit scary. Is this a bug?

    log_trace("try too many args {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}", v1, v4, v1, v4, v1, v4, v1, v4, v1, v4, v1); log_trace("try too many args {}, {}, {}, {}, {}, {}, {}, {}, {}, {}", v1, v4, v1, v4, v1, v4, v1, v4, v1, v4); log_trace("try too many args {}, {}, {}, {}, {}, {}, {}, {}, {}", v1, v4, v1, v4, v1, v4, v1, v4, v1); log_trace("try too many args {}, {}, {}, {}, {}, {}, {}, {}", v1, v4, v1, v4, v1, v4, v1, v4); log_trace("try too many args {}, {}, {}, {}, {}, {}, {}", v1, v4, v1, v4, v1, v4, v1); log_trace("try too many args {}, {}, {}, {}, {}, {}", v1, v4, v1, v4, v1, v4);

    opened by funkyshoc 11
  • negative numbers

    negative numbers

    Is it true that for logging negative numbers, that you need to cast your variable with a (i64)/etc? Logging negative numbers by having to remember your variable type and therefore cast it correctly for all integer based variables could be very problematic (since many parts of my code use uint64_t and int64_t) Or is it just that the gcc 4.8 I'm running doesn't like some part of the code.

    In your example code for example I get the following line of 2=4294966065 instead of 2=-1231:

    00000000000.017025326 [err_] message 40, 1=true, 2=4294966065, 3=432.12, 4=param4, 5=0xffffffffffffffff

    opened by funkyshoc 8
  • timestamping options

    timestamping options

    Do you have a "user friendly" timestamp option?

    Example:

    2017-04-15 09:16:00.105972 Test my data

    Vs:

    00000000003.321687819 Test my data

    Also - if you have one producer that wants the timestamp printed and another that does not on two different files -- do you have to start two threads as it stands or is there still a single thread but multiple log files/multiple configurations option?

    opened by funkyshoc 6
  • QUESTION: Way to log string variables

    QUESTION: Way to log string variables

    Which way is better?

    std::string str = "THIS IS A TEST";
    
    log_debug("Example string log: {}", mal::lit(str.c_str()));
    

    or

    log_debug("Example string log: {}", mal::deep_copy(str));

    Thanks in advance!

    opened by bellekci 3
  • compile error with too many arguments

    compile error with too many arguments

    I get a nice compiler error in one case where I exceed 14 arguments. It is nice that it catches the problem on compile time in that case. However if I do want to exceed 14 arguments, is there a define or some config option I can set to allow additional arguments?

    opened by funkyshoc 3
  • logging individual characters (char)

    logging individual characters (char)

    Hello the logger seems to log individual characters such as 'C' as the ASCII 67. I think there is quite a bit of value in logging them as is such as C

    (for the native type char)

    00000000000.019507570 [err_] message 37, char ('C') = 67

    If there is no way to do so, do you think this would be easy to change?

    opened by funkyshoc 3
  • Add CMake

    Add CMake

    This is a preliminary attempt to port your library to a reasonable build system (even though it only has a single file to compile). It should work on "normal" platforms that aren't 10 years old and using an unsupported operating system. It supports building with GCC and MSVC and with/without Boost. However, since sane compilers have supported C++11 for years, I'm of the opinion that Boost support, which is currently broken (see my other pull request), should be dropped. But that's up to you.

    Doesn't work with clang because of very weird platform detection code in include/mal_log/util/system.hpp. Seems weird to force people to use GCC these days.

    I've tested it on Ubuntu 16.04 (with GCC 5.4 and Boost 1.58) and on Windows 10 (with MSVC 2017 x64). MSVC 2017 build tools can be downloaded for free and can target insecure OS's like Windows XP if needed (with the cmake -T flag) so I can see no reason to support older versions of Microsoft's build tools (though this CMake file probably works with them).

    Could not get things to compile on a docker container for Ubuntu 12.04 LTS (with GCC 4.6 and Boost 1.46), but I don't think it makes sense to support that kind of old platform. 12.04 LTS supports ends in a few days anyway and Boost 1,46 is missing boost::atomic and boost::chrono. A newer boost version on that old compiler may have worked.

    To build on Unix:

    mkdir builddir
    cd builddir
    cmake [-DCMAKE_BUILD_TYPE=[Release,RelWithDebInfo,Debug,etc (defaults to Debug per CMake)]] ..
    make
    make install
    

    To build with MSVC and NMake (from dev command prompt):

    mkdir builddir
    cd builddir
    cmake ..
    cmake --build . [--config [Release,RelWithDebInfo,Debug,etc]]
    cmake --build . [--config [Release,RelWithDebInfo,Debug,etc]] install
    

    Other usual CMake options, generators, etc are supported.

    Custom install directory can be specified (good for staging): cmake .. -DCMAKE_INSTALL_PREFIX=whatever

    Show build options (USE_BOOST, STRIP_LOG_[LEVEL], etc) cmake -LH ..

    opened by woobs 2
  • Can I add new log levels?

    Can I add new log levels?

    spdlog has trace support which is great for development - it's only included if DEBUG is enabled or a specific flag is passed at compile time. Do you support the same?

    opened by ruipacheco 1
  • printf/sprintf format related warnings when targetting x64

    printf/sprintf format related warnings when targetting x64

    This pull request contained the required fixes:

    https://github.com/RafaGago/ufo-log/pull/3

    In a first instance using ostreams for the example files and the preprocessor for the internals could work.

    bug 
    opened by RafaGago 1
  • Updated the benchmark for better spdlog results

    Updated the benchmark for better spdlog results

    Hi, I added the following for better spdlog results:

    1. spdlog async works best when max queue size is not so big.
    2. Added spdlog sync test (which is faster than async - as async mode is still work in progress)

    Anyway, it is good stuff. Going to borrow ideas (and maybe code if ok :) for future spdlog async improvments..

    Would be nice if you rerun and publish new results

    opened by gabime 1
Owner
null
Custom Arduino-based temperature and humidity data logger.

Arduino Datalogger A custom data logger based on ATmega32u4 with a custom PCB. The device features the following main components: Atmel ATmega32u4 MCU

Valentin Bersier 7 Jul 19, 2021
An easy to build CO2 Monitor/Meter with Android and iOS App for real time visualization and charting of air data, data logger, a variety of communication options (BLE, WIFI, MQTT, ESP-Now) and many supported sensors.

CO2-Gadget An easy to build CO2 Monitor/Meter with cell phone App for real time visualization and charting of air data, datalogger, a variety of commu

Mariete 22 Aug 5, 2022
Anotter USB temperature logger that can record up to four channels with thermocouple or NTCs connected via CDC directly or SCPI to USB.

temperature-logger Anotter USB temperature logger that can record up to four channels with thermocouple or NTCs connected via CDC directly or SCPI to

Jana Marie Hemsing 48 Aug 11, 2022
Disable OTA Update for iOS & iPadOS for 14 - 14.3 (Non Jailbreak devices)

OTADisabler-App Disable OTA Update for iOS & iPadOS for 14 - 14.3 (Non Jailbreak devices) Support Devices iOS 14.0-14.3 (confirmed on iOS 14.1 and abo

ichitaso 29 Sep 14, 2022
Exploit allowing you to read registry hives as non-admin on Windows 10 and 11

HiveNightmare aka SeriousSam, or now CVE-2021–36934. Exploit allowing you to read any registry hives as non-admin. What is this? An zero day exploit f

Kevin Beaumont 610 Sep 30, 2022
Faster Non-Integer Sample Rate Conversion

Non-Integer Sample Rate Conversion This repository contains a comparison of sample-rate conversion (SRC) algorithms, with an emphasis on performance f

null 25 Jul 5, 2022
Demagnetization tensor of non-equidistant magnetic layers

Demagnetization tensor of non-equidistant magnetic layers A small standalone project calculating the demagnetization tensor from [1] in multi-threaded

magnum.af 1 Dec 3, 2021
Next-gen Rowhammer fuzzer that uses non-uniform, frequency-based patterns.

Blacksmith Rowhammer Fuzzer This repository provides the code accompanying the paper Blacksmith: Scalable Rowhammering in the Frequency Domain that is

Computer Security Group @ ETH Zurich 171 Sep 23, 2022
A repository for experimenting with elf loading and in-place patching of android native libraries on non-android operating systems.

droidports: A repository for experimenting with elf loading and in-place patching of android native libraries on non-android operating systems. Discla

João Henrique 23 Aug 5, 2022
a unix inspired, non posix compliant micro kernel (more of a monolithic kernel for now though) that i am working on in my spare time

toy-kernel a unix inspired, non posix compliant micro kernel (more of a monolithic kernel for now though) that i am working on in my spare time prereq

czapek 13 Jun 14, 2022
Non-blocking blinking patterns and smooth fade effects for your LEDs and buzzers.

Arduino-Blinkenlight Featured on hackster.io! ?? Supercharge your status-LEDs ?? This library gives you non-blocking blinking patterns and smooth fade

Thomas Feldmann 22 Sep 15, 2022
Reactive - Simple, non intrusive reactive programming library for C++. (Events + Observable Properties + Reactive Properties)

Header only reactive C++ library. Thread-safe, memory-safe. Concsists from: Event (C# like Event, but thread-safe, does not block while event occurs,

null 59 Jul 28, 2022
Async non-blocking multi-protocol networking library for C/C++

Fossa: Multi-Protocol Networking Library Note: As of September 21st 2015, Fossa project has been merged back into Mongoose project Fossa is a multi-pr

Cesanta Software 416 Sep 25, 2022
A simple PoC to demonstrate that is possible to write Non writable memory and execute Non executable memory on Windows

WindowsPermsPoC A simple PoC to demonstrate that is possible to write Non writable memory and execute Non executable memory on Windows You can build i

Lorenzo Maffia 55 Jul 21, 2022
Blend between two non-concentric non-circular cones for use in a Pose Space system

BlurRelax Why does maya not have a smooth or relax deformer? They've got brushes, sure, but no deformers. This one is pretty fast. I've got it working

Tyler Fox 2 Dec 27, 2021
A DC power monitor and data logger

Hoverboard Power Monitor I wanted to gain a better understanding of the power consumption of my hoverboard during different riding situations. For tha

Niklas Roy 22 May 1, 2021
Android-Syscall-Logger

Android-Syscall-Logger A kernel module that hook some of your system call on your Android Device by rewriting syscall table. Prerequisite pixel 1 andr

null 92 Sep 28, 2021
📝 Kernel module that can be used as a replacement for logger or logwrapper

Kernel logger Kernel logger is a kernel module that can be used as a replacement for logger or logwrapper. Its log is similar to systemd's journal and

Tian Yuanhao 38 May 21, 2022
Custom Arduino-based temperature and humidity data logger.

Arduino Datalogger A custom data logger based on ATmega32u4 with a custom PCB. The device features the following main components: Atmel ATmega32u4 MCU

Valentin Bersier 7 Jul 19, 2021