Receive and process logs from the Linux kernel.

Related tags

Logging netconsd
Overview

Netconsd: The Netconsole Daemon

Continuous Integration

This is a daemon for receiving and processing logs from the Linux Kernel, as emitted over a network by the kernel's netconsole module. It supports both the old "legacy" text-only format, and the new extended format added in v4.4.

The core of the daemon does nothing but process messages and drop them: in order to make the daemon useful, the user must supply one or more "output modules". These modules are shared object files which expose a small ABI that is called by netconsd with the content and metadata for netconsole messages it receives.

This README explains how to build netconsd and use it with one of the existing output modules in the modules/ directory. The end discusses how to write your own custom output module.

Building netconsd

The default Makefile target intended for production use has no external dependencies besides glibc. To build it, just say make: you'll end up with a single executable in this directory called netconsd, and a *.so file for every module in the modules/ directory.

The Makefile includes a few other handy targets:

  • debug: Adds the usual debug flags, and also enables the ASAN and UBSAN sanitizers. You'll need to install libasan/libubsan on your system to build this target and run the binaries.
  • 32bit: Forces 32-bit compilation on x86_64 systems, for easily testing portability to 32-bit CPU architectures. You'll need to install 32-bit libraries if your distro doesn't have them.
  • debug32: Union of the 32bit and debug targets.
  • disasm: Emits verbose annotated disassembly in *.s files.

If you want to build the daemon with clang, just append CC="clang" to your make invocation. All the above targets should build with both clang and gcc.

Running netconsd

Setting up the server

By default, netconsd will use 1 listener and 2 worker threads, and listen on port 1514 for messages. You can use -l, -w, and -u respectively to change the defaults.

There's no universal wisdom about how many threads to use: just experiment with different numbers and use netconsblaster to load up the server. Both the blaster and the server will print how many packets they sent/processed.

If you run out of memory and OOM, you need more workers; if you see messages being dropped, you need more listeners. The tuning here will obviously depend on what your output module does: make sure to pass it when you do your testing.

For the simplest setup, just run:

$ make -s
$ ./netconsd ./modules/printer.so

Netconsd will always listen on INADDR_ANY and IN6ADDR_ANY. So far there's been no reason to make that configurable: if you care, open an issue and we will.

Setting up the client

The netconsole module takes a parameter like this:

netconsole=${sport}@${saddr}/${intf},${dport}@${daddr}/${dmac}

The fields are as follows:

  1. sport: Source port for the netconsole UDP packets
  2. saddr: Source address for the netconsole UDP packets
  3. intf: The name of the interface to send the UDP packets from
  4. dport: Destination port for the netconsole UDP packets
  5. daddr: Destination address for the netconsole UDP packets
  6. dmac: Destination L2 MAC address for the netconsole UDP packets

We need (6) because of how low-level netconsole is: it can't consult the routing table to send the packet, so it must know a priori what MAC address to use in the Ethernet frame it builds.

If you're talking to a server on the same L2 segment as the client, use the MAC address of that server. Otherwise, use the MAC address of your router. You can use the following quick shell one-liners to easily get the MAC of the router:

  • IPv6: ip -6 neighbor show | grep router
  • IPv4: sudo arp –a | grep gateway

Here are a couple examples for the parameter above:

IPv6: [email protected]:db00:11:801e:face:0:31:0/eth0,[email protected]:db00:11:d0be:face:0:1b:0/c0:8c:60:3d:0d:bc
IPv4: [email protected]/eth0,[email protected]/00:00:0c:9f:f1:90

Prepending + to the cmdline will cause kernels that support it to use extended netconsole, which you almost certainly want. Kernels too old to support extcon will silently ignore the +.

Once you have your parameter constructed, just insert the module with it:

$ sudo modprobe netconsole [email protected]:db00:11:801e:face:0:31:0/eth0,[email protected]:db00:11:d0be:face:0:1b:0/c0:8c:60:3d:0d:bc

You're good to go!

Testing on the client

Now that everything is running, you can use /dev/kmsg to write some logs:

/dev/kmsg' $ sudo bash -c 'echo "<0>OMG!" > /dev/kmsg' ">
$ sudo bash -c 'echo "Hello world!" > /dev/kmsg'
$ sudo bash -c 'echo "<0>OMG!" > /dev/kmsg'

The <0> tells the kernel what loglevel to use: 0 is KERN_EMERG, which ensures your message will actually get transmitted.

Writing an output module

Interface to netconsd

Output modules are shared object files loaded with dlopen() at runtime by netconsd. Netconsd will look for three functions in your module:

  1. int netconsd_output_init(int worker_thread_count)
  2. void netconsd_output_handler(int thread, struct in6_addr *src, struct msgbuf *buf, struct ncrx_msg *msg)
  3. void netconsd_output_exit(void)

If (1) exists, it is called when your module is loaded: the argument tells you how many worker threads netconsd is going to call your module from. If you return non-zero from this function, netconsd will abort() and exit.

If (3) exists, it is called when netconsd unloads your module.

For every message it receives, netconsd will call (2) in your module. The code must be reentrant: netconsd_output_handler() will be called concurrently from all of the worker threads in netconsd. The thread argument tells you which worker is invoking the function, which makes it easy to have per-thread data.

Netconsd uses a consistent hash to decide which worker to pass messages to, so messages from same remote address will always be queued to the same thread.

The src argument will always point to an in6_addr struct containing the source address of the netconsole packet. If the source was an IPv4 address, it will be formatted like ::FFFF: (see man ipv6 for details).

If the message had extended metadata, msg will point to the ncrx_msg struct containing that metadata and buf will be NULL. Otherwise, msg will be NULL and buf will point to a msgbuf struct with the raw message text.

Output modules must not modify the structures passed in. The memory backing all the pointers passed in will be freed immediately after the handler returns.

Building the modules

For modules written in C this is trivial: just compile with -shared.

For modules written in C++ it can be a bit trickier: you will probably need to build with -static-libstdc++ and/or -static-libgcc to make this work.

See the code and Makefile in modules/ for some examples of the above.

Contributing

See the CONTRIBUTING file for how to help out.

License

netconsd is BSD licensed, see the LICENSE file for more information.

netconsd was originally written by Calvin Owens as part of fbkutils in 2016, with later contributions by several other people. This repository is a direct continuation of that codebase.

You might also like...
Portable, simple and extensible C++ logging library
Portable, simple and extensible C++ logging library

Plog - portable, simple and extensible C++ logging library Pretty powerful logging library in about 1000 lines of code Introduction Hello log! Feature

A DC power monitor and data logger
A DC power monitor and data logger

Hoverboard Power Monitor I wanted to gain a better understanding of the power consumption of my hoverboard during different riding situations. For tha

An ATTiny85 implementation of the well known sleep aid. Includes circuit, software and 3d printed case design
An ATTiny85 implementation of the well known sleep aid. Includes circuit, software and 3d printed case design

dodowDIY An ATTiny85 implementation of the well known sleep aid. Includes circuit, software and 3d printed case design The STL shells are desiged arou

A BSD-based OS project that aims to provide an experience like and some compatibility with macOS
A BSD-based OS project that aims to provide an experience like and some compatibility with macOS

What is Helium? Helium is a new open source OS project that aims to provide a similar experience and some compatibiilty with macOS on x86-64 sytems. I

Netdata's distributed, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration.
Netdata's distributed, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration.

Netdata is high-fidelity infrastructure monitoring and troubleshooting. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.

A revised version of NanoLog which writes human readable log file, and is easier to use.
A revised version of NanoLog which writes human readable log file, and is easier to use.

NanoLogLite NanoLogLite is a revised version of NanoLog, and is easier to use without performance compromise. The major changes are: NanoLogLite write

Minimalistic logging library with threads and manual callstacks

Minimalistic logging library with threads and manual callstacks

A Fast and Convenient C++ Logging Library for Low-latency or Real-time Environments

xtr What is it? XTR is a C++ logging library aimed at applications with low-latency or real-time requirements. The cost of log statements is minimised

Grafana - The open-source platform for monitoring and observability
Grafana - The open-source platform for monitoring and observability

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

Comments
  • Handful of small cleanups/fixes

    Handful of small cleanups/fixes

    Hi @kernelslacker and @davide125 ,

    I saw this repo show up, thought I'd pitch in by fixing a few silly things I did in 2016. I think the 3rd patch may fix the open issue about endianness (I didn't see it until I was about to submit this). The srcaddr_order/thread_order thing is very confusing... I might send along another PR to clean it up in a week or two.

    All the best, Calvin

    CLA Signed 
    opened by jcalvinowens 5
  • netconsblaster fails to build on s390x

    netconsblaster fails to build on s390x

    https://koji.fedoraproject.org/koji/taskinfo?taskID=77675284

    netconsblaster.c:33:2: error: #error "Sorry, IPv6 address permutation code assumes a little-endian CPU"
    
    bug good first issue 
    opened by davide125 2
  • Support setting a destination UDP port

    Support setting a destination UDP port

    Summary: This just adds an option to the netconsd netconsblaster to enable setting the UDP destination port. This is useful in testing a netconsd with some test traffic.

    Reviewed By: kernelslacker

    Differential Revision: D35416810

    CLA Signed fb-exported 
    opened by jof 1
Owner
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Facebook
Tiny ANSI C lib for logs

logger.c An ANSI C (C86) lib for logs Easy to use and easy. Build You can build this lib or copy/paste sources files in your project. cd build make

Diesirae 7 Apr 10, 2022
CrashLogger - A dll injected into process to dump stack when crashing.

CrashLogger A dll injected into process to dump stack when crashing

YQ 20 Sep 3, 2022
Uberlog - Cross platform multi-process C++ logging system

uberlog uberlog is a cross platform C++ logging system that is: Small Fast Robust Runs on Linux, Windows, OSX MIT License Small Two headers, and three

IMQS Software 14 Aug 15, 2022
📝 Kernel module that can be used as a replacement for logger or logwrapper

Kernel logger Kernel logger is a kernel module that can be used as a replacement for logger or logwrapper. Its log is similar to systemd's journal and

Tian Yuanhao 38 May 21, 2022
log4cplus is a simple to use C++ logging API providing thread-safe, flexible, and arbitrarily granular control over log management and configuration. It is modelled after the Java log4j API.

% log4cplus README Short Description log4cplus is a simple to use C++17 logging API providing thread--safe, flexible, and arbitrarily granular control

null 1.3k Sep 14, 2022
Colorful Logging is a simple and efficient library allowing for logging and benchmarking.

Colorful-Logging "Colorful Logging" is a library allowing for simple and efficient logging as well for benchmarking. What can you use it for? -Obvious

Mateusz Antkiewicz 1 Feb 17, 2022
View and log aoe-api requests and responses

aoe4_socketspy View and log aoe-api requests and responses Part 1: https://www.codereversing.com/blog/archives/420 Part 2: https://www.codereversing.c

Alex Abramov 9 Aug 6, 2022