Using io_uring Linux Kernel interface from Python by JITing C code with MetaCall.

Overview

MetaCall Python C io_uring Example

io_uring is a new Linux Kernel interface that speeds up I/O operations in comparison to previous implementations like libuv (the library that NodeJS uses internally for handling I/O).

This interface is offered through liburing, which provides a C API for accessing it. We could write a Python extension by using Python C API, or use ctypes in order to call to the library.

Developing those wrappers is costly because either we have to write C/C++ or Python boilerplate. So instead of doing that, we will be using MetaCall in order to achieve this.

MetaCall allows to transparently call to C functions, we can implement anything in C and without need to compile it manually (it will be JITed at load time). So basically we can load C code directly into Python. For example:

int sum(int a, int b) {
	return a + b;
}
from sum.c import sum

sum(3, 4) # 7

With this we can use our C library not only from Python but from any other language, for example, NodeJS:

const { sum } = require("./sum.c");

sum(3, 4) // 7

We will be avoiding all the boilerplate and we will have a single interface for all languages. The calls will be also type safe and we will avoid a lot of errors and time for maintaining the wrappers for each language that we can spend focusing on the development. Here's an example of Cython + LWAN, as you can see, although the performance is excellent, the boilerplate is huge in comparison to MetaCall.

In this example we want to bring the power of io_uring to Python for maximizing the speed of I/O and outperform Python native primitives like http.server module. For demonstrating it, we have a server_listen function which creates a simple HTTP server in the port 8000. As example, we are also passing another parameter as a callback called fibonacci. This is a function that will act as a handler for the POST method, we will use it for calculating the fibonacci sequence. When doing a POST against the server, we will use the path as the Nth element of fibonacci sequence to be calculated. In our C code this will get translated automatically into a function pointer with the signature long (*handler)(long) transparently. Invoking the callback handler from C land will execute the function fibonacci in Python land.

Docker

Building and running with Docker:

docker build -t metacall/python-c-io_uring-example .
docker run --rm -p 8000:8000 -it metacall/python-c-io_uring-example

Testing

For calculating the Fibonacci of 15 and 17 respectively:

curl -X POST http://localhost:8000/15
610
curl -X POST http://localhost:8000/17
1597
Owner
MetaCall
MetaCall
A kernel module that patches Linux kernel "on-the-fly" to skip TASK_RSS_EVENTS_THRESH check in check_sync_rss_stat

split-rss-counting-patch A kernel module that patches Linux kernel "on-the-fly" to skip TASK_RSS_EVENTS_THRESH check in check_sync_rss_stat. Why? Read

Bao-Hiep Le 3 Mar 6, 2022
The source for the Linux kernel used in Windows Subsystem for Linux 2 (WSL2)

Introduction The WSL2-Linux-Kernel repo contains the kernel source code and configuration files for the WSL2 kernel. Reporting Bugs If you discover an

Microsoft 5.9k Aug 2, 2022
Historic source code for version 0.01 of the Linux kernel

Linux v0.01 Source Code (A historic repository of the first official release of the Linux Kernel) About This Repo This repo is a means of keeping and

Robert Menes 3 Jun 27, 2022
CppADCodeGen with an easy Eigen interface and Python bindings.

CppADCodeGenEigenPy CppADCodeGen with an easy Eigen interface and Python bindings. This project has been tested on Ubuntu 16.04, 18.04, and 20.04. It

Adam Heins 11 May 18, 2022
Geant4-Python interface

Geant4Py Geant4-Python interface. Formerly c++-python bridge was realized with Boost-Python. In Geant4 version 11, we migrate to pybind11 as a Python

Koichi Murakami 4 Mar 23, 2022
A low level Operating System designed using Linux Kernel

Conqueror A low level Operating System designed using Linux Kernel To develop the basic low level operating system, we need following Virtual Machine

mahendra gandham 8 May 27, 2022
The purpose of this project is to create a modern C++17 header-only interface to the FreeRTOS kernel API

FreeRTOS-Cpp The purpose of this project is to create a modern C++17 header-only interface to the FreeRTOS kernel API. Goals of this project include:

Jon Enz 16 Jul 16, 2022
AX.25 protocol (packet radio and APRS) interface / bridging / switching / terminal related software for Linux

CB3ROB-AX25-TOOLS-LINUX 2021-11-23T00:58:49Z (TUESDAY) cb3rob-kiss-tcp-attach.c links ax0 interface to KISS-TCP TNC, soundmodem (direwolf), or multipl

CB3ROB TACTICAL SYSTEMS 1 Jan 1, 2022
ARPANET NCP for Linux, and IMP-host interface to SIMH

ARPANET Network Control Program for Linux, interfacing with the SIMH IMP emulator. This is a work in progress. At this point, the NCP is not a high qu

Lars Brinkhoff 3 Jul 21, 2022
Loads a signed kernel driver which allows you to map any driver to kernel mode without any traces of the signed / mapped driver.

CosMapper Loads a signed kernel driver (signed with leaked cert) which allows you to map any driver to kernel mode without any traces of the signed /

null 118 Jul 26, 2022
A kernel module to provide /system/xbin/su to Android Kernel (especially to WSA)

WSA-Kernel-SU Intro This is a kernel module to provide /system/xbin/su to Android Kernel (especially to WSA). Only works on 4.17+ kernel. For older ke

LSPosed 132 Jul 27, 2022
NetHunter Kernel for the OnePlus 7 Series based on DragonHeart Kernel

Linux kernel ============ This file was moved to Documentation/admin-guide/README.rst Please notice that there are several guides for kernel develop

cyberknight777 9 Mar 13, 2022
a unix inspired, non posix compliant micro kernel (more of a monolithic kernel for now though) that i am working on in my spare time

toy-kernel a unix inspired, non posix compliant micro kernel (more of a monolithic kernel for now though) that i am working on in my spare time prereq

czapek 13 Jun 14, 2022
Arduino code to interface with quadrature-encoder mice, specifically the Depraz mouse

Depraz Mice on USB via Arduino This code lets you connect a Depraz mouse to a modern computer via USB. The Depraz mouse has a male DE-9 connector but

John Floren 5 Jul 8, 2022
Quick fix to iphone usb tethering with ios14 or higher for Linux kernel lower than 5.10.4

Quick fix to Linux Iphone USB tethering with IOS 14 or higher (Tested with ubuntu 18.04, kernel 5.4.0-65, if you fail in the build, please download yo

null 23 Jul 8, 2022
Linux kernel module to fight against police terror

protecc is a Linux kernel module that will shut down your computer when a predefined USB device is removed from the system.

null 21 Jun 24, 2022
Notes and codes for Linux Kernel (SJTU-CS353)

Linux-Kernel-notes Notes and codes for Linux Kernel (SJTU-CS353) Lab 01: Module Programming 02: Process Management 03: Memory Management Notes Lec 1.

Zi-Han Liu 21 Jun 29, 2022
Linux kernel module for Onion Omega2 to control WS2811/WS2812 LEDs

omega2-ws2811-lkm Linux kernel module for Onion Omega2 to control WS2811/WS2812 LEDs. It's using a bit-banging, so you can use any GPIO pin. Also, it

Alexey 'Cluster' Avdyukhin 15 Jun 7, 2022
Linux kernel labs

Linux Kernel Labs This repo illustrates how to write Linux kernel module labs safely and gracefully in a simple and concise way. Contrary to most work

Alex Chi 50 May 11, 2022