A high performance, shared memory, lock free, cross platform, single file, no dependencies, C++11 key-value store

Related tags

Miscellaneous simdb
Overview

alt text

SimDB

A high performance, shared memory, lock free, cross platform, single file, no dependencies, C++11 key-value store.

SimDB is part of LAVA (Live Asynchronous Visualized Architecture) which is a series of single file, minimal dependency, C++11 files to create highly concurrent software while the program being written runs live with internal data visualized.

  • Hash based key-value store created to be a fundamental piece of a larger software architecture.

  • High Performance - Real benchmarking needs to be done, but superficial loops seem to run conservatively at 500,000 small get() and put() calls per logical core per second. Because it is lock free the performance scales well while using at least a dozen threads.

  • Shared Memory - Uses shared memory maps on Windows, Linux, and OS X without relying on any external dependencies. This makes it exceptionally good at interprocess communication.

  • Lock Free - The user facing functions are thread-safe and lock free with the exception of the constructor (to avoid race conditions between multiple processes creating the memory mapped file at the same time).

  • Cross Platform - Compiles with Visual Studio 2013 and ICC 15.0 on Windows, gcc 5.4 on Linux, gcc on OS X, and clang on OS X.

  • Single File - simdb.hpp and the C++11 standard library is all you need. No Windows SDK or any other dependencies, not even from the parent project.

  • Apache 2.0 License - No need to GPL your whole program to include one file.

This has already been used for both debugging and visualization, but should be treated as alpha software. Though there are no known outstanding bugs, there are almost certainly bugs (and small design issues) waiting to be discovered and so will need to be fixed as they arise.

Getting Started

simdb db("test", 1024, 4096);

This creates a shared memory file that will be named "simdb_test". It will be a file in a temp directory on Linux and OSX and a 'section object' in the current windows session namespace (basically a temp file complicated by windows nonsense).

It will have 4096 blocks of 1024 bytes each. It will contain about 4 megabytes of space in its blocks and the actual file will have a size of about 4MB + some overhead for the organization (though the OS won't write pages of the memory map to disk unless it is neccesary).

auto dbs = simdb_listDBs();

This will return a list of the simdb files in the temp directory as a std::vector of std::string. Simdb files are automatically prefixed with "simdb_" and thus can searched for easily here. This can make interprocess communication easier so that you can do things like list the available db files in a GUI. It is here for convenience largely because of how difficult it is to list the temporary memory mapped files on windows.

db.put("lock free", "is the way to be");

SimDB works with arbitrary byte buffers for both keys and values. This example uses a convenience function to make a common case easier.

string s = db.get("lock free");       // returns "is the way to be"

This is another convenience function for the same reason. Next will be an example of the direct functions that these wrap.

string lf  = "lock free";

string way = "is the way to be";

i64    len = db.len( lf.data(), (u32)lf.length() );

string way2(len,'\0');

bool    ok = db.get( lf.data(), (u32)lf.length(), (void*)way.data(), (u32)way.length() );

Here we can see the fundamental functions used to interface with the db. An arbitrary bytes buffer is given for the key and another for the value. Keep in mind here that get() can fail, since another thread can delete or change the key being read between the call to len() (which gets the number of bytes held in the value of the given key) and the call to get(). Not shown is del(), which will take a key and delete it.

Inside simdb.hpp there is a more extensive explanation of the inner working and how it achieves lock free concurrency

Comments
  • Is this still maintained?

    Is this still maintained?

    Hi I was wondering if you plan to continue maintaining this repo. That last update to the core file was 5 months ago but I noticed the same file over on your https://github.com/LiveAsynchronousVisualizedArchitecture/lava which was updated around 19 days ago. Should I use this repo or just extract that file from the LAVA repo for my uses?

    Mostly asking because I wanted to make sure whether the changes to the DB library in the LAVA repo was specific to that project or if the changes were generally useful.

    Thanks for making this project!

    opened by DomtronVox 5
  • compile error on linux

    compile error on linux

    I want to know how to deal with the compile error on linux. Thanks!

    simdb.hpp:505:5: error: no matching function for call to 'atomic_compare_exchange_strong_explicit(CncrLst::au64*&, CncrLst::u64*&, CncrLst::au64&, std::memory_order, std::memory_order)' ); ^ simdb.hpp: In member function 'CncrLst::u32 CncrLst::nxt(CncrLst::u32)': simdb.hpp:538:54: error: use of deleted function 'std::atomic::atomic(const std::atomic&)' }while( !headCmpEx( &curHead.asInt, nxtHead.asInt) ); ^ simdb.hpp:538:54: error: use of deleted function 'std::atomic::atomic(const std::atomic&)' }while( !headCmpEx( &curHead.asInt, nxtHead.asInt) ); ^

    simdb.hpp:1230:112: error: 'vi' was not declared in this scope auto runIfMatch(VerIdx vi, const void* const buf, u32 len, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const -> std::pair<Match, T> // std::pair<Match, decltype(f(vi))> ^~ simdb.hpp:1230:110: error: there are no arguments to 'f' that depend on a template parameter, so a declaration of 'f' must be available [-fpermissive] auto runIfMatch(VerIdx vi, const void* const buf, u32 len, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const -> std::pair<Match, T> // std::pair<Match, decltype(f(vi))> ^ simdb.hpp:1230:110: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated) simdb.hpp:1230:112: error: 'vi' was not declared in this scope auto runIfMatch(VerIdx vi, const void* const buf, u32 len, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const -> std::pair<Match, T> // std::pair<Match, decltype(f(vi))> ^~ simdb.hpp:1230:110: error: there are no arguments to 'f' that depend on a template parameter, so a declaration of 'f' must be available [-fpermissive] auto runIfMatch(VerIdx vi, const void* const buf, u32 len, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const -> std::pair<Match, T> // std::pair<Match, decltype(f(vi))> ^ simdb.hpp:1348:99: error: 'vi' was not declared in this scope bool runMatch(const void *const key, u32 klen, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const ^~ simdb.hpp:1348:97: error: there are no arguments to 'f' that depend on a template parameter, so a declaration of 'f' must be available [-fpermissive] bool runMatch(const void *const key, u32 klen, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const ^ simdb.hpp:1348:99: error: 'vi' was not declared in this scope bool runMatch(const void *const key, u32 klen, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const ^~ simdb.hpp:1348:97: error: there are no arguments to 'f' that depend on a template parameter, so a declaration of 'f' must be available [-fpermissive] bool runMatch(const void *const key, u32 klen, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const ^ make: *** [/home/iplature/iplaturetest/iPlature/source/iplature_exegeneral:149: simdbTest.o] Error 1

    opened by zwm0502 0
  • Compile error on linux with gcc-7.3

    Compile error on linux with gcc-7.3

    Hello, I met some problems when I compiled with simdb.hpp .How to deal with them. Thanks!

    In file included from simdbTest.cpp:5:0: simdb.hpp: In member function 'bool CncrLst::headCmpEx(CncrLst::u64*, au64)': simdb.hpp:505:5: error: no matching function for call to 'atomic_compare_exchange_strong_explicit(CncrLst::au64*&, CncrLst::u64*&, CncrLst::au64&, std::memory_order, std::memory_order)' ); ^ In file included from simdb.hpp:205:0, from simdbTest.cpp:5:

    simdb.hpp: At global scope: simdb.hpp:1230:112: error: 'vi' was not declared in this scope auto runIfMatch(VerIdx vi, const void* const buf, u32 len, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const -> std::pair<Match, T> // std::pair<Match, decltype(f(vi))> ^~ simdb.hpp:1230:110: error: there are no arguments to 'f' that depend on a template parameter, so a declaration of 'f' must be available [-fpermissive] auto runIfMatch(VerIdx vi, const void* const buf, u32 len, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const -> std::pair<Match, T> // std::pair<Match, decltype(f(vi))> ^ simdb.hpp:1230:110: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated) simdb.hpp:1230:112: error: 'vi' was not declared in this scope auto runIfMatch(VerIdx vi, const void* const buf, u32 len, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const -> std::pair<Match, T> // std::pair<Match, decltype(f(vi))> ^~ simdb.hpp:1230:110: error: there are no arguments to 'f' that depend on a template parameter, so a declaration of 'f' must be available [-fpermissive] auto runIfMatch(VerIdx vi, const void* const buf, u32 len, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const -> std::pair<Match, T> // std::pair<Match, decltype(f(vi))> ^ simdb.hpp:1348:99: error: 'vi' was not declared in this scope bool runMatch(const void *const key, u32 klen, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const ^~ simdb.hpp:1348:97: error: there are no arguments to 'f' that depend on a template parameter, so a declaration of 'f' must be available [-fpermissive] bool runMatch(const void *const key, u32 klen, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const ^ simdb.hpp:1348:99: error: 'vi' was not declared in this scope bool runMatch(const void *const key, u32 klen, u32 hash, FUNC f, T defaultRet = decltype(f(vi))() ) const ^~ simdb.hpp:1348:97: error: there are no arguments to 'f' that depend on a template parameter, so a declaration of 'f' must be available [-fpermissive]

    opened by zwm0502 0
  • compile error

    compile error

    simdb.hpp: In member function ‘CncrLst::u32 CncrLst::nxt()’:
    simdb.hpp:519:54: error: use of deleted function ‘std::atomic<long unsigned int>::atomic(const std::atomic<long unsigned int>&)’
      519 |     }while( !headCmpEx( &curHead.asInt, nxtHead.asInt) );
          |                                                      ^
    
    
    opened by wxd237 0
  • How to achieve cross platform IPC

    How to achieve cross platform IPC

    Hi, first, thank you for your wonderful work. I was able to compile and run the simdb code. From a single process, I'm able to create a simdb using the constructor and write to and read from the db. However, I don't know where to start with doing cross platform IPC. My intended platforms are OSX and Windows. Specifically,

    1. If I create a simdb in Proc1 and write some key value pairs to it, how can I read it from Proc2?
    2. How can this be done in a cross platform way so it works on both OSX and Windows?
    opened by rohanbabu 0
Owner
null
An AI for playing NES Tetris at a high level. Based primarily on search & heuristic, with high quality board evaluation through value iteration.

StackRabbit An AI that plays NES Tetris at a high level. Primarily based on search & heuristic, with high-quality board eval through value iteration.

Greg Cannon 230 Dec 2, 2022
An efficient, small mobile key-value storage framework developed by WeChat. Works on Android, iOS, macOS, Windows, and POSIX.

中文版本请参看这里 MMKV is an efficient, small, easy-to-use mobile key-value storage framework used in the WeChat application. It's currently available on Andr

Tencent 15.3k Dec 5, 2022
Modern transactional key-value/row storage library.

Sophia is advanced transactional MVCC key-value/row storage library. How does it differ from other storages? Sophia is RAM-Disk hybrid storage. It is

Dmitry Simonenko 1.8k Nov 11, 2022
A simple Roblox exploit written in C++ Everything in the C++ file is original work besides the dependencies, free for you to use.

headhunter A simple Roblox exploit written in C++ Everything in the C++ file is original work besides the dependencies, free for you to use. This code

ster ster 41 Nov 30, 2022
A single-file GLTF loader with no dependencies.

mvImporter Basic Usage #define MV_IMPORTER_IMPLEMENTATION #include "mvImporter.h" int main() { mvGLTFModel gltfmodel = mvLoadGLTF("C/dev/Resource

Jonathan Hoffstadt 10 Nov 23, 2022
Memory Process File System (MemProcFS) is an easy and convenient way of viewing physical memory as files in a virtual file system

The Memory Process File System (MemProcFS) is an easy and convenient way of viewing physical memory as files in a virtual file system.

Ulf Frisk 1.7k Dec 2, 2022
Opt - Class template designed to express optionality without having to sacrifice memory to store additional bool flag

mp::opt<T, Policy> mp::opt<T, Policy> is a class template designed to express optionality. It has interface similar to std::optional<T> (see here) but

Mateusz Pusz 9 Jun 5, 2022
⌨️ Personal key mapping for The Key.

The Key Personal key mapping for The Key. This firmware configures: The first key to be the mute key on single tap, and the pause/play key on double t

Zihua Li 1 Dec 25, 2021
Allows to swap the Fn key and left Control key and other tweaks on Macbook Pro and Apple keyboards in GNU/Linux

A patched hid-apple kernel module UPDATE August 2020: swap_fn_leftctrl is now built-in in Linux 5.8 ?? UPDATE Jun 2020: New feature added (swap_fn_f13

Zakhar Semenov 303 Nov 23, 2022
Cross-platform C++11 header-only library for memory mapped file IO

mio An easy to use header-only cross-platform C++11 memory mapping library with an MIT license. mio has been created with the goal to be easily includ

null 1.4k Nov 26, 2022
Several single-file, cross-platform, public domain libraries for C/C++ that I use for learning / testing

HTC Several single-file, cross-platform, public domain libraries for C/C++ that I use for learning / testing (Not meant for production code). This is

Chris 19 Nov 5, 2022
Embed read-only filesystems into any C++11 program w. a single header, zero dependencies and zero modifications to your code

c-embed Embed read-only filesystems into any C++11 program w. a single header, zero dependencies and zero modifications to your code. Usage c-embed al

Nick McDonald 8 Nov 23, 2022
An eventing framework for building high performance and high scalability systems in C.

NOTE: THIS PROJECT HAS BEEN DEPRECATED AND IS NO LONGER ACTIVELY MAINTAINED As of 2019-03-08, this project will no longer be maintained and will be ar

Meta Archive 1.7k Nov 20, 2022
Shared to msvcrt.dll or ucrtbase.dll and optimize the C/C++ application file size.

VC-LTL - An elegant way to compile lighter binaries. 简体中文 I would like to turn into a stone bridge, go through 500 years of wind, 500 years of Sun, ra

Chuyu Team 253 Nov 29, 2022
Flock is a C++ library supporting lock-free locks

Flock : A library for lock-free locks Flock is a C++ library supporting lock-free locks as described in the paper: Lock-free Locks: Revisited Naama Be

null 22 Nov 12, 2022
A dependency free, embeddable debugger for Lua in a single file (.lua or .c)

debugger.lua A simple, embedabble debugger for Lua 5.x, and LuaJIT 2.x. debugger.lua is a simple, single file, pure Lua debugger that is easy to integ

Scott Lembcke 593 Nov 29, 2022
PHP Encoder, protect PHP scripts in PHP 8 and PHP 7, High Performance, Compitable with X86_64, MIPS, ARM platform and Ubuntu/Centos/OpenWRT system.

What's FRICC2? FRICC2 is a PHP Script encryption tool. When you are developing a commercial software using PHP, the script can be distributed as encry

Hoowa Sun 42 Nov 23, 2022
A single file, single function, header to make notifications on the PS4 easier

Notifi Synopsis Adds a single function notifi(). It functions like printf however the first arg is the image to use (NULL and any invalid input should

Al Azif 9 Oct 4, 2022
Khepri is a Cross-platform agent, the architecture and usage like Coblat Strike but free and open-source.

Khepri Free,Open-Source,Cross-platform agent and Post-exploiton tool written in Golang and C++ Description Khepri is a Cross-platform agent, the archi

Young 1.4k Nov 24, 2022