Modern, asynchronous, and wicked fast C++11 client for Redis

Related tags

Database redox
Overview

redox

Modern, asynchronous, and wicked fast C++11 client for Redis [Build Status] (https://travis-ci.org/hmartiro/redox)

Redox is a C++ interface to the Redis key-value store that makes it easy to write applications that are both elegant and high-performance. Communication should be a means to an end, not something we spend a lot of time worrying about. Redox takes care of the details so you can move on to the interesting part of your project.

Features:

  • Expressive asynchronous and synchronous API, templated by return value
  • Callbacks can be lambdas, class methods, bind expressions, or any std::function
  • Thread-safe - use one client in multiple threads or multiple clients in one
  • Automatic pipelining, even for synchronous calls from separate threads
  • Low-level access when needed
  • Accessible and robust error handling
  • Configurable logging level and output to any ostream
  • Full support for binary data (keys and values)
  • Fast - developed for robotics applications
  • 100% clean Valgrind reports

Redox is built on top of hiredis and libev. It uses only the asynchronous API of hiredis, even for synchronous commands. There is no dependency on Boost or any other libraries.

Benchmarks

Benchmarks are given by averaging the results of ten trials of the speed tests in examples/ on an AWS t2.medium instance running Ubuntu 14.04 (64-bit) and a local Redis server.

  • speed_test_async_multi over TCP: 879,589 commands/s
  • speed_test_async_multi over Unix socket: 901,683 commands/s
  • speed_test_async over TCP: 203,285 commands/s
  • speed_test_async over Unix socket: 301,823 commands/s
  • speed_test_sync over TCP: 21,072 commands/s
  • speed_test_sync over Unix socket: 24,911 commands/s

A mid-range laptop gives comparable results. Numbers can be much higher on a high-end machine.

Tutorial

This section introduces the main features of redox. Look in examples/ for more inspiration.

Hello world

Here is the simplest possible redox program:

#include <iostream>
#include <redox.hpp>

using namespace std;
using namespace redox;

int main(int argc, char* argv[]) {

  Redox rdx;
  if(!rdx.connect("localhost", 6379)) return 1;

  rdx.set("hello", "world!");
  cout << "Hello, " << rdx.get("hello") << endl;

  rdx.disconnect();
  return 0;
}

Compile and run:

$ g++ hello.cpp -o hello -std=c++11 -lredox -lev -lhiredis
$ ./hello
Hello, world!

This example is synchronous, in the sense that the commands don't return until a reply is received from the server.

Asynchronous commands

In a high-performance application, we don't want to wait for a reply, but instead do other work. At the core of Redox is a generic asynchronous API for executing any Redis command and providing a reply callback. The command method accepts a Redis command in the form of an STL vector of strings, and a callback to be invoked when a reply is received or if there is an error.

rdx.command<string>({"GET", "hello"}, [](Command<string>& c) {
  if(c.ok()) {
    cout << "Hello, async " << c.reply() << endl;
  } else {
    cerr << "Command has error code " << c.status() << endl;
  }
});

This statement tells redox to run the command GET hello. The <string> template parameter means that we want the reply to be put into a string and that we expect the server to respond with something that can be put into a string. The full list of reply types is listed in this document and covers convenient access to anything returned from the Redis protocol. The input vector can contain arbitrary binary data.

The second argument is a callback function that accepts a reference to a Command object of the requested reply type. The Command object contains the reply and any error information. If c.ok() is true, the expected reply is accessed from c.reply() (a string in this case). If c.ok() is false, then the error code is given by c.status(), which can report an error or nil reply, a reply of the wrong type, a send error, etc. The callback is guaranteed to be invoked exactly once, and the memory for the Command object is freed automatically once the callback returns.

Here is a simple example of running GET hello asynchronously ten times:

Redox rdx;

// Block until connected, localhost by default
if(!rdx.connect()) return 1;

auto got_reply = [](Command<string>& c) {
  if(!c.ok()) return;
  cout << c.cmd() << ": " << c.reply() << endl;
};

for(int i = 0; i < 10; i++) rdx.command<string>({"GET", "hello"}, got_reply);

// Do useful work
this_thread::sleep_for(chrono::milliseconds(10));

rdx.disconnect(); // Block until disconnected

The .command() method returns immediately, so this program doesn't wait for a reply from the server - it just pauses for ten milliseconds and then shuts down. If we want to shut down after we get all replies, we could do something like this:

Redox rdx;
if(!rdx.connect()) return 1;

int total = 10; // Number of commands to run
atomic_int count(0); // Number of replies expected
auto got_reply = [&](Command<string>& c) {
  count++;
  if(c.ok()) cout << c.cmd() << " #" << count << ": " << c.reply() << endl;
  if(count == total) rdx.stop(); // Signal to shut down
};

for(int i = 0; i < total; i++) rdx.command<string>({"GET", "hello"}, got_reply);

// Do useful work

rdx.wait(); // Block until shut down complete

This example tracks of how how many replies are received and signals the Redox instance to stop once they all process. We use an std::atomic_int to be safe because the callback is invoked from a separate thread. The stop() method signals Redox to shut down its event loop and disconnect from Redis. The wait() method blocks until stop() has been called and everything is brought down. The disconnect() method used earlier is just a call to stop() and then a call to wait().

Synchronous commands

Redox implements synchronous commands by running asynchronous commands and waiting on them with condition variables. That way, we can reap the benefits of pipelining between synchronous commands in different threads. The commandSync method provides a similar API to command, but instead of a callback returns a Command object when a reply is received.

Command<string>& c = rdx.commandSync<string>({"GET", "hello"});
if(c.ok()) cout << c.cmd() << ": " << c.reply() << endl;
c.free();

When using synchronous commands, the user is responsible for freeing the memory of the Command object by calling c.free(). The c.cmd() method just returns a string representation of the command (GET hello in this case).

Looping and delayed commands

We often want to run commands on regular invervals. Redox provides the commandLoop method to accomplish this. It is easier to use and more efficient than running individual commands in a loop, because it only creates a single Command object. commandLoop takes a command vector, a callback, and an interval (in seconds) to repeat the command. It then runs the command on the given interval until the user calls c.free().

Command<string>& cmd = rdx.commandLoop<string>({"GET", "hello"}, [](Command<string>& c) {
  if(c.ok()) cout << c.cmd() << ": " << c.reply() << endl;
}, 0.1);

this_thread::sleep_for(chrono::seconds(1));
cmd.free();
rdx.disconnect();

Finally, commandDelayed runs a command after a specified delay (in seconds). It does not return a command object, because the memory is automatically freed after the callback is invoked.

rdx.commandDelayed<string>({"GET", "hello"}, [](Command<string>& c) {
  if(c.ok()) cout << c.cmd() << ": " << c.reply() << endl;
}, 1);
this_thread::sleep_for(chrono::seconds(2));

Convenience methods

The four methods command, commandSync, commandLoop, and commandDelayed form the core of Redox's functionality. There are convenience methods provided that are simple wrappers over the core methods. Some examples of those are .get(), .set(), .del(), and .publish(). These methods are nice because they return simple values, and there are no Command objects or template parameters. However, they make strong assumptions about how to deal with errors (ignore or throw exceptions), and since their implementations are a few lines of code it is often easier to create custom convenience methods for your application.

Publisher / Subscriber

Redox provides an API for the pub/sub functionality of Redis. Publishing is done just like any other command using a Redox instance. There is a separate Subscriber class that receives messages and provides subscribe/unsubscribe and psubscribe/punsubscribe methods.

Redox rdx; Subscriber sub;
if(!rdx.connect() || !sub.connect()) return 1;

sub.subscribe("hello", [](const string& topic, const string& msg) {
  cout << topic << ": " << msg << endl;
});

for(int i = 0; i < 10; i++) {
  rdx.publish("hello", "this is a pubsub message");
  this_thread::sleep_for(chrono::milliseconds(500));
}

sub.disconnect(); rdx.disconnect();

strToVec and vecToStr

Redox provides helper methods to convert between a string command and a vector of strings as needed by its API. rdx.strToVec("GET foo") will return an std::vector<std::string> containing GET and foo as entries. rdx.vecToStr({"GET", "foo"}) will return the string GET foo.

No-Wait Mode

Redox provides a no-wait mode, which tells the event loop not to sleep in between processing events. It means that the event thread will run at 100% CPU, but it can greatly improve performance when critical. It is disabled by default and can be enabled with rdx.noWait(true);.

Reply types

These the available template parameters in redox and the Redis return types they can hold. If a given command returns an incompatible type you will get a WRONG_TYPE or NIL_REPLY status.

  • <redisReply*>: All reply types, returns the hiredis struct directly
  • <char*>: Simple Strings, Bulk Strings
  • <std::string>: Simple Strings, Bulk Strings
  • <long long int>: Integers
  • <int>: Integers (careful about overflow, long long int recommended)
  • <std::nullptr_t>: Null Bulk Strings, any other receiving a nil reply will get a NIL_REPLY status
  • <std::vector<std::string>>: Arrays of Simple Strings or Bulk Strings (in received order)
  • <std::set<std::string>>: Arrays of Simple Strings or Bulk Strings (in sorted order)
  • <std::unordered_set<std::string>>: Arrays of Simple Strings or Bulk Strings (in no order)

Installation

Instructions provided are for Ubuntu, but all components are platform-independent.

Build from source

Get the build environment and dependencies:

sudo apt-get install git cmake build-essential
sudo apt-get install libhiredis-dev libev-dev

Build the library:

mkdir build && cd build
cmake ..
make

Install into system directories (optional):

sudo make install

Build examples and test suite

Enable examples using ccmake or the following:

cmake -Dexamples=ON ..
make examples

To run the test suite, first make sure you have gtest set up, then:

cmake -Dtests=ON ..
make test_redox
./test_redox

Build documentation

Redox documentation is generated using doxygen.

cd docs
doxygen

The documentation can then be viewed in a browser at docs/html/index.html.

Build RPM and DEB packages

Basic support to build RPMs and DEBs is in the build system. To build them, issue the following commands:

mkdir release && cd release
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
make package

NOTE: To build RPM packages, you will need rpmbuild.

Contributing

Redox is in its early stages and I am looking for feedback and contributors to make it easier, faster, and more robust. Open issues on GitHub or message me directly.

Redox is not currently recommended for production use. It has no support yet for sentinels or clusters. Feel free to provide them!

Comments
  • Added multithreading test that hangs.

    Added multithreading test that hangs.

    Added a unit test in order to reproduce a deadlock when executing synchronous SET and DEL commands concurrently. (You might argue why we do synchronous commands concurrently, but that's the nature of our applications at the moment). Anyways, this looks like a serious issue that needs fixing. I have added some output files containing more information: pstack.output: Contains the pstack output of the test_redox process after it deadlocks. From the thread stacks, it's clear that one thread (#2) deadlocks in a wait of the condition_variable (line 43 in command.cpp). monitor.output: Contains the redis monitor output. This shows that the thread executing the SET commands executes to completion, but the thread executing the DEL command deadlocks. redis.output: Contains the output of executing SET and DEL commands concurrently using redis-cli. This doesn't show a deadlock (as expected).

    Executing the unit test reliably reproduces the deadlock on my machine.

    opened by bveldhoen 6
  • error loading shared libraries

    error loading shared libraries

    I need some help, i get this error after I compile the "hello world" example with the given command: g++ hello.cpp -o hello -std=c++11 -lredox -lev -lhiredis . So the compilation works good, without errors, but the execution gives me this error: "error while loading shared libraries: libredox.so.0: cannot open shared object file: No such file or directory"

    NOTE: Please try not to be mean when you explain to me the problem, I'm kind of a noob with those things. I am thinking that I didn't do the installation correctly.

    opened by c0d3Gems 5
  • Added lastError() to Command.

    Added lastError() to Command.

    Hi Hayk! Cool project! We're very seriously looking into using it. I have some ideas to add functionality based on our needs.

    As a start, I added a way to retrieve the Redis error by setting and exposing the last error that occurred while executing a command. Pls. let me know if it is in line with what you'd expect!

    Cheers, Bram Veldhoen

    opened by bveldhoen 4
  • update cmake for hiredis and libev

    update cmake for hiredis and libev

    when I try to build redox with custom hiredis location,

    mkdir build
    cd build
    cmake -DHIREDIS_ROOT_DIR=/home/fifilyu/workspace/third_party/hiredis ..
    

    I get the following error:

    [[email protected] build]$ make Scanning dependencies of target redox [ 8%] Building CXX object CMakeFiles/redox.dir/src/client.cpp.o In file included from /home/fifilyu/workspace/redox/src/client.cpp:23:0: /home/fifilyu/workspace/redox/include/redox/client.hpp:37:29: fatal error: hiredis/hiredis.h: No such file or directory

    opened by fifilyu 3
  • Add convenience classes for sets and hashes

    Add convenience classes for sets and hashes

    This enhancement is in line with issue #4. It adds two classes RedoxSet and RedoxHash to make it easier for a user to interact with the Redis backed using a simpler API. It also contains some whitespace clean-up.

    opened by esindril 2
  • AUTH command

    AUTH command

    Hi,

    I am using Redis with authentication so doing this: rdx.command({"auth", "mypassword"}, [](Command& c) { if(!c.ok()) return; cout << "Auth is done." << endl; });

    And seeing this error message: (2015-12.10 22:26:02) [Error] auth mypassword: Received reply of type 5, expected type 3.

    in spite of this error, the next commands are processed fine.

    Could you explain what is going on?

    Regards,

    Ian

    opened by ianpark 2
  • Add CMake find module for hiredis and libev

    Add CMake find module for hiredis and libev

    Contains:

    • add find module for hiredis and libev
    • make compilation fail if dependencies not met
    • improve portability of the build and use if possible the GNUInstallDirs package (>= CMake 2.8.5)
    opened by esindril 2
  • Ignoring strict aliasing in some usages of libev macros.

    Ignoring strict aliasing in some usages of libev macros.

    With compiler warning -Wstrict-aliasing enabled, usage of the macros ev_async_timer and ev_timer_init yields said warnings. Created templates to locally turn off these warnings. Compiles without warnings with gcc 4.9.2 and clang 3.5.

    opened by bveldhoen 2
  • ev_run without looping

    ev_run without looping

    Right now, the event loop calls ev_run(loop, EVRUN_NOWAIT) in a loop to maximize throughput. However, this hogs CPU on the event loop thread. If we just call ev_run(loop), such that the loop is internal to libev, throughput slows down up to 5x. Maybe use a CV to run the loop slower until we have a lot of commands to process.

    enhancement 
    opened by hmartiro 2
  • Removes logging

    Removes logging

    We remove logging because Clipper does it's own logging and error handling and I ran into a race condition in the Redox Logger (specifically, in the Logstream destructor).

    opened by dcrankshaw 1
  • Removing logging

    Removing logging

    Is there a config file where we can edit the logging level? Or alternatively is there a way to do so programmatically? This seems like the relevant class.

    opened by ujvl 1
  • Support commands returning vector<int>

    Support commands returning vector

    Some Redis commands, such as SMISMEMBER, return a list of integers. Trying to treat the reply as a vector doesn't work because the reply object has already parsed the data as integers, not strings. Supporting this is a simple change.

    I've added an example of the usage in examples/data_types.cpp. It now prints:

    Last 5 elements as a vector: 10 9 8 7 6 
    Last 5 elements as a hash: 10 8 7 9 6 
    Last 5 elements as a set: 10 6 7 8 9 
    Are {1, 2, 3, 4, 5} in the set, respectively: 0 1 1 0 1 
    
    opened by czinn 0
  • Is it possible to execute a transaction?

    Is it possible to execute a transaction?

    I'm assuming this variation shouldn't be a problem, but if I were to execute:

    rdx.command<string>({"multi"});
    rdx.command<string>({"SET", "number", "8"});
    rdx.command<string>({"publish", "info", "number_updates"});
    rdx.command<string>({"exec"});
    
    • would it execute as an atomic transaction with proper ordering?
    • If my string message is large (5 - 10 MB) would it cause any problems, I mean regarding performance and correctness? (by performance I'm only concerned if it has extra overhead compared to hiredis)
    opened by picanumber 0
  • Compilation issue

    Compilation issue

    Issue:

    [ 41%] Built target redox [ 50%] Linking CXX executable pub_sub /usr/bin/ld: libredox.so.0.3.0: undefined reference to `hi_malloc(unsigned long)' collect2: error: ld returned 1 exit status make[2]: *** [CMakeFiles/pub_sub.dir/build.make:87: pub_sub] Error 1 make[1]: *** [CMakeFiles/Makefile2:80: CMakeFiles/pub_sub.dir/all] Error 2 make: *** [Makefile:152: all] Error 2 Build complete (4 errors, 0 warnings): /home/laxman/eclipse-workspace/redox/build/default

    opened by RavadaLaxmanrao 0
  • how to add newly created pkg path

    how to add newly created pkg path

    I have just build the redox package . Now how do i link the headers and libraries in cmakelist.txt file?

    
    cmake_minimum_required(VERSION 2.6 FATAL_ERROR)
    
    project(pcl_visualizer_viewports)
    
    find_package(PCL 1.2 REQUIRED)
    
    include_directories(${PCL_INCLUDE_DIRS} /home/rob/redox/include/)
    link_directories(${PCL_LIBRARY_DIRS})
    add_definitions(${PCL_DEFINITIONS})
    
    add_executable (pcl_visualizer_demo pcl_visualizer_demo.cpp)
    target_link_libraries (pcl_visualizer_demo ${PCL_LIBRARIES} /home/rob/redox/build/)
    
    

    the error comming is:

    /usr/bin/ld: CMakeFiles/pcl_visualizer_demo.dir/pcl_visualizer_demo.cpp.o: undefined reference to symbol 'redisAsyncHandleWrite'
    //usr/lib/x86_64-linux-gnu/libhiredis.so.0.13: error adding symbols: DSO missing from command line
    collect2: error: ld returned 1 exit status
    CMakeFiles/pcl_visualizer_demo.dir/build.make:367: recipe for target 'pcl_visualizer_demo' failed
    make[2]: *** [pcl_visualizer_demo] Error 1
    
    opened by roboticsai 0
  • Support generation of a pkg-config redox.pc file

    Support generation of a pkg-config redox.pc file

    Hello,

    This PR adds the generation of a pkg-config file for redox when calling make/ninja/etc. install.

    In brief, pkg-config eases the utilization of software libraries by allowing the developers of a library to define how the library should be used — instead of letting end users struggling with it. It therefore allows redox users to link/include it without having to know internal details such as include path, the library name or the dependencies that should also be included/linked.

    Some build systems support the definition of dependencies via pkg-config (e.g., this is the main way to define dependencies in Meson, CMake can use pkg-config dependencies).

    The following two examples show how a pkg-config redox file could help users by compiling the example located in examples/basic.cpp — after installing the new redox version.

    Makefile usage example

    basic: basic.cpp
    	g++ -o basic `pkg-config --cflags --libs redox` basic.cpp
    

    Meson usage example

    project('basic', 'cpp')
    redox = dependency('redox')
    executable('basic', 'basic.cpp', dependencies: redox)
    
    opened by mpoquet 0
Owner
Hayk Martiros
Autonomy @ Skydio http://haykmartiros.com/
Hayk Martiros
redis-cpp is a header-only library in C++17 for Redis (and C++11 backport)

redis-cpp - lightweight C++ client library for Redis redis-cpp is a C++17 library for executing Redis commands with support for pipelines and the publ

null 76 Nov 22, 2022
A redis module, similar to redis zset, but you can set multiple scores for each member to support multi-dimensional sorting

TairZset: Support multi-score sorting zset Introduction Chinese TairZset is a data structure developed based on the redis module. Compared with the na

Alibaba 58 Nov 21, 2022
Minimalistic C client for Redis >= 1.2

This Readme reflects the latest changed in the master branch. See v1.0.0 for the Readme and documentation for the latest release (API/ABI history). HI

Redis 5.5k Dec 5, 2022
C++11 Lightweight Redis client: async, thread-safe, no dependency, pipelining, multi-platform

C++11 Lightweight Redis client: async, thread-safe, no dependency, pipelining, multi-platform

Simon Ninon 1k Dec 3, 2022
Redis client written in C++

redis-plus-plus Overview Features Branches Installation Install hiredis Install redis-plus-plus Run Tests (Optional) Use redis-plus-plus In Your Proje

null 991 Nov 22, 2022
Aredis - a clean redis C++ client

aredis a clean redis C++ client redis_conn rc; if (rc.connect("127.0.0.1"/*host*/, 6379/*port*/, nullptr/*password*/, 1/*db*/)) { redis_command cmd;

Evan 26 Dec 10, 2021
A C++ Redis client

redis3m A C++ Redis client, born to bring my experience using Redis and C++ on a opensource library. Main goals Provide a simple and efficient wrapper

Luca Marturana 184 Apr 21, 2022
The PostgreSQL client API in modern C++

C++ client API to PostgreSQL {#mainpage} Dmitigr Pgfe (PostGres FrontEnd, hereinafter referred to as Pgfe) - is a C++ client API to PostgreSQL servers

Dmitry Igrishin 136 Nov 9, 2022
Trilogy is a client library for MySQL-compatible database servers, designed for performance, flexibility, and ease of embedding.

Trilogy is a client library for MySQL-compatible database servers, designed for performance, flexibility, and ease of embedding.

GitHub 475 Nov 15, 2022
The official C++ client API for PostgreSQL.

libpqxx Welcome to libpqxx, the C++ API to the PostgreSQL database management system. Home page: http://pqxx.org/development/libpqxx/ Find libpqxx on

Jeroen Vermeulen 705 Dec 3, 2022
Beryl-cli is a client for the BerylDB database server

Beryl-cli is a client for the BerylDB database server. It offers multiple commands and is designed to be fast and user-friendly.

BerylDB 11 Oct 9, 2022
C++ client library for PostgreSQL

Welcome to taoPQ taoPQ is a lightweight C++ client library for accessing a PostgreSQL➚ database. It has no dependencies beyond libpq➚, the C applicati

The Art of C++ 231 Nov 30, 2022
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

Overview GridDB is Database for IoT with both NoSQL interface and SQL Interface. Please refer to GridDB Features Reference for functionality. This rep

GridDB 1.9k Nov 24, 2022
ObjectBox C and C++: super-fast database for objects and structs

ObjectBox Embedded Database for C and C++ ObjectBox is a superfast C and C++ database for embedded devices (mobile and IoT), desktop and server apps.

ObjectBox 151 Dec 2, 2022
❤️ SQLite ORM light header only library for modern C++

SQLite ORM SQLite ORM light header only library for modern C++ Status Branch Travis Appveyor master dev Advantages No raw string queries Intuitive syn

Yevgeniy Zakharov 1.7k Nov 29, 2022
Modern cryptography for PostgreSQL using libsodium.

pgsodium pgsodium is an encryption library extension for PostgreSQL using the libsodium library for high level cryptographic algorithms. pgsodium can

Michel Pelletier 355 Nov 26, 2022
OrioleDB – building a modern cloud-native storage engine

OrioleDB is a new storage engine for PostgreSQL, bringing a modern approach to database capacity, capabilities and performance to the world's most-loved database platform.

OrioleDB 1.3k Dec 1, 2022
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large scale graphs with dozens of billions of vertices (nodes) and trillions of edges, with milliseconds of latency.

vesoft inc. 833 Nov 26, 2022
SiriDB is a highly-scalable, robust and super fast time series database

SiriDB is a highly-scalable, robust and super fast time series database. Build from the ground up SiriDB uses a unique mechanism to operate without a global index and allows server resources to be added on the fly. SiriDB's unique query language includes dynamic grouping of time series for easy analysis over large amounts of time series.

SiriDB 468 Nov 10, 2022