Pelikan is Twitter's unified cache backend

Overview

Pelikan

Pelikan is Twitter's framework for developing cache services. It is:

  • Fast: Pelikan provides high-throughput and low-latency caching solutions.

  • Reliable: Pelikan is designed for large-scale deployment and the implementation is informed by our operational experiences.

  • Modular: Pelikan is a framework for rapidly developing new caching solutions by focusing on the inherent architectural similarity between caching services and providing reusable low-level components.

License: Apache-2.0 Build Status: C Build Status: Rust Fuzz Status: Rust Zulip Chat

Website | Chat

Content

Overview

After years of using and working on various cache services, we built a common framework that reveals the inherent architectural similarity among them.

By creating well-defined modules, most of the low-level functionalities are reused as we create different binaries. The implementation learns from our operational experiences to improve performance and reliability, and leads to software designed for large-scale deployment.

The framework approach allows us to develop new features and protocols quickly.

Products

Currently Pelikan yields three main products, all of which are backends/servers.

  • pelikan_twemcache: a Twemcache replacement
  • pelikan_slimcache: a Memcached-like server with ultra-low memory overhead- compared to Memcached/Redis, the per-key overhead is reduced by up to 90%
  • pelikan_pingserver: an over-engineered, production-ready ping server useful as a tutorial and for measuring baseline RPC performance
  • [Experimental]pelikan_segcache: a Memcached-like server with extremely high memory efficiency and excellent core scalability. See our NSDI'21 paper for design and evaluation details.
  • [Experimental]pelikan_segcache_rs: a Rust implementation of pelikan_segcache which includes TLS support and is the preferred Segcache implementation.
  • [Experimental]pelikan_pingserver_rs: a Rust implementation of pelikan_pingserver which includes TLS support.

Features

  • runtime separation of control and data plane
  • predictably low latencies via lockless data structures, worker never blocks
  • per-module config options and metrics that can be composed easily
  • multiple storage and protocol implementations, easy to further extend
  • low-overhead command logger for hotkey and other important data analysis

Building Pelikan

Requirement

  • platform: Mac OS X or Linux
  • build tools: cmake (>=2.8)
  • compiler: gcc (>=4.8) or clang (>=3.1)
  • (optional) unit testing framework: check (>=0.10.0). See below.

Build

git clone https://github.com/twitter/pelikan.git
mkdir _build && cd _build
cmake ..
make -j

The executables can be found under _bin/ (under build directory)

To run all the tests, including those on ccommon, run:

make test

To skip building tests, replace the cmake step with the following:

cmake -DCHECK_WORKING=off ..

Install check

To compile and run tests, you will have to install [check]. Please follow instructions in the project.

Note: we highly recommend installing the latest version of check from source, as there are, unfortunately, a linker bug in packages installed by the current versions of brew (OS X), CentOS and Ubuntu LTS. The bug does not affect building executables.

Building Pelikan (Rust)

Requirement

  • Rust stable toolchain
  • C toolchain: llvm/clang (>= 7.0)
  • Build tools: cmake (>= 3.2)

Build

git clone https://github.com/twitter/pelikan.git
cd pelikan
cargo build --release

Tests

cargo test

Usage

Using pelikan_twemcache as an example, other executables are highly similar.

To get info of the service, including usage format and options, run:

_bin/pelikan_twemcache -h

To launch the service with default settings, simply run:

_bin/pelikan_twemcache

To launch the service with the sample config file, run:

_bin/pelikan_twemcache config/twemcache.conf

NOTE: The Rust servers use TOML configuration files, sample configs can be found in the config folder of this repository.

You should be able to try out the server using an existing memcached client, or simply with telnet.

$ telnet localhost 12321
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
set foo 0 0 3
bar
STORED

Attention: use admin port for all non-data commands.

$ telnet localhost 9999
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
version
VERSION 0.1.0
stats
STAT pid 54937
STAT time 1459634909
STAT uptime 22
STAT version 100
STAT ru_stime 0.019172
...

Configuration

Pelikan is file-first when it comes to configurations, and currently is config-file only. You can create a new config file following the examples included under the config directory.

Tip: to get a list of config options for each executable, use -c option:

_bin/pelikan_twemcache -c

Community

Stay in touch

Contributing

Please take a look at our community manifesto and coding style guide.

If you want to submit a patch, please follow these steps:

  1. create a new issue
  2. fork on github & clone your fork
  3. create a feature branch on your fork
  4. push your feature branch
  5. create a pull request linked to the issue

Documentation

We have made progress and are actively working on documentation, and will put it on our website. Meanwhile, check out the current material under docs/

License

This software is licensed under the Apache 2.0 license, see LICENSE for details.

Issues
  • end-to-end tests for pelikan servers

    end-to-end tests for pelikan servers

    We need some end-to-end test cases and harness to make sure the servers behave as expected. Initially looking into the most basic, deterministic scenarios. In these cases a sequence of request/response pairs will be used, the requests are sent to the server and the responses have to match exactly.

    Since we don't have an official client library in Pelikan just yet ( @kevyang wrote one for memcached, but it needs a bit of polishing which I will do soon), we can start with sending serialized strings and treating responses the same way. Under this approach, the only part in the response that needs attention is the cas value for gets commands. Exact values can be ensured if we locally track the number of writes sent to the server; alternatively we can use regex to parse out the responses.

    I'm proposing separating the test cases from the harness to maximize readability and flexibility.

    Test cases can be stored in a file with some delimiter between individual request/response pairs, e.g.:

    >>>
    set foo 0 0 3
    bar
    <<<
    STORED
    
    >>>
    gets foo
    <<<
    VALUE foo 0 3 1
    bar
    END
    

    And test harness can read from such a file, as well as a server config file to learn the whereabouts of the server, and play the tests against it. I've looked at test harness written in Perl, Tcl, Bash and Python, and I lean toward Python for readability, given integration tests don't have particular performance requirements. Perl, Tcl and Python are pretty much universally installed on all POSIX compatible systems (not to mention bash), so availability is not a problem for any of them. Anything else that's worth looking at, or is there an argument for/against any of them?

    /cc @seppo0010

    enhancement 
    opened by thinkingfish 10
  • flaky test in travis?

    flaky test in travis?

    $ cd ../test && python -m unittest discover
    
    F
    
    ======================================================================
    
    FAIL: test_miss (integration.test_twemcache_basic.TwemcacheBasicTest)
    
    ----------------------------------------------------------------------
    
    Traceback (most recent call last):
    
      File "/home/travis/build/twitter/pelikan/test/integration/test_twemcache_basic.py", line 7, in test_miss
    
        self.assertMetrics(('request_parse', 1), ('get', 1), ('get_key_miss', 1))
    
      File "/home/travis/build/twitter/pelikan/test/integration/base.py", line 40, in assertMetrics
    
        stats.get(k, None),
    
    AssertionError: Expected request_parse to be 1, got 0 instead
    
    ----------------------------------------------------------------------
    
    Ran 1 test in 0.005s
    
    FAILED (failures=1)
    
    The command "cd ../test && python -m unittest discover" exited with 1.
    
    Done. Your build exited with 1.
    

    This occurred for all tests last night, went away this morning, and has now returned.

    opened by thinkingfish 8
  • coding style guide update

    coding style guide update

    1. remove recommendation about one-var-per-line;
    2. add section about forward declaration;
    3. relax wording on nested levels;
    4. update how to break down a long line to reflect our actual practice- 8-char indentation after the first line;
    5. clarify when to and why use of stdint types;
    6. one typo fixed.
    opened by thinkingfish 7
  • Fix cleaning after partial request

    Fix cleaning after partial request

    Using partial request, like from rpc-perf, causes first request buffer not to be returned to requests pool.

    This commits adds function which will return this buffer to pool.

    opened by jschmieg 6
  • pelikan changes needed to refactor option/setup

    pelikan changes needed to refactor option/setup

    This works with (locally merged) ccommon PR of the same purpose.

    It's a lot of changes. But the point is basically illustrated in the main.c's, note it is noticeably shorter and more regular to setup an application (https://github.com/thinkingfish/pelikan/blob/option_refactor/src/twemcache/main.c vs. the current way https://github.com/thinkingfish/pelikan/blob/master/src/twemcache/main.c)

    We also have prettier option printing now:

    
    Setting & Default Values:
    NAME                            TYPE            DEFAULT              DESCRIPTION
    daemonize                       boolean         no                   daemonize the process
    pid_filename                    string          NULL                 file storing the pid
    admin_host                      string          NULL                 admin interfaces listening on
    admin_port                      string          9999                 admin port
    admin_timeout                   unsigned int    100                  evwait timeout
    admin_nevent                    unsigned int    1024                 evwait max nevent returned
    admin_tw_tick                   unsigned int    10                   timing wheel tick size (ms)
    admin_tw_cap                    unsigned int    1000                 # ticks in timing wheel
    admin_tw_ntick                  unsigned int    100                  max # ticks processed at once
    server_host                     string          NULL                 interfaces listening on
    server_port                     string          12321                port listening on
    server_timeout                  unsigned int    100                  evwait timeout
    server_nevent                   unsigned int    1024                 evwait max nevent returned
    worker_timeout                  unsigned int    100                  evwait timeout
    worker_nevent                   unsigned int    1024                 evwait max nevent returned
    allow_flush                     boolean         no                   allow flushing on the data port
    klog_file                       string          NULL                 command log file
    klog_backup                     string          NULL                 command log backup file
    klog_nbuf                       unsigned int    2097152              command log buf size
    klog_intvl                      unsigned int    100                  command log flush interval in ms
    klog_sample                     unsigned int    100                  command log sample ratio
    klog_max                        unsigned int    1073741824           max klog file size - disk usage doubles with backup
    request_poolsize                unsigned int    4096                 request pool size
    response_poolsize               unsigned int    4096                 response pool size
    slab_size                       unsigned int    1048576              Slab size
    slab_mem                        unsigned int    67108864             Max memory by slabs (byte)
    slab_prealloc                   boolean         yes                  Pre-allocate slabs at setup
    slab_evict_opt                  unsigned int    1                    Eviction strategy
    slab_use_freeq                  boolean         yes                  Use items in free queue?
    slab_profile                    string          NULL                 Specify entire slab profile
    slab_item_min                   unsigned int    44                   Minimum item size
    slab_item_max                   unsigned int    1048544              Maximum item size
    slab_item_growth                double          1.250000             Slab class growth factor
    slab_use_cas                    boolean         yes                  Store CAS value in item
    slab_hash_power                 unsigned int    16                   Power for lookup hash table
    array_nelem_delta               unsigned int    16                   max nelem delta during expansion
    buf_init_size                   unsigned int    16384                default size when buf is created
    buf_poolsize                    unsigned int    0                    buf pool size
    dbuf_max_power                  unsigned int    8                    max number of doubling
    debug_log_level                 unsigned int    4                    debug log level
    debug_log_file                  string          NULL                 debug log file
    debug_log_nbuf                  unsigned int    4194304              debug log buf size
    debug_log_intvl                 unsigned int    100                  debug log flush interval in ms (only applies if buf size > 0)
    buf_sock_poolsize               unsigned int    0                    buf_sock limit
    tcp_backlog                     unsigned int    128                  tcp conn backlog limit
    tcp_poolsize                    unsigned int    0                    tcp conn pool size
    
    opened by thinkingfish 6
  • Move _post_read and _admin_post_read to specific impl directories

    Move _post_read and _admin_post_read to specific impl directories

    This is to address #44 , to support pingserver and redis.

    But this also has the undesirable side-effect of duplicating the code in:

    • twemcache/worker_helper.c and slimcache/worker_helper.c
    • twemcache/admin_helper.c and slimcache/admin_helper.c
    opened by sagar0 6
  • fetching large amount of data

    fetching large amount of data

    Hi, I used twemcache for caching large amount of data(consisting of approximately 111000 characters), however I cannot fetch more than 80000 characters when I try to get the value back.

    opened by den1zk 5
  • To fork or not to fork?

    To fork or not to fork?

    Shall we each clone the repo under our own account, and submit pull requests from there, or shall we continue to create branches under the main repo?

    Only those with write-access to the main repo can create branches, which means the workflow is different for those who have this access and those who don't. On the other hand, it is easier to see what branches are outstanding when they are all in one place, before a PR is created, and it doesn't require we modify how we setup our local repo yet again.

    question 
    opened by thinkingfish 5
  • make twemcache deal with partial request value

    make twemcache deal with partial request value

    This is the first diff to address issue #122

    Subsequent diff will add more unit tests, as well as handle partial value when writing responses (when wbuf is full)

    opened by thinkingfish 4
  • disable partial-parsing and request-caching

    disable partial-parsing and request-caching

    This addresses the bug in https://github.com/twitter/pelikan/issues/98

    Note that now the data field is not used at all in handling memcached requests. But in general it is nice to have a wildcard field for generic structures like buf_sock, since users often want context affiliated with each socket. So not using it doesn't necessarily mean we should take out the data field from the underlying library.

    opened by thinkingfish 4
  • feat: #368 make admin protocol parsing tolerant of leading/trailing whitespace

    feat: #368 make admin protocol parsing tolerant of leading/trailing whitespace

    fixes #368 by trimming the command of leading/trailing whitespace before matching

    I'm assuming this is what is intended based on reading the C code, but I might be wrong

    opened by hderms 3
  • proxy: stateful request routing

    proxy: stateful request routing

    As implemented in #416, the proxy simply routes requests to backends randomly. For a cache proxy, we will need to support some stateful request routing (consistent hashing across the backends). This will also require considerations to be made for fan-out for multi-key get requests for memcache protocol.

    opened by brayniac 0
  • rust/core: move admin features out of proxy and server

    rust/core: move admin features out of proxy and server

    The admin features in the proxy and server core modules are essentially duplicated. This can be factored out into its own module (eg: core/admin) to reduce duplication.

    opened by brayniac 0
  • proxy: handle in-flight requests during shutdown

    proxy: handle in-flight requests during shutdown

    As implemented in #416 the proxy makes no attempt to gracefully handle in-flight requests during shutdown. We should make changes to the logic so that in-flight requests are allowed to complete (with some timeout), and responses routed back to the clients before the connections are terminated.

    opened by brayniac 0
  • proxy: improved service discovery

    proxy: improved service discovery

    The proxy implemented in #416 needs to handle service discovery. The zookeeper service discovery included in the initial implementation is very Aurora/Finagle specific.

    One option to improve this is to expand on the admin commands and send topology changes from a sidecar process to the proxy via the admin port. This would allow for custom handling of service discovery without the need to support a wide variety of approaches within the proxy.

    As part of this, we need to also be able to handle changes to the serverset and ability to provide some key to hash on (eg shard id)

    opened by brayniac 0
  • proxy: add mTLS support

    proxy: add mTLS support

    The ping proxy implemented in #416 does not have mTLS support. We will want to add this to the proxy so we can have full end-to-end service to service mTLS support.

    opened by brayniac 0
  • refactor: use iterators in seg hashtable

    refactor: use iterators in seg hashtable

    As noted in #414, we can use iterators to reduce code duplication within the hashtable module.

    This change adds two iterator types to the hashtable, allowing us to iterate through item info entries for a given hash.

    opened by brayniac 2
Releases(0.1.2)
Owner
Twitter
Twitter 💙 #opensource
Twitter
A late bound, hope-for-the-best dyld shared cache extractor

yolo_dsc A late bound, hope-for-the-best dyld shared cache extractor why? There are other dsc_extract utilities. They usually require some combination

Rick Mark 14 Mar 17, 2022
null 225 May 14, 2022
Unified Gaussian Preintegrated Measurements (UGPMs)

This repository provides the C++ implementation of the preintegration methods presented in our RSS'21 paper titled Continuous Integration over SO(3) for IMU Preintegration (with video here ). If you are using that code for any purpose, please cite the corresponding work as explained at the end of this page.

Centre for Autonomous Systems, University of Technology Sydney 47 May 2, 2022
KernInfra, a unified kernel operation framework

KernInfra KernInfra is a developer-friendly kernel read-write framework. Why KernInfra KernInfra is built to address the following engineering issues:

null 23 Apr 29, 2022
Provide a unified trading framework and connectors to popular trading venues

Boost.connector Provide a unified trading framework and connectors to popular trading venues This is currently NOT an official Boost library. Introduc

Richard Hodges 6 Nov 24, 2021
XTAO Unified Distributed Storage

Anna - A branch project from CEPH Anna is a XTAO project branched from CEPH distributed storage. CEPH is a nice opensource project for unified distrib

XTAO Technolgy 3 Nov 12, 2021
StarPU: A Unified Runtime System for Heterogeneous Multicore Architectures

StarPU: A Unified Runtime System for Heterogeneous Multicore Architectures coverage report What is StarPU? StarPU is a runtime system that offers supp

null 8 Mar 24, 2022
A little experiment to have multicore OCaml with effects on iOS. In particular the GCD backend for eio.

Effects on iOS Very WIP & Experimental Overview A little experiment to have multicore OCaml with effects on iOS (currently just the simulator). In par

Patrick Ferris 11 May 11, 2022
A backend implementation for xdg-desktop-portal

xdg-desktop-portal-lxqt A backend implementation for xdg-desktop-portal that is using Qt/KF5. Building xdg-desktop-portal-lxqt Dependencies: xdg-deskt

LXQt 7 Apr 19, 2022
Stock exchange simulator made in Swing using Java with logic backend in C++ giving it faster load time and better data control

StockSimulator Stock exchange simulator made in Swing using Java with logic backend in C++ giving it faster load time and better data control Features

Dušan Todorović 0 Mar 1, 2022
OneFlow Backend For Triton Inference Server

Triton Inference Server OneFlow Backend

ZeKai Zhou 3 Jan 6, 2022
A Direct3D9 to Vulkan layer using the DXVK backend. [Upstreamed to DXVK]

This work has been upstreamed and is continuing development there This repo is only open for the remaining issues on the tracker https://github.com/do

Joshie 810 Apr 30, 2022
A low-latency LRU approximation cache in C++ using CLOCK second-chance algorithm. Multi level cache too. Up to 2.5 billion lookups per second.

LruClockCache Low-latency LRU approximation cache in C++ using CLOCK second-chance algorithm. (see wiki for details) using MyKeyType = std::string; us

Hüseyin Tuğrul BÜYÜKIŞIK 26 Apr 22, 2022
VE Font Cache is a single header-only GPU font rendering library designed for game engines.

VE Font Cache is a single header-only GPU font rendering library designed for game engines. It aims to: Be fast and simple to integrate. Take advantag

Xi Ma Chen 339 May 3, 2022
RdpCacheStitcher is a tool that supports forensic analysts in reconstructing useful images out of RDP cache bitmaps.

RdpCacheStitcher is a tool that supports forensic analysts in reconstructing useful images out of RDP cache bitmaps. Using raw RDP cache tile bitmaps extracted by tools like e.g. ANSSI's BMC-Tools as input, it provides a graphical user interface and several placement heuristics for stitching tiles together so that meaningful images or even full screenshots can be reconstructed.

Bundesamt für Sicherheit in der Informationstechnik 161 May 2, 2022
In-kernel cache based on eBPF.

BMC BMC (BPF Memory Cache) is an in-kernel cache for memcached. It enables runtime, crash-safe extension of the Linux kernel to process specific memca

Orange 332 May 11, 2022
Insomniac games cache simulation tool plugin for UE4

Insomniac Games CacheSim plugin for UE4 This plugin for Unreal Engine 4 lets you use the Insomniac Games Cache Simulation tool to detect cache misses

Toni Rebollo Berná 26 Jan 19, 2022
deserter is the first of its kind targeted DNS cache poisoner

deserter is the first of its kind targeted DNS cache poisoner. It is capable of DNS cache poisoning without bruteforcing the target ID and source port - instead, it sniffs out DNS probes and uses the information inside to craft poisoned responses and send them back to the target.

null 72 May 8, 2022
deserter is a targeted DNS cache poisoner.

deserter is a targeted DNS cache poisoner. It is capable of DNS cache poisoning without bruteforcing the target ID and source port - instead, it sniffs out DNS probes and uses the information inside to craft poisoned responses and send them back to the target.

null 72 May 8, 2022
A command-line tool to extract dylib files from the dyld shared cache file.

DyldExtractor A command-line tool to extract dylib files from the dyld shared cache file. Starting with macOS 11, standalone binaries of system librar

Cyandev 8 Mar 12, 2022
A late bound, hope-for-the-best dyld shared cache extractor

yolo_dsc A late bound, hope-for-the-best dyld shared cache extractor why? There are other dsc_extract utilities. They usually require some combination

Rick Mark 14 Mar 17, 2022
A CLI for extracting libraries from Apple's dyld shared cache file

dyld-shared-cache-extractor As of macOS Big Sur, instead of shipping the system libraries with macOS, Apple ships a generated cache of all built in dy

Keith Smiley 150 May 2, 2022
CMake checks cache helper modules – for fast CI CMake builds!

cmake-checks-cache Cross platform CMake projects do platform introspection by the means of "Check" macros. Have a look at CMake's How To Write Platfor

Cristian Adam 56 Mar 7, 2022
a unified framework for modeling chemically reactive systems

Reaktoro Reaktoro is a unified framework for modeling chemically reactive systems. It provides methods for chemical equilibrium and kinetic calculatio

Reaktoro 95 May 14, 2022
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

Cossack Labs 1.5k May 10, 2022
The official Open-Asset-Importer-Library Repository. Loads 40+ 3D-file-formats into one unified and clean data structure.

Open Asset Import Library (assimp) A library to import and export various 3d-model-formats including scene-post-processing to generate missing render

Open Asset Import Library 7.8k May 11, 2022
Convenient unified display of the most relevant technical and tag data for video and audio files.

MediaInfoLib README MediaInfo(Lib) is a convenient unified display of the most relevant technical and tag data for video and audio files. MediaInfoLib

MediaArea 434 May 7, 2022
null 225 May 14, 2022
XMRig is a high performance, open source, cross platform RandomX, KawPow, CryptoNight and AstroBWT unified CPU/GPU miner

XMRig is a high performance, open source, cross platform RandomX, KawPow, CryptoNight and AstroBWT unified CPU/GPU miner and RandomX benchmark. Official binaries are available for Windows, Linux, macOS and FreeBSD.

null 6.9k May 13, 2022