Pelikan is Twitter's framework for developing cache services. It is:
Fast: Pelikan provides high-throughput and low-latency caching solutions.
Reliable: Pelikan is designed for large-scale deployment and the implementation is informed by our operational experiences.
Modular: Pelikan is a framework for rapidly developing new caching solutions by focusing on the inherent architectural similarity between caching services and providing reusable low-level components.
After years of using and working on various cache services, we built a common framework that reveals the inherent architectural similarity among them.
By creating well-defined modules, most of the low-level functionalities are reused as we create different binaries. The implementation learns from our operational experiences to improve performance and reliability, and leads to software designed for large-scale deployment.
The framework approach allows us to develop new features and protocols quickly.
Currently Pelikan yields three main products, all of which are backends/servers.
pelikan_twemcache: a Twemcache replacement
pelikan_slimcache: a Memcached-like server with ultra-low memory overhead- compared to Memcached/Redis, the per-key overhead is reduced by up to 90%
pelikan_pingserver: an over-engineered, production-ready ping server useful as a tutorial and for measuring baseline RPC performance
pelikan_segcache: a Memcached-like server with extremely high memory efficiency and excellent core scalability. See our NSDI'21 paper for design and evaluation details.
pelikan_segcache_rs: a Rust implementation of
pelikan_segcachewhich includes TLS support and is the preferred Segcache implementation.
pelikan_pingserver_rs: a Rust implementation of
pelikan_pingserverwhich includes TLS support.
- runtime separation of control and data plane
- predictably low latencies via lockless data structures, worker never blocks
- per-module config options and metrics that can be composed easily
- multiple storage and protocol implementations, easy to further extend
- low-overhead command logger for hotkey and other important data analysis
- platform: Mac OS X or Linux
- build tools:
- (optional) unit testing framework:
check (>=0.10.0). See below.
git clone https://github.com/twitter/pelikan.git mkdir _build && cd _build cmake .. make -j
The executables can be found under
_bin/ (under build directory)
To run all the tests, including those on
To skip building tests, replace the
cmake step with the following:
cmake -DCHECK_WORKING=off ..
To compile and run tests, you will have to install
[check]. Please follow instructions in the project.
Note: we highly recommend installing the latest version of
check from source, as there are, unfortunately, a linker bug in packages installed by the current versions of
brew (OS X),
Ubuntu LTS. The bug does not affect building executables.
Building Pelikan (Rust)
- Rust stable toolchain
- C toolchain:
llvm/clang (>= 7.0)
- Build tools:
cmake (>= 3.2)
git clone https://github.com/twitter/pelikan.git cd pelikan cargo build --release
pelikan_twemcache as an example, other executables are highly similar.
To get info of the service, including usage format and options, run:
To launch the service with default settings, simply run:
To launch the service with the sample config file, run:
NOTE: The Rust servers use TOML configuration files, sample configs can be found in the
config folder of this repository.
You should be able to try out the server using an existing memcached client, or simply with
$ telnet localhost 12321 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. set foo 0 0 3 bar STORED
admin port for all non-data commands.
$ telnet localhost 9999 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. version VERSION 0.1.0 stats STAT pid 54937 STAT time 1459634909 STAT uptime 22 STAT version 100 STAT ru_stime 0.019172 ...
Pelikan is file-first when it comes to configurations, and currently is config-file only. You can create a new config file following the examples included under the
Tip: to get a list of config options for each executable, use
Stay in touch
- Join our project chat on for questions and discussions
- Follow us on Twitter: @pelikan_cache
- Visit http://pelikan.io
If you want to submit a patch, please follow these steps:
- create a new issue
- fork on github & clone your fork
- create a feature branch on your fork
- push your feature branch
- create a pull request linked to the issue
We have made progress and are actively working on documentation, and will put it on our website. Meanwhile, check out the current material under
This software is licensed under the Apache 2.0 license, see LICENSE for details.