Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Last update: Jan 9, 2023

Overview

Kvrocks is an open-source key-value database which is based on rocksdb and compatible with Redis protocol. Intention to decrease the cost of memory and increase the capability while compared to Redis. The design of replication and storage was inspired by rocksplicator and blackwidow.

Kvrocks has the following key features:

Redis protocol, user can use redis client to visit the kvrocks
Namespace, similar to redis db but use token per namespace
Replication, async replication using binlog like MySQL
High Available, supports redis sentinel to failover when master or slave was failed

Thanks for designers @田凌宇 and @范世丽 contribute the kvrocks logo for us.

Who uses kvrocks

Tickets a pull reqeust to let us known that you're using kvrocks and add your logo to README

Building kvrocks

requirements

g++ (required by c++11, version >= 4.8)
autoconf automake libtool snappy

Build

NOTE: You should install the snappy first:

# Centos/Redhat
sudo yum install -y epel-release && sudo yum install -y git gcc gcc-c++ make snappy snappy-devel autoconf automake libtool which gtest gtest-devel

# Ubuntu
sudo apt-get install gcc g++ make libsnappy-dev autoconf automake libtool which libgtest-dev

# MACOSX
brew install snappy googletest

It is as simple as:

$ git clone --recursive https://github.com/kvrockslabs/kvrocks.git
$ cd kvrocks
$ make -j4

Running kvrocks

$ ./src/kvrocks -c kvrocks.conf

Running test cases

NOTE: You should install the googletest first

make test

Supported platforms

centos 6/7
ubuntu
macosx

Try kvrocks using Docker

$ docker run -it -p 6666:6666 kvrocks/kvrocks
$ redis-cli -p 6666

127.0.0.1:6666> get a
(nil)

Namespace

namespace was used to isolate data between users. unlike all the redis databases can be visited by requirepass, we use one token per namespace. requirepass was regraded as admin token, only admin token allows to access the namespace command, as well as some commands like config, slaveof, bgsave, etc…

namespace del ns1 OK ">

# add token
127.0.0.1:6666>  namespace add ns1 mytoken
OK

# update token
127.0.0.1:6666> namespace set ns1 new_token
OK

# list namespace
127.0.0.1:6666> namespace get *
1) "ns1"
2) "new_token"
3) "__namespace"
4) "foobared"

# delete namespace
127.0.0.1:6666> namespace del ns1
OK

DOCs

Migrate tools

migrate from redis to kvrocks, use redis-migrate-tool which was developed by vipshop
migrate from kvrocks to redis. use kvrocks2redis in build dir

Performance

Hardware

CPU: 48 cores Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
Memory: 32 GiB
NET: Intel Corporation I350 Gigabit Network Connection
DISK: 2TB NVMe Intel SSD DC P460

Benchmark Client: multi-thread redis-benchmark(unstable branch)

1. Commands QPS

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns / 128 payload

latency: 99.9% < 10ms

2. QPS on different payloads

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns

latency: 99.9% < 10ms

3. QPS on different workers

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns / 128 payload

latency: 99.9% < 10ms

License

Kvrocks is under the BSD-3-Clause license. See the LICENSE file for details.

Comments

Implement the data structure and partial commands of the Redis stream
Available commands:

XADD

XDEL

XINFO STREAM

XLEN

XRANGE

XREAD

XREVRANGE

XTRIM

Design notes Link to the Redis data type description: https://redis.io/docs/manual/data-types/streams/ A stream is a sequence of entries. Each entry has a unique ID in the form "1-2" where two 64-bit numbers are divided by a hyphen. By default, the first number is set to a millisecond timestamp, and the second one is a so-called sequence number - for cases when more than one entry was added at the same millisecond. The value of an entry is a set of key-value pairs. In case of the command: XADD s1 1-0 key1 val1 key2 val2 the ID is 1-0 and the value is key1 val1 key2 val2. In RocksDB entries are represented as key-value pairs where the key is formed as: key | version | entry-ID-milliseconds-value | entry-ID-sequence-number-value and the value is encoded as: key1-length(fixed-32) | key1 | val1-length(fixed-32) | val1 | key2-length(fixed-32) | key2 | val2-length(fixed-32) | val2. Thanks to the structure of a key, all entries in a stream are sorted in chronological order. As for value decoding: this is the first idea that came to my mind and maybe it's not very efficient because has an overhead (4 bytes on every argument). Why did I introduce such a weird encoding scheme? Because if you are reading entries, Redis responds not with a single string:

1) 1) "s1" 2) 1) 1) "1-0" 2) 1) "key1" 2) "val1" 3) "key2" 4) "val2"

Perhaps, command args can be joined with a ' '(space) into a single string and this string should be saved in RocksDB? After reading, it will be split while constructing the reponse. With this encoding scheme, I was thinking about the possible spaces inside arguments and how to deal with them?

Differences from Redis

XTRIM and XADD with trim possibility: nearly exact trimming (via ~) is not possible due to implementation details (no radix tree here). However, LIMIT option is working while in Redis it is allowed only in combination with ~. LIMIT can be disallowed to be consistent with Redis protocol. I didn't do that because I want to hear opinions from kvrocks maintainers.

Replication is not implemented yet. Basically, I didn't test streams in a cluster configuration. Perhaps, the plain XREAD on a replica will work, but blocking XREAD that unblocks after XADD on the master - I'm sure that some code should be written. It would be greatly appreciated if maintainers provide me with some hints about how to implement this.

Consumer groups are not implemented. I'm thinking about the possible implementation.

Right now I'm looking for any maintainers' feedback from the adding-new-data-type perspective (maybe, I didn't add a new column family to some filter/checker/extractor, etc.) and information about proper replicating a stream from master to other nodes.

This closes #532
new feature release notes major decision
opened by torwig 32

Flaky test on expire precision

Search before asking

[X] I had searched in the issues and found no similar issues.

Version

https://github.com/apache/incubator-kvrocks/pull/981/commits/461d0e8cc95cf959df50db2d75fecffbfd919875

Minimal reproduce step

Flaky test on CI env.

What did you expect to see?

Tests passed.

What did you see instead?

--- FAIL: TestString (12.24s)
    --- FAIL: TestString/GETEX_EXAT_option (0.00s)
        assertions.go:36: 
            	Error Trace:	/home/runner/work/incubator-kvrocks/incubator-kvrocks/tests/gocase/unit/type/strings/assertions.go:36
            	            				/home/runner/work/incubator-kvrocks/incubator-kvrocks/tests/gocase/unit/type/strings/strings_test.go:155
            	Error:      	"11s" is not less than or equal to "10s"
            	Test:       	TestString/GETEX_EXAT_option
    --- FAIL: TestString/GETEX_PXAT_option (0.00s)
        assertions.go:36: 
            	Error Trace:	/home/runner/work/incubator-kvrocks/incubator-kvrocks/tests/gocase/unit/type/strings/assertions.go:36
            	            				/home/runner/work/incubator-kvrocks/incubator-kvrocks/tests/gocase/unit/type/strings/strings_test.go:162
            	Error:      	"11s" is not less than or equal to "10s"
            	Test:       	TestString/GETEX_PXAT_option

Anything Else?

cc @git-hulk please take a look.

Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

bug

opened by tisonkun 31

[QUESTION] Change the license to Apache License Version 2.0

Hi, all Kvrocks contributors, we want to donate Kvrocks to Apache Software Foundation, the first step for us is changing the license to Apache License Version 2.0. As we know, Kvrocks is an open source software under the BSD-3-Clause license, we truly thank every contributor and user, you guys make Kvrocks great and have an active community, before we could submit PR to change license, we must to get your approval.

@git-hulk @karelrooted @ShooterIT @Alfejik @caipengbo @ChrisZMF @shangxiaoxiong @ColinChamber @guoxiangCN @calvinxiao @patpatbear @WyattJia @smartlee @popunit @clyang @byronhe @QQxiaoyuyu @nioshield @singgel @liangjingyang @gaoyuanzong @rongzeng54 @unext-ops @fishery86 @xujianhai666 @LykxSassinator

Please reply "Approved" if you agree.

opened by ShooterIT 28

同步基本不可用

masterA 有数据 slaveA slaveof过来，数据可以同步，flushall masterA的数据， slaveA 没有同步清空掉。都执行过dbsize scan 2)masterB 没数据 slaveB 同步过来up后，masterB上执行写数据,数据没有同步到slaveB 3)slaveof后 slave节点上经常会报以下错误, 但这个目录是清空后再启动slave的

E0216 13:35:53.319345 26935 storage.cc:703] [storage] Failed to delete dir: IO error: file rmdir: ./kvrocksdatamaster/backup/meta: Directory not empty
E0216 13:35:53.415446 26935 replication.cc:584] [replication] Failed to restore backup while IO error: No such file or directoryWhile opening a file for sequentially reading: ./kvrocksdatamaster/backup/meta/5: No such file or directory
Segmentation fault (core dumped)

opened by smartlee 23

Revert "Fix the misuse of multi-get API"

Reverts KvrocksLabs/kvrocks#442 l find that new multi-get api can get keys which have been deleted before. the case could be reproduced with my PR KvrocksLabs/kvrocks/pull/444 . It seems a bug of RocksDB and I would submit a relevant PR to RocksDB in the near future.

opened by shangxiaoxiong 21
✨ feat: luajit replace lua
Luajit replace lua

rm test case "cmsgpack can pack and unpack circular references". Becase luajit's core code diffrent from lua. Also, different compilers produce different code execution effects (GCC&clang on Ubuntu).
opened by xiaobiaozhao 18

Add `StatusOr` for error handling in modern C++ style

In kvrocks we use a style like Status process(input..., T* output) to handle errors. It works well, but in C++ we can do better via constructing type-safe union types.

In this PR, we propose a new class template StatusOr<T>, which contains EITHER a value typed T OR an error status. Here is a simple example:

StatusOr<std::string> hello(std::string x) {
  if (x.size() > 10) {
    return {Status::NotOK, "string too long"}; // returns an error
  }

  return x + " hello"; // returns a value (no error occurred)
};

StatusOr<std::string> hi(std::string x) {
  if (x.size() < 5) {
    return {Status::NotOK, "string too short"};
  }

  auto res = hello(x); // call `hello` which returns `StatusOr`
  if (!res) return res; // forward error status

  return "hi " + *res; // use it via deref
};

assert(*hi("twice") == "hi twice hello");

auto s = hi("x");
assert(!s && s.Msg() == "string too short"); // s has no value now

auto l = hi("xxxxxxxxxxx");
assert(!l && l.Msg() == "string too long");

We maximize the use of move semantics in C++ to eliminate redundant copies and optimize the storage, so that developers do not need to worry too much about its performance when using it.

opened by PragmaTwice 17

Add TLS support for kvrocks connections
We bring TLS support for kvrocks, referring to redis TLS.

TODO:

[x] conditional compilation

[x] more friendly error messages

[x] more options for SSL cert/key/cacert and ciphers

[x] thread safety for openssl and eventbuffer

[x] hot-reload tls options

[x] tests

temporarily not in plan (of this PR):

tls for replication

Why do we use shared library of OpenSSL and do not pin the version of OpenSSL?

OpenSSL contains many cryptographic algorithms and implementations of the TLS protocol, which are security-critical. We hope that users will be able to quickly update the OpenSSL that kvrocks depends on to a new version in the event of a serious security vulnerability in OpenSSL (In fact OpenSSL has had several serious security incidents). If static linking is used, users may need to modify the cmake file and recompile, and kvrocks developers also need to always pay attention to the security vulnerabilities of OpenSSL and maintain version of cmake dependencies, which is very troublesome.

Supported options:

tls_port

tls_cert_file

tls_key_file

tls_key_file_pass

tls_ca_cert_file

tls_ca_cert_dir

tls_auth_clients

tls_prefer_server_ciphers

tls_ciphers

tls_ciphersuites

tls_protocols

tls_session_caching

tls_session_cache_size

tls_session_cache_timeout
opened by PragmaTwice 16
Tracking issue for build system enhancements
Search before asking

[X] I had searched in the issues and found no similar issues.

Motivation

Modern CMake provides many amazing features to organize the build structure of a project in a more convenient way, and we can make full use of them to improve the development experience! @tisonkun had a good suggestion in https://github.com/apache/incubator-kvrocks/issues/562#issuecomment-1123771508 of creating a tracking issue to help keep the discussion more focused, so this issue is created for that purpose.

Solution

The list below tracks some of the improvements that have been proposed so far, and I'd appreciate it if anyone came up with new ideas for improvements, PRs for current improvements, or participated in this discussion!

[x] #561

[x] #562

[x] #574

[x] #569

[x] #563

[x] #580

[x] #589

[x] #590

Are you willing to submit a PR?

[X] I'm willing to submit a PR!

enhancement
opened by PragmaTwice 16
Memory leaks in Lua::redisGenericCommand
Search before asking

[X] I had searched in the issues and found no similar issues.

Version

redisGenericCommand use raiseError to process errors, but raiseError will perform a longjmp in the lua_error api call, so raiseError never returns.

longjmp does not compatible with RAII in C++ (C++ exceptions may use longjmp in implementation but the compiler will insert some dtor call autometically and carefully, i.e. stack unwinding), so every object in the stack of redisGenericCommand will never be destroyed, i.e. no destructor will be call, hence all resource allocated in ctor like std::vector, std::unique_ptr etc. cannot be released.

@git-hulk

Minimal reproduce step

./runtest --dont-clean on a kvrocks binary with Address Sanitizer enabled. (try to build with #599) You can only run tests in unit/scripting to save time : )

What did you expect to see?

No memory leaks

What did you see instead?

Memory leaks are reported by Address Sanitizer

Anything Else?

No response

Are you willing to submit a PR?

[X] I'm willing to submit a PR!

bug
opened by PragmaTwice 15
[BUG] Failed to open DB with missing CURRENT file
Describe the bug A clear and concise description of what the bug is.

OS: MAC OS 10.15.7

Version 2.0.5 @6a4ed16a

get following error message when try to start up kvrocks ( first time run )

E1229 15:41:02.217978 276704704 main.cc:314] Failed to open: IO error: No such file or directory: While opening a file for sequentially reading: /tmp/kvrocks/db/CURRENT: No such file or directory

/tmp/kvrocks directory did exist but no sub-directory ./db under it.
opened by jasonhe88 15
Wrong substring in kvrocks2redis config
Search before asking

[X] I had searched in the issues and found no similar issues.

Version

unstable

Minimal reproduce step

https://github.com/apache/incubator-kvrocks/blob/44ffb9a742792fd559b4147678fd1dbb1e5bc9d4/utils/kvrocks2redis/config.cc#L109

What did you expect to see?

--

What did you see instead?

--

Anything Else?

No response

Are you willing to submit a PR?

[X] I'm willing to submit a PR!

bug
opened by PragmaTwice 0
Persist the cluster nodes info after applying the cluster topology
This closes #1021

Currently, the cluster nodes' info is only stored in memory and we need to re-sync the cluster topology after restarting. It's very inconvenient and confusing for most users.

Solutoin

Persist the cluster nodes' info in the local disk if the topology was changed. The file location is {config->dir}/nodes.conf and the format is below:

version 2 id 07c37dfeb235213a872192d90877d0cd55635b92 node 07c37dfeb235213a872192d90877d0cd55635b91 127.0.0.1 59129 master - 1-2 4-8193 10000 10002-11002 16381-16383 node 07c37dfeb235213a872192d90877d0cd55635b92 127.0.0.1 59137 master - 0

Load and parse the nodes.conf if the cluster mode is enabled and the nodes file is exists
opened by git-hulk 2
Allow to enable the async_io option to improve the performance

Here are some comparisons between async_io and normal

big value 500B

small value 50B

monitor diff

The green line is kvrocks compiled by async io，this pr. ts* command using seek function to scan data from db.

summarize

| | cpu | mem | latency | qps | | | ------ | ---- | ---- | -------- | ---- | --- | | async | 110% | 100% | 100% | 120% | | | normal | 100% | 100% | 130-400% | 100% | |

async_io is better in seek read, higher qps, lower latency, and slightly higher cpu usage. Perfect for introducing projects

opened by xiaobiaozhao 2
BITOPS doesn't support BIT option
Search before asking

[X] I had searched in the issues and found no similar issues.

Version

latest

Minimal reproduce step

from redis command : https://redis.io/commands/bitpos/ redis> SET mykey "\x00\xff\xf0" "OK"

What did you expect to see?

redis> BITPOS mykey 1 7 15 BIT (integer) 9

What did you see instead?

redis> BITPOS mykey 1 7 15 BIT (integer) 57

Anything Else?

No response

Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

A-docs enhancement
opened by lt1946 3
Failed test case in CI: TestSlotMigrateThreeNodes
Search before asking

[X] I had searched in the issues and found no similar issues.

Version

unstable

Minimal reproduce step

https://github.com/apache/incubator-kvrocks/actions/runs/3749755928/jobs/6368682143#step:12:158

What did you expect to see?

pass

What did you see instead?

fail

Anything Else?

No response

Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

bug
opened by PragmaTwice 0

Releases(v2.2.0)

v2.2.0(Nov 10, 2022)
Notice

For this release, we are still using rocksdb 6.29, but we will update to rocksdb 7 in the next release (as in the unstable branch).

New Features

Allow using LuaJIT to improve the performance of the Lua script (#697)

Add the new command GETDEL (#771)

Add the new command SMISMEMBER (#778)

Add the new data structure Redis STREAM (#745)

Add TLS support for kvrocks connections (#797)

Add the new command HELLO (#881)

Add the new command EVAL_RO (#782)

Add the new command DISK (#882)

Add the new command GETEX (#961)

Support CLI options for the kvrocks server (#1037)

Add the new command options for ZADD (#1022)

Add ZLIB dependency for compression in rocksdb (#1078)

Improvements

Switch the default value of log-dir to stdout (#754)

Add fpm packaging to x.py (#752)

Propagate sanitizer flags to all cmake deps (#759)

Move runtest to x.py with customized server/cli path (#765)

Add StatusOr for error handling in modern C++ style (#768)

Add ParseInt in StatusOr style (#787)

Show function name in backtrace (#796)

Optimize get bitmap string and remove redundant num2bit array (#793)

Allows Kvrocks to listen to only the unix socket (#809)

Update dependency to latest version (lz4, libevent, gtest) (#828)

Export the rocksdb write options (#885)

Support quoted string and inline comment in config (#849)

Add the FilterBlobByKey for SubKeyFilter (#902)

Avoid using Get when iterating sub-keys during migration (#906)

Support EXAT/PEXAT option in the set command (#901)

Return from HRANGE if the key wasn't found (#923)

Update jemalloc to 5.3.0 (#940)

Use ParseInt to replace sto*, ato* (#924)

Add UniqueFD and ScopeExit (#973)

Align georadius behavior of redis (#993)

Align the expire option behavior of redis (#1017)

Support config set backup-dir new-dir (#1026)

Use a consistent way to get the current time (#1038)

Add support for showing used_memory_startup (#1044)

Improve stack trace format and clean main.cc (#1053)

Add command parser (#1032)

Add WriteBatchInspector to inspect WriteBatch (#1069)

Improve libunwind linking in CMake (#1079)

Bug Fixes

Fix didn't return the out of range error in cluster import command(#767)

Fix should forbid to create of the default namespace (#772)

Fix CMake configuration error in snappy (#803)

Fix inline protocol don't allow LF only EOL (#808)

Fix successor commands won't be processed before receiving the next read event (#839)

Fix georadius should return an empty set if the key does not exist (#845)

Fix config set rocksdb.blob_garbage_collection_age_cutoff (#914)

Fix ping with arg should return a bulk string (#933)

Fix server cannot exit properly when enabling cluster mode (#969)

Fix shouldn't lowercase the configuration value in kvrocks2redis(#971)

Fix don't duplicate killed clients which have already flagged kCloseAfterReply (#1020)

Fix properly parse cmake version (#1043)

Fix rename-command only the first one works(#1047)

Fix CAS and CAD return statuses (#1055)

Fix random crash in the migration test case (#1068)

Test Refactoring

TCL tests are now moved to Go, refer to #811.

Misc

Replace cpplint with clang-format (#979)

Refactor source file structure (#989)

Update C++ standard to 17 (#1006)

Use clang-tidy instead of cppcheck (#1025)

Source code(tar.gz)
Source code(zip)
v2.1.0(Aug 3, 2022)
New features

Support source packaging by x.py (#729)

Support to use lz4 compression in RocksDB (#584)

Allow to send logs to stdout/stderr (#613)

LPOP/RPOP support pop multi elements (#594)

Add LMove command (#577)

Support to listen on the IPV6 address (#554)

Add support for listening to unix socket (#531)

Support BITOP command (#471)

Support unlink command (#517)

Improvements

replace shell scripts with x.py (#725)

Use lock_guard to replace the manual mutex lock/unlock (#723)

Build docker image with travis ci (#724)

Return values instead of passing pointers (#722)

Don't call GetCurrentTime in expired and type when possible (#709)

Remove the slice ToString() when encoding or decoding the key (#707)

Use the const reference for avoiding copy (#704)

Check sanitizer status for TCL tests in CI (#698)

CMake: support ninja as a build backend (#626)

Build static lib for glog (#618)

Improve using of ASan and TSan in CMake build (#599)

PSYNC based on Unique Replication Sequence ID (#538)

Remove handwritten makefiles to keep only one build system (#576)

Use FetchContent instead of ExternalProject and git submodules in CMake (#564)

Upgrade the rocksdb version to 6.29.5(latest version for 6.x) (#555)

Support to send auth command before migrating slots (#514)

Allow to dynamic resize the level_compaction_dynamic_level_bytes and level base size (#497)

Bugfixes

Make the set command consistent with redis (#730)

Fix some time dependencies in replication.tcl (#688)

Fix get rocksdb ops tcl test case (#668)

Fix typos in config file (#669)

Fix replication tests should wait for server restarted (#660)

Fix the role command can't be used when loading data (#661)

Fix Wrongly parsed the RESP empty/null array (#652)

Fix compile warnings in kvrocks (#653)

Fix unmatched iterator in Cluster::SetClusterNodes (#646)

Fix load namespaces from the config file would ignore case (#642)

Fix RocksDB can't auto resume after disk quota exceeded error (#628)

Fix don't swallow the error when creating column families failed (#620)

Compile lua by C++ compilers to avoid memory leaks (#614)

Fix the slave may crash when restoring the db (#606)

Fix leaks reported by ASan (#605)

Fix can't find jemalloc when building with makefile on MacOSX (#557)

Stop sending files if failed in full synchronization (#539)

Avoid requirepass and masterauth conflicts with the namespace tokens (#507)

Fix slave can't resync with master after password change (#520)

Fix rocksdb cmake depends should use list instead of string (#518)

Fix spop command should return bulk string instead of array when without count(#515)

Fix wrong replication state (#506)

Avoid flush all redis db in kvrocks2redis (#498)

Fix failing in setting thread name with string more than 16 byte (#496)

Avoid accessing slot_migrate_ before it is created (#472)

Fix the connection fd was used after free (#479)

Others

Add use case CIRCL.lu (#546)

Change license to Apache 2.0 (#533)

Add DISCLAIMER (#558)

Apply Apache License header for Tcl files (#548)

Add .asf.yaml file (#545)

Apply Apache License header for source code (#543)

Source code(tar.gz)
Source code(zip)
v2.0.6(Jan 28, 2022)
Thanks ChrisZMF who implements slot-based data migration for Kvrocks cluster mode, this authentically makes kvrocks cluster scalable. Even though it already is used in production environments, you still should be cautious of it.

Here is a comprehensive list of changes in this release compared to 2.0.5

New features

Support slot-based data migration (#430) Biggest feature of this version, authentically makes kvrocks scalable, user can use CLUSTERX MIGRATE command to migrate slot and use CLUSTERX SETSLOT to change cluster slot distribution after finishing migration.

Make level0_file_num_compaction_trigger of RocksDB configurable (#455)

Support CLUSTERX SETSLOT command (#463)

Improvements

Support executing INFO command during restoring database (#375)

Auto Rewrite the config after the namespace was updated (#423)

Support to convert sortedint type to zset type in kvrocks2redis (#431)

Optimize the seek of RocksDB iterators (#438)

Make the bitmap range the same with that of Redis (#465)

Support GET command on bitmap type (#447)

Support to use row cache of RocksDB (#407)

Bug fixes

Fix the parameter blob_garbage_collection_age_cutoff would be always zero (#425)

Fix config rewrite overwrite the first rename-command with the last one (#429)

Fix zrevrangebyscore for multi members with the same max score (#434)

Fix data race in SetCurrentConnection (#440)

Fix failing full replication when enabling slave-empty-db-before-fullsync (#446)

Fix memory leak when full replication (#461) Maybe leak dozen kilobytes per full replication

Fix the bitmap range is smaller than that of Redis (#465)

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.6-release.x86_64.rpm(129.69 MB)
kvrocks_2.0.6-release_amd64.deb(118.73 MB)
v2.0.5(Dec 7, 2021)
Here is a comprehensive list of changes in this release compared to 2.0.4

New features

Implement zrevrangebylex command (#381)

Use some rocksdb new feature (#395)

Support blob db (key-value separation) which requires RocksDB version greater than 6.18

Enable partitioned index/filter for SST and hash index for data block

Enable whole key filter for memtable

Remove dynamic resize block and SST

Only apply compression to SST which level is more than 2

Please note that, if your rocksdb version is lower than 6.6.0, it can't reload the data of new version, you should update cautiously.

Implement CAS and CAD commands for string type (#415)

Support to use row cache of RocksDB (#407)

Improvements

Add rocksdb ops statistics into the info command (#387) These statistics can more truly reflect the load of rocksdb.

Add the maxclients to the info command output (#391)

Add write stall/stop stats to info command output (#396)

Make kvrocks2redis work (#404)

Adjust the max open file limit according to the max clients and rocksdb max open files (#418)

Related projects

Kvrocks exporter, a fork of oliver006/redis_exporter to export the kvrocks metrics, https://github.com/KvrocksLabs/kvrocks_exporter

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.5-release.x86_64.rpm(127.80 MB)
kvrocks_2.0.5-release_amd64.deb(116.75 MB)
v2.0.4(Nov 2, 2021)
Here is a comprehensive list of changes in this release compared to 2.0.3

New features

Support LUA script commands (#363) (#369) Currently, support EVAL, EVALSHA, and SCRIPT commands, but SCRIPT KILL and DEBUG subcommands are not supported.

Improvements

Allow metadata and subkey CF to share a single block cache (#365) (#373)

Add master_repl_offset in INFO command output (#374)

Bugfixes

Fixed dynamic resize target_file_size_base bug (#368) Before we only apply this strategy to the default CF(i.e. subkey CF)

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.4-release.x86_64.rpm(127.47 MB)
kvrocks_2.0.4-release_amd64.deb(116.46 MB)
v2.0.3(Sep 13, 2021)
If your kvrocks instances are 2.0.2 and use redis-sentinel to implement HA, you should upgrade urgently, please see #352 for details.

Here is a comprehensive list of changes in this release compared to 2.0.2

Improvements

Use the rocksdb MultiGet interface in the MGET command (#331)

Avoid useless space allocation in SetBit (#338)

Auto set master/replica replication by cluster topology (#356)

Support client kill type subcommands to adapt redis-sentinel (#352) (#361) From 2.0.2, we support MULTI-EXEC, but don't support client kill type $type, so MULTI-EXEC will be rollbacked when failing to parse unknown kill type in client command, and cause redis-sentinel fails to failover.

Bugfixes

Fix SETNX and MSETNX commands can't guarantee atomicity (#337)

Fix the replicas won't reconnect when failing to connect with master (#344)

Fix master pauses synchronizing with replicas (#347)

Fix wrong mater port if enabling master-use-repl-port (#360)

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.3-release.x86_64.rpm(126.69 MB)
kvrocks_2.0.3-release_amd64.deb(115.68 MB)
v2.0.2(Jul 19, 2021)
We design new cluster solution for kvrocks, but it is experimental, don't recommend to deploy in production environments. And because in cluster mode, to be efficient to migrate keys based on slot when scale, we encode key with slot id, but in order to be compatible with the old version, we don't do that in standalone mode, so data is not compatible between standalone mode with cluster mode, you must migrate data if you want to change mode, otherwise, kvrocks will make data corrupt. In order to better support the cluster mode, we are developing a meta server to manage kvrocks cluster, coming soon!

Here is a comprehensive list of changes in this release compared to 2.0.1

New features

Support rename command in config file (#272)

Support transaction (#285) Support MULTI, EXEC, DISCARD command, but don't support WATCH and UNWATCH commands.

Support the auto-resize-block-and-sst config directive (#289)

Support to resize rocksdb.block_size automatically (#294)

Support cluster mode (#302) Similar with redis cluster mode, but use CLUSTERX command to set cluster topology (#324).

Data encoding changes

Encode key with slot id (#291) (#330) In cluster mode, to be efficient to migrate keys by slot when scale, we encode key with slot id, but in standalone mode, we don't. That is to say, data is not compatible between standalone mode with cluster mode, you must migrate data if you want to change mode, otherwise, kvrocks will make data corrupt.

Bug fixes

Set TCP_KEEPALIVE for replication connection (#284) Replicas can handle network failure with master, before this commit, network failure may result in unrecoverable replication interrupt.

Fix ZSET can't add the same member with different scores (#298)

Make CONFIG SET command thread safe (#310)

Fix LTRIM only delete first item, LREM can't correctly remove repeated items (#314)

Improvements

HSET command supports to set multiple fields (#274)

Allow to use host in the master-slave replication (#301)

Adapt redis sentinel ping check (#316)

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.2-release.x86_64.rpm(126.28 MB)
kvrocks_2.0.2-release_amd64.deb(115.27 MB)
v2.0.1(May 18, 2021)
Here is a comprehensive list of changes in this release compared to 2.0.0

New features

Support COMMAND command (#251)

Support to purge backups on fullsync to reduce disk space usage (#245)

Bug fixes

Fix uninitialized Rocksdb.level0_stop_writes_trigger (#236)

Fix condition race for ReclaimOldDBPtr (#246)

Fix data race for accessing database in some commands (#253) (#261) In spop, zremrangebylex, zremrangebyscore commands, the lock guard for accessing the database may not work actually.

Fix returning wrong string range in GETRANGE command (#254)

Fix skipping wrong CURRENT file when full synchronization (#260) Resuming broken transfer based files may be not available if replicas skip transferred wrong CURRENT file of RocksDB.

Improvements

Replicas can execute publish command (#238)

Optimizations for avoiding unnecessary data copy (#239) (#247)

Allow to run the kvrocks without assigning the config file (#264)

Dependencies

Upgrade the jemalloc version to the special commit (#265) To be able to compile on macOS, upgrade jemalloc to the production test commit which mentioned in jemalloc/jemalloc#2060.

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.1-release.x86_64.rpm(125.30 MB)
kvrocks_2.0.1-release_amd64.deb(114.31 MB)
v1.3.3(May 18, 2021)
Here is a comprehensive list of changes in this release compared to 1.3.2

Bug fixes

Fix rocksdb can't auto resume after no space error (#229)

Fix uninitialized Rocksdb.level0_stop_writes_trigger (#236)

Fix condition race for ReclaimOldDBPtr (#246)

Fix returning wrong string range in GETRANGE command (#254)

Fix data race for accessing database in some commands (#253) In spop, zremrangebylex, zremrangebyscore commands, the lock guard for accessing the database may not work actually.

Improvements

Replicas can execute publish command (#238)

Source code(tar.gz)
Source code(zip)
v2.0.0(Apr 23, 2021)
Kvrocks 2.0 brings some new useful features that solve actual problems in the production environment. Please note that this version instance still can replicate from old version master but old version replicas can't replicate new version master. Otherwise, we start to use new logo, thanks @tianlingyu1997

Here is a comprehensive list of changes in this release compared to 1.3.2

New features

Don't use extra port for replication (#200)

Use checkpoint to implement data snapshot when full replication (#208) Reduce disk space and bandwidth usage and decrease the time of replicas recovery during full synchronization.

Use checkpoint to implement kvrocks backup (#232)

Replicas can empty db before fullsync (#233) Replicas use less disks space if enabled, but have a risk to lose data.

Command changes

Add QPS and input/output kbps for INFO command (#213)

Add pubsub_patterns for INFO command (#234)

Slowlog omits args or argument string if too many or big (#215)

Cluster

Remove codis support (#217) From now, we doesn't support codis, you can adopt pre-sharding to implement cluster, and we are developing new cluster solution that is similar with redis cluster mode, please see #219.

Bugfixes

Fix kvrocks can't auto resume after no space error (#229)

Dependencies

Upgrade rocksdb to latest tag v6.19.3 (#226)

Source code(tar.gz)
Source code(zip)
v1.3.2(Apr 20, 2021)
Here is a comprehensive list of changes in this release compared to 1.3.1

Bug fixes

Fix incorrect used_db_size in INFO command (#204)

Fix the SST file creation time maybe 0 in some unknown conditions (#211)

Fix get corruption writebatch data after psync restart (#221)

Improvements

Optimize TCP keepalive detection time (#201)

Source code(tar.gz)
Source code(zip)
v1.3.0(Feb 8, 2021)
Here is a comprehensive list of changes in this release compared to 1.2.0

License

Change the license from MIT to BSD-3-Clause

New features

Support to set separate log directory to store server logs (#155)

Implement PSETEX command (#156)

Implement MSETNX command (#165)

Improvements

Improve the precision of the INCRBYFLOAT/HINCRBYFLOAT command to double (#169)

Support +INF/-INF for the score of zset type members (#172)

The UNSUBSCRIBE and PUNSUBSCRIBE commands support multiple channels or patterns (#174)

Behavior changes

The UNSUBSCRIBE and PUNSUBSCRIBE commands have the same reply as Redis (#174)

The same message format as Redis for pattern subscribers when publishing message (#175) Message format is from 'message channel message' to 'pmessage pattern channel message'

Bug fixes

Fix skipping some SST files when picking compaction files (#154)

Handle negative expire time in set command to avoid deleting keys (#156)

List type command doesn't rewrite other types of data (#157)

Only set positive expire time when migrating key (#160)

Don't store an empty key for setrange command (#161)

Fix SETBIT/GETBIT bit offset out of range (#164)

Fix INCRBY/INCRBYFLOAT command operates on wrong value and precision check (#167)

Fix HINCRBY/HINCRBYFLOAT command operates on wrong value and precision check (#168)

RPOPLPUSH command doesn't operate non list type (#170)

LINSERT command returns -1 if don't find index (#170)

Fix LTRIM command range calculation when start is a very large negative number (#170)

SMOVE command doesn't operate non set type key (#171)

Fix ZREMRANGEBYRANK command corrupts the size of zset (#172)

SUBSCRIBE and PSUBSCRIBE commands return the sum of subscription and psubscription (#174)

Fix PUNSUBSCRIBE command doesn't take effort (#174)

Allow setting slowlog-log-slower-than to -1 to disable the slowlog (#178)

Source code(tar.gz)
Source code(zip)
v1.2.0(Jan 8, 2021)

Source code(tar.gz)
Source code(zip)
kvrocks-1.2.0-167827.release.x86_64.rpm(19.98 MB)
v1.1.32(Oct 28, 2020)

Source code(tar.gz)
Source code(zip)
kvrocks-1.1.32-142122.release.x86_64.rpm(19.96 MB)
v1.1.27(Aug 26, 2020)
FIX: bitcount return wrong number (cause by typo)

Source code(tar.gz)
Source code(zip)
kvrocks-1.1.27-126340.release.x86_64.rpm(19.90 MB)
v1.1.6(Feb 28, 2020)
Upgrade the libevent to stable 2.1.11

FIX: create the backup at the same time may cause a crash while the CreateNewBackupWithMetadata wasn't thread-safe call

FIX: some incorrect implements in redis command

Source code(tar.gz)
Source code(zip)
v.1.1.2(Dec 6, 2019)

Source code(tar.gz)
Source code(zip)
v1.1.0(Nov 14, 2019)

Source code(tar.gz)
Source code(zip)
v1.0.3(Sep 18, 2019)

Source code(tar.gz)
Source code(zip)
v1.0.1(Aug 23, 2019)
ADD: supports flushall/object command

FIX: incorrect command name, flushall => flushdb

FIX: hscan return not found when key not exists

Source code(tar.gz)
Source code(zip)

Owner

Kvrocks Labs

Home of Kvrocks

GitHub

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB

24.3k Jan 5, 2023

KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language

KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language. It is designed for persistent memory and provides unified APIs for both volatile and persistent scenarios. It also demonstrates several optimization methods for high performance with persistent memory. Besides providing the basic APIs of key-value store, it offers several advanced features, like transaction, snapshot as well.

186 Nov 16, 2022

RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis.

A query and indexing engine for Redis, providing secondary indexing, full-text search, and aggregations.

4k Jan 5, 2023

Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large-scale graphs with billions of vertices (nodes) and trillions of edges, with milliseconds of latency. It delivers enterprise-grade high performance to simplify the most complex data sets imaginable into meaningful and useful information.

8.4k Jan 9, 2023

🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

?? ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

12.8k Jan 9, 2023

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB. Scylla embraces a shared-nothing approach that increases throughput and storage capacity to realize order-of-magnitude performance improvements and reduce hardware costs.

8.9k Jan 4, 2023

FEDB is a NewSQL database optimised for realtime inference and decisioning application

FEDB is a NewSQL database optimised for realtime inference and decisioning applications. These applications put real-time features extracted from multiple time windows through a pre-trained model to evaluate new data to support decision making. Existing in-memory databases cost hundreds or even thousands of milliseconds so they cannot meet the requirements of inference and decisioning applications.

1.2k Jan 2, 2023

The MongoDB Database

23k Jan 1, 2023

Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.

1.9k Jan 8, 2023

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB

24.3k Jan 5, 2023

KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language

186 Nov 16, 2022

FoundationDB - the open source, distributed, transactional key-value store

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as

12k Dec 31, 2022

RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis.

A query and indexing engine for Redis, providing secondary indexing, full-text search, and aggregations.

4k Jan 5, 2023

redis-cpp is a header-only library in C++17 for Redis (and C++11 backport)

redis-cpp - lightweight C++ client library for Redis redis-cpp is a C++17 library for executing Redis commands with support for pipelines and the publ

77 Dec 11, 2022

A redis module, similar to redis zset, but you can set multiple scores for each member to support multi-dimensional sorting

TairZset: Support multi-score sorting zset Introduction Chinese TairZset is a data structure developed based on the redis module. Compared with the na

60 Dec 1, 2022

libmdbx is an extremely fast, compact, powerful, embedded, transactional key-value database, with permissive license

One of the fastest embeddable key-value ACID database without WAL. libmdbx surpasses the legendary LMDB in terms of reliability, features and performance.

1.1k Dec 19, 2022

An Embedded NoSQL, Transactional Database Engine

UnQLite - Transactional Embedded Database Engine

1.8k Dec 24, 2022

Kunlun distributed DBMS is a NewSQL OLTP relational distributed database management system

Kunlun distributed DBMS is a NewSQL OLTP relational distributed database management system. Application developers can use Kunlun to build IT systems that handles terabytes of data, without any effort on their part to implement data sharding, distributed transaction processing, distributed query processing, crash safety, high availability, strong consistency, horizontal scalability. All these powerful features are provided by Kunlun.

114 Dec 26, 2022

Utility for testing random and pseudorandom sequences, either as bytes or bit streams, reporting entropy, mean value, serial correlation, chi square, and Monte Carlo estimate of an value, serial correlation, chi square, and Monte Carlo estimate of π.

ENT — Fourmilab Random Sequence Tester The Fourmilab Random Sequence Tester, ent, applies various tests to sequences of bytes stored in files and repo

24 Dec 15, 2022

Kreon is a key-value store library optimized for flash-based storage

Kreon is a key-value store library optimized for flash-based storage, where CPU overhead and I/O amplification are more significant bottlenecks compared to I/O randomness.

Computer Architecture and VLSI Systems (CARV) Laboratory

24 Jul 14, 2022

Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Related tags

Overview

Who uses kvrocks

Building kvrocks

requirements

Build

Running kvrocks

Running test cases

Supported platforms

Try kvrocks using Docker

Namespace

DOCs

Migrate tools

Performance

Hardware

1. Commands QPS

2. QPS on different payloads

3. QPS on different workers

License

Comments

Search before asking

Version

Minimal reproduce step

What did you expect to see?

What did you see instead?

Anything Else?

Are you willing to submit a PR?

Search before asking

Motivation

Solution

Are you willing to submit a PR?

Search before asking

Version

Minimal reproduce step

What did you expect to see?

What did you see instead?

Anything Else?

Are you willing to submit a PR?

Search before asking

Version

Minimal reproduce step

What did you expect to see?

What did you see instead?

Anything Else?

Are you willing to submit a PR?

Solutoin

big value 500B

small value 50B

monitor diff

summarize

Search before asking

Version

Minimal reproduce step

What did you expect to see?

What did you see instead?

Anything Else?

Are you willing to submit a PR?

Search before asking

Version

Minimal reproduce step

What did you expect to see?

What did you see instead?

Anything Else?

Are you willing to submit a PR?

Releases(v2.2.0)

v2.2.0(Nov 10, 2022)

Notice

New Features

Improvements

Bug Fixes

Test Refactoring

Misc

v2.1.0(Aug 3, 2022)

New features

Improvements

Bugfixes

Others

v2.0.6(Jan 28, 2022)

New features