Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Overview

kvrocks_logo

image image GitHub license GitHub stars

Kvrocks is an open-source key-value database which is based on rocksdb and compatible with Redis protocol. Intention to decrease the cost of memory and increase the capability while compared to Redis. The design of replication and storage was inspired by rocksplicator and blackwidow.

Kvrocks has the following key features:

  • Redis protocol, user can use redis client to visit the kvrocks
  • Namespace, similar to redis db but use token per namespace
  • Replication, async replication using binlog like MySQL
  • High Available, supports redis sentinel to failover when master or slave was failed

Thanks for designers @田凌宇 and @范世丽 contribute the kvrocks logo for us.

Who uses kvrocks

Meitu Ctrip
Baidu BaishanCloud

Tickets a pull reqeust to let us known that you're using kvrocks and add your logo to README

Building kvrocks

requirements

  • g++ (required by c++11, version >= 4.8)
  • autoconf automake libtool snappy

Build

NOTE: You should install the snappy first:

# Centos/Redhat
sudo yum install -y epel-release && sudo yum install -y git gcc gcc-c++ make snappy snappy-devel autoconf automake libtool which gtest gtest-devel

# Ubuntu
sudo apt-get install gcc g++ make libsnappy-dev autoconf automake libtool which libgtest-dev

# MACOSX
brew install snappy googletest

It is as simple as:

$ git clone --recursive https://github.com/kvrockslabs/kvrocks.git
$ cd kvrocks
$ make -j4

Running kvrocks

$ ./src/kvrocks -c kvrocks.conf

Running test cases

NOTE: You should install the googletest first

make test

Supported platforms

  • centos 6/7
  • ubuntu
  • macosx

Try kvrocks using Docker

$ docker run -it -p 6666:6666 kvrocks/kvrocks
$ redis-cli -p 6666

127.0.0.1:6666> get a
(nil)

Namespace

namespace was used to isolate data between users. unlike all the redis databases can be visited by requirepass, we use one token per namespace. requirepass was regraded as admin token, only admin token allows to access the namespace command, as well as some commands like config, slaveof, bgsave, etc…

namespace del ns1 OK ">
# add token
127.0.0.1:6666>  namespace add ns1 mytoken
OK

# update token
127.0.0.1:6666> namespace set ns1 new_token
OK

# list namespace
127.0.0.1:6666> namespace get *
1) "ns1"
2) "new_token"
3) "__namespace"
4) "foobared"

# delete namespace
127.0.0.1:6666> namespace del ns1
OK

DOCs

Migrate tools

  • migrate from redis to kvrocks, use redis-migrate-tool which was developed by vipshop
  • migrate from kvrocks to redis. use kvrocks2redis in build dir

Performance

Hardware

  • CPU: 48 cores Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
  • Memory: 32 GiB
  • NET: Intel Corporation I350 Gigabit Network Connection
  • DISK: 2TB NVMe Intel SSD DC P460

Benchmark Client: multi-thread redis-benchmark(unstable branch)

1. Commands QPS

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns / 128 payload

latency: 99.9% < 10ms

image

2. QPS on different payloads

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns

latency: 99.9% < 10ms

image

3. QPS on different workers

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns / 128 payload

latency: 99.9% < 10ms

image

License

Kvrocks is under the BSD-3-Clause license. See the LICENSE file for details.

Comments
  • Implement the data structure and partial commands of the Redis stream

    Implement the data structure and partial commands of the Redis stream

    Available commands:

    • XADD
    • XDEL
    • XINFO STREAM
    • XLEN
    • XRANGE
    • XREAD
    • XREVRANGE
    • XTRIM

    Design notes Link to the Redis data type description: https://redis.io/docs/manual/data-types/streams/ A stream is a sequence of entries. Each entry has a unique ID in the form "1-2" where two 64-bit numbers are divided by a hyphen. By default, the first number is set to a millisecond timestamp, and the second one is a so-called sequence number - for cases when more than one entry was added at the same millisecond. The value of an entry is a set of key-value pairs. In case of the command: XADD s1 1-0 key1 val1 key2 val2 the ID is 1-0 and the value is key1 val1 key2 val2. In RocksDB entries are represented as key-value pairs where the key is formed as: key | version | entry-ID-milliseconds-value | entry-ID-sequence-number-value and the value is encoded as: key1-length(fixed-32) | key1 | val1-length(fixed-32) | val1 | key2-length(fixed-32) | key2 | val2-length(fixed-32) | val2. Thanks to the structure of a key, all entries in a stream are sorted in chronological order. As for value decoding: this is the first idea that came to my mind and maybe it's not very efficient because has an overhead (4 bytes on every argument). Why did I introduce such a weird encoding scheme? Because if you are reading entries, Redis responds not with a single string:

    1) 1) "s1"
       2) 1) 1) "1-0"
             2) 1) "key1"
                2) "val1"
                3) "key2"
                4) "val2"
    

    Perhaps, command args can be joined with a ' '(space) into a single string and this string should be saved in RocksDB? After reading, it will be split while constructing the reponse. With this encoding scheme, I was thinking about the possible spaces inside arguments and how to deal with them?

    Differences from Redis

    1. XTRIM and XADD with trim possibility: nearly exact trimming (via ~) is not possible due to implementation details (no radix tree here). However, LIMIT option is working while in Redis it is allowed only in combination with ~. LIMIT can be disallowed to be consistent with Redis protocol. I didn't do that because I want to hear opinions from kvrocks maintainers.

    Replication is not implemented yet. Basically, I didn't test streams in a cluster configuration. Perhaps, the plain XREAD on a replica will work, but blocking XREAD that unblocks after XADD on the master - I'm sure that some code should be written. It would be greatly appreciated if maintainers provide me with some hints about how to implement this.

    Consumer groups are not implemented. I'm thinking about the possible implementation.

    Right now I'm looking for any maintainers' feedback from the adding-new-data-type perspective (maybe, I didn't add a new column family to some filter/checker/extractor, etc.) and information about proper replicating a stream from master to other nodes.

    This closes #532

    new feature release notes major decision 
    opened by torwig 32
  • Flaky test on expire precision

    Flaky test on expire precision

    Search before asking

    • [X] I had searched in the issues and found no similar issues.

    Version

    https://github.com/apache/incubator-kvrocks/pull/981/commits/461d0e8cc95cf959df50db2d75fecffbfd919875

    Minimal reproduce step

    Flaky test on CI env.

    What did you expect to see?

    Tests passed.

    What did you see instead?

    --- FAIL: TestString (12.24s)
        --- FAIL: TestString/GETEX_EXAT_option (0.00s)
            assertions.go:36: 
                	Error Trace:	/home/runner/work/incubator-kvrocks/incubator-kvrocks/tests/gocase/unit/type/strings/assertions.go:36
                	            				/home/runner/work/incubator-kvrocks/incubator-kvrocks/tests/gocase/unit/type/strings/strings_test.go:155
                	Error:      	"11s" is not less than or equal to "10s"
                	Test:       	TestString/GETEX_EXAT_option
        --- FAIL: TestString/GETEX_PXAT_option (0.00s)
            assertions.go:36: 
                	Error Trace:	/home/runner/work/incubator-kvrocks/incubator-kvrocks/tests/gocase/unit/type/strings/assertions.go:36
                	            				/home/runner/work/incubator-kvrocks/incubator-kvrocks/tests/gocase/unit/type/strings/strings_test.go:162
                	Error:      	"11s" is not less than or equal to "10s"
                	Test:       	TestString/GETEX_PXAT_option
    

    Anything Else?

    cc @git-hulk please take a look.

    Are you willing to submit a PR?

    • [ ] I'm willing to submit a PR!
    bug 
    opened by tisonkun 31
  • [QUESTION] Change the license to Apache License Version 2.0

    [QUESTION] Change the license to Apache License Version 2.0

    Hi, all Kvrocks contributors, we want to donate Kvrocks to Apache Software Foundation, the first step for us is changing the license to Apache License Version 2.0. As we know, Kvrocks is an open source software under the BSD-3-Clause license, we truly thank every contributor and user, you guys make Kvrocks great and have an active community, before we could submit PR to change license, we must to get your approval.

    @git-hulk @karelrooted @ShooterIT @Alfejik @caipengbo @ChrisZMF @shangxiaoxiong @ColinChamber @guoxiangCN @calvinxiao @patpatbear @WyattJia @smartlee @popunit @clyang @byronhe @QQxiaoyuyu @nioshield @singgel @liangjingyang @gaoyuanzong @rongzeng54 @unext-ops @fishery86 @xujianhai666 @LykxSassinator

    Please reply "Approved" if you agree.

    opened by ShooterIT 28
  • 同步基本不可用

    同步基本不可用

    1. masterA 有数据 slaveA slaveof过来,数据可以同步,flushall masterA的数据, slaveA 没有同步清空掉。都执行过dbsize scan 2)masterB 没数据 slaveB 同步过来up后 ,masterB上执行写数据,数据没有同步到slaveB 3)slaveof后 slave节点上经常会报以下错误, 但这个目录是清空后再启动slave的
    E0216 13:35:53.319345 26935 storage.cc:703] [storage] Failed to delete dir: IO error: file rmdir: ./kvrocksdatamaster/backup/meta: Directory not empty
    E0216 13:35:53.415446 26935 replication.cc:584] [replication] Failed to restore backup while IO error: No such file or directoryWhile opening a file for sequentially reading: ./kvrocksdatamaster/backup/meta/5: No such file or directory
    Segmentation fault (core dumped)
    
    opened by smartlee 23
  • Revert

    Revert "Fix the misuse of multi-get API"

    Reverts KvrocksLabs/kvrocks#442 l find that new multi-get api can get keys which have been deleted before. the case could be reproduced with my PR KvrocksLabs/kvrocks/pull/444 . It seems a bug of RocksDB and I would submit a relevant PR to RocksDB in the near future.

    opened by shangxiaoxiong 21
  • ✨ feat: luajit replace lua

    ✨ feat: luajit replace lua

    1. Luajit replace lua
    2. rm test case "cmsgpack can pack and unpack circular references". Becase luajit's core code diffrent from lua. Also, different compilers produce different code execution effects (GCC&clang on Ubuntu).
    opened by xiaobiaozhao 18
  • Add `StatusOr` for error handling in modern C++ style

    Add `StatusOr` for error handling in modern C++ style

    In kvrocks we use a style like Status process(input..., T* output) to handle errors. It works well, but in C++ we can do better via constructing type-safe union types.

    In this PR, we propose a new class template StatusOr<T>, which contains EITHER a value typed T OR an error status. Here is a simple example:

    StatusOr<std::string> hello(std::string x) {
      if (x.size() > 10) {
        return {Status::NotOK, "string too long"}; // returns an error
      }
    
      return x + " hello"; // returns a value (no error occurred)
    };
    
    StatusOr<std::string> hi(std::string x) {
      if (x.size() < 5) {
        return {Status::NotOK, "string too short"};
      }
    
      auto res = hello(x); // call `hello` which returns `StatusOr`
      if (!res) return res; // forward error status
    
      return "hi " + *res; // use it via deref
    };
    
    assert(*hi("twice") == "hi twice hello");
    
    auto s = hi("x");
    assert(!s && s.Msg() == "string too short"); // s has no value now
    
    auto l = hi("xxxxxxxxxxx");
    assert(!l && l.Msg() == "string too long");
    

    We maximize the use of move semantics in C++ to eliminate redundant copies and optimize the storage, so that developers do not need to worry too much about its performance when using it.

    opened by PragmaTwice 17
  • Add TLS support for kvrocks connections

    Add TLS support for kvrocks connections

    We bring TLS support for kvrocks, referring to redis TLS.

    TODO:

    • [x] conditional compilation
    • [x] more friendly error messages
    • [x] more options for SSL cert/key/cacert and ciphers
    • [x] thread safety for openssl and eventbuffer
    • [x] hot-reload tls options
    • [x] tests

    temporarily not in plan (of this PR):

    • tls for replication

    Why do we use shared library of OpenSSL and do not pin the version of OpenSSL?

    OpenSSL contains many cryptographic algorithms and implementations of the TLS protocol, which are security-critical. We hope that users will be able to quickly update the OpenSSL that kvrocks depends on to a new version in the event of a serious security vulnerability in OpenSSL (In fact OpenSSL has had several serious security incidents). If static linking is used, users may need to modify the cmake file and recompile, and kvrocks developers also need to always pay attention to the security vulnerabilities of OpenSSL and maintain version of cmake dependencies, which is very troublesome.


    Supported options:

    • tls_port
    • tls_cert_file
    • tls_key_file
    • tls_key_file_pass
    • tls_ca_cert_file
    • tls_ca_cert_dir
    • tls_auth_clients
    • tls_prefer_server_ciphers
    • tls_ciphers
    • tls_ciphersuites
    • tls_protocols
    • tls_session_caching
    • tls_session_cache_size
    • tls_session_cache_timeout
    opened by PragmaTwice 16
  • Tracking issue for build system enhancements

    Tracking issue for build system enhancements

    Search before asking

    • [X] I had searched in the issues and found no similar issues.

    Motivation

    Modern CMake provides many amazing features to organize the build structure of a project in a more convenient way, and we can make full use of them to improve the development experience! @tisonkun had a good suggestion in https://github.com/apache/incubator-kvrocks/issues/562#issuecomment-1123771508 of creating a tracking issue to help keep the discussion more focused, so this issue is created for that purpose.

    Solution

    The list below tracks some of the improvements that have been proposed so far, and I'd appreciate it if anyone came up with new ideas for improvements, PRs for current improvements, or participated in this discussion!

    • [x] #561
    • [x] #562
    • [x] #574
    • [x] #569
    • [x] #563
    • [x] #580
    • [x] #589
    • [x] #590

    Are you willing to submit a PR?

    • [X] I'm willing to submit a PR!
    enhancement 
    opened by PragmaTwice 16
  • Memory leaks in Lua::redisGenericCommand

    Memory leaks in Lua::redisGenericCommand

    Search before asking

    • [X] I had searched in the issues and found no similar issues.

    Version

    redisGenericCommand use raiseError to process errors, but raiseError will perform a longjmp in the lua_error api call, so raiseError never returns.

    longjmp does not compatible with RAII in C++ (C++ exceptions may use longjmp in implementation but the compiler will insert some dtor call autometically and carefully, i.e. stack unwinding), so every object in the stack of redisGenericCommand will never be destroyed, i.e. no destructor will be call, hence all resource allocated in ctor like std::vector, std::unique_ptr etc. cannot be released.

    @git-hulk

    Minimal reproduce step

    ./runtest --dont-clean on a kvrocks binary with Address Sanitizer enabled. (try to build with #599) You can only run tests in unit/scripting to save time : )

    What did you expect to see?

    No memory leaks

    What did you see instead?

    Memory leaks are reported by Address Sanitizer

    Anything Else?

    No response

    Are you willing to submit a PR?

    • [X] I'm willing to submit a PR!
    bug 
    opened by PragmaTwice 15
  • [BUG]  Failed to open DB with missing CURRENT file

    [BUG] Failed to open DB with missing CURRENT file

    Describe the bug A clear and concise description of what the bug is.

    • OS: MAC OS 10.15.7
    • Version 2.0.5 @6a4ed16a

    get following error message when try to start up kvrocks ( first time run )

    E1229 15:41:02.217978 276704704 main.cc:314] Failed to open: IO error: No such file or directory: While opening a file for sequentially reading: /tmp/kvrocks/db/CURRENT: No such file or directory

    /tmp/kvrocks directory did exist but no sub-directory ./db under it.

    opened by jasonhe88 15
  • Wrong substring in kvrocks2redis config

    Wrong substring in kvrocks2redis config

    Search before asking

    • [X] I had searched in the issues and found no similar issues.

    Version

    unstable

    Minimal reproduce step

    https://github.com/apache/incubator-kvrocks/blob/44ffb9a742792fd559b4147678fd1dbb1e5bc9d4/utils/kvrocks2redis/config.cc#L109

    What did you expect to see?

    --

    What did you see instead?

    --

    Anything Else?

    No response

    Are you willing to submit a PR?

    • [X] I'm willing to submit a PR!
    bug 
    opened by PragmaTwice 0
  • Persist the cluster nodes info after applying the cluster topology

    Persist the cluster nodes info after applying the cluster topology

    This closes #1021

    Currently, the cluster nodes' info is only stored in memory and we need to re-sync the cluster topology after restarting. It's very inconvenient and confusing for most users.

    Solutoin

    • Persist the cluster nodes' info in the local disk if the topology was changed. The file location is {config->dir}/nodes.conf and the format is below:
    version 2
    id 07c37dfeb235213a872192d90877d0cd55635b92
    node 07c37dfeb235213a872192d90877d0cd55635b91 127.0.0.1 59129 master -  1-2 4-8193 10000 10002-11002 16381-16383
    node 07c37dfeb235213a872192d90877d0cd55635b92 127.0.0.1 59137 master -  0
    
    • Load and parse the nodes.conf if the cluster mode is enabled and the nodes file is exists
    opened by git-hulk 2
  • Allow to enable the async_io option to improve the performance

    Allow to enable the async_io option to improve the performance

    Here are some comparisons between async_io and normal

    big value 500B

    image

    small value 50B

    image

    monitor diff

    The green line is kvrocks compiled by async io,this pr. ts* command using seek function to scan data from db. image

    summarize

    | | cpu | mem | latency | qps | | | ------ | ---- | ---- | -------- | ---- | --- | | async | 110% | 100% | 100% | 120% | | | normal | 100% | 100% | 130-400% | 100% | |

    async_io is better in seek read, higher qps, lower latency, and slightly higher cpu usage. Perfect for introducing projects

    opened by xiaobiaozhao 2
  • BITOPS doesn't support BIT option

    BITOPS doesn't support BIT option

    Search before asking

    • [X] I had searched in the issues and found no similar issues.

    Version

    latest

    Minimal reproduce step

    from redis command : https://redis.io/commands/bitpos/ redis> SET mykey "\x00\xff\xf0" "OK"

    What did you expect to see?

    redis> BITPOS mykey 1 7 15 BIT (integer) 9

    What did you see instead?

    redis> BITPOS mykey 1 7 15 BIT (integer) 57

    Anything Else?

    No response

    Are you willing to submit a PR?

    • [ ] I'm willing to submit a PR!
    A-docs enhancement 
    opened by lt1946 3
  • Failed test case in CI: TestSlotMigrateThreeNodes

    Failed test case in CI: TestSlotMigrateThreeNodes

    Search before asking

    • [X] I had searched in the issues and found no similar issues.

    Version

    unstable

    Minimal reproduce step

    https://github.com/apache/incubator-kvrocks/actions/runs/3749755928/jobs/6368682143#step:12:158

    What did you expect to see?

    pass

    What did you see instead?

    fail

    Anything Else?

    No response

    Are you willing to submit a PR?

    • [ ] I'm willing to submit a PR!
    bug 
    opened by PragmaTwice 0
Releases(v2.2.0)
  • v2.2.0(Nov 10, 2022)

    Notice

    For this release, we are still using rocksdb 6.29, but we will update to rocksdb 7 in the next release (as in the unstable branch).

    New Features

    • Allow using LuaJIT to improve the performance of the Lua script (#697)
    • Add the new command GETDEL (#771)
    • Add the new command SMISMEMBER (#778)
    • Add the new data structure Redis STREAM (#745)
    • Add TLS support for kvrocks connections (#797)
    • Add the new command HELLO (#881)
    • Add the new command EVAL_RO (#782)
    • Add the new command DISK (#882)
    • Add the new command GETEX (#961)
    • Support CLI options for the kvrocks server (#1037)
    • Add the new command options for ZADD (#1022)
    • Add ZLIB dependency for compression in rocksdb (#1078)

    Improvements

    • Switch the default value of log-dir to stdout (#754)
    • Add fpm packaging to x.py (#752)
    • Propagate sanitizer flags to all cmake deps (#759)
    • Move runtest to x.py with customized server/cli path (#765)
    • Add StatusOr for error handling in modern C++ style (#768)
    • Add ParseInt in StatusOr style (#787)
    • Show function name in backtrace (#796)
    • Optimize get bitmap string and remove redundant num2bit array (#793)
    • Allows Kvrocks to listen to only the unix socket (#809)
    • Update dependency to latest version (lz4, libevent, gtest) (#828)
    • Export the rocksdb write options (#885)
    • Support quoted string and inline comment in config (#849)
    • Add the FilterBlobByKey for SubKeyFilter (#902)
    • Avoid using Get when iterating sub-keys during migration (#906)
    • Support EXAT/PEXAT option in the set command (#901)
    • Return from HRANGE if the key wasn't found (#923)
    • Update jemalloc to 5.3.0 (#940)
    • Use ParseInt to replace sto*, ato* (#924)
    • Add UniqueFD and ScopeExit (#973)
    • Align georadius behavior of redis (#993)
    • Align the expire option behavior of redis (#1017)
    • Support config set backup-dir new-dir (#1026)
    • Use a consistent way to get the current time (#1038)
    • Add support for showing used_memory_startup (#1044)
    • Improve stack trace format and clean main.cc (#1053)
    • Add command parser (#1032)
    • Add WriteBatchInspector to inspect WriteBatch (#1069)
    • Improve libunwind linking in CMake (#1079)

    Bug Fixes

    • Fix didn't return the out of range error in cluster import command(#767)
    • Fix should forbid to create of the default namespace (#772)
    • Fix CMake configuration error in snappy (#803)
    • Fix inline protocol don't allow LF only EOL (#808)
    • Fix successor commands won't be processed before receiving the next read event (#839)
    • Fix georadius should return an empty set if the key does not exist (#845)
    • Fix config set rocksdb.blob_garbage_collection_age_cutoff (#914)
    • Fix ping with arg should return a bulk string (#933)
    • Fix server cannot exit properly when enabling cluster mode (#969)
    • Fix shouldn't lowercase the configuration value in kvrocks2redis(#971)
    • Fix don't duplicate killed clients which have already flagged kCloseAfterReply (#1020)
    • Fix properly parse cmake version (#1043)
    • Fix rename-command only the first one works(#1047)
    • Fix CAS and CAD return statuses (#1055)
    • Fix random crash in the migration test case (#1068)

    Test Refactoring

    TCL tests are now moved to Go, refer to #811.

    Misc

    • Replace cpplint with clang-format (#979)
    • Refactor source file structure (#989)
    • Update C++ standard to 17 (#1006)
    • Use clang-tidy instead of cppcheck (#1025)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Aug 3, 2022)

    New features

    • Support source packaging by x.py (#729)
    • Support to use lz4 compression in RocksDB (#584)
    • Allow to send logs to stdout/stderr (#613)
    • LPOP/RPOP support pop multi elements (#594)
    • Add LMove command (#577)
    • Support to listen on the IPV6 address (#554)
    • Add support for listening to unix socket (#531)
    • Support BITOP command (#471)
    • Support unlink command (#517)

    Improvements

    • replace shell scripts with x.py (#725)
    • Use lock_guard to replace the manual mutex lock/unlock (#723)
    • Build docker image with travis ci (#724)
    • Return values instead of passing pointers (#722)
    • Don't call GetCurrentTime in expired and type when possible (#709)
    • Remove the slice ToString() when encoding or decoding the key (#707)
    • Use the const reference for avoiding copy (#704)
    • Check sanitizer status for TCL tests in CI (#698)
    • CMake: support ninja as a build backend (#626)
    • Build static lib for glog (#618)
    • Improve using of ASan and TSan in CMake build (#599)
    • PSYNC based on Unique Replication Sequence ID (#538)
    • Remove handwritten makefiles to keep only one build system (#576)
    • Use FetchContent instead of ExternalProject and git submodules in CMake (#564)
    • Upgrade the rocksdb version to 6.29.5(latest version for 6.x) (#555)
    • Support to send auth command before migrating slots (#514)
    • Allow to dynamic resize the level_compaction_dynamic_level_bytes and level base size (#497)

    Bugfixes

    • Make the set command consistent with redis (#730)
    • Fix some time dependencies in replication.tcl (#688)
    • Fix get rocksdb ops tcl test case (#668)
    • Fix typos in config file (#669)
    • Fix replication tests should wait for server restarted (#660)
    • Fix the role command can't be used when loading data (#661)
    • Fix Wrongly parsed the RESP empty/null array (#652)
    • Fix compile warnings in kvrocks (#653)
    • Fix unmatched iterator in Cluster::SetClusterNodes (#646)
    • Fix load namespaces from the config file would ignore case (#642)
    • Fix RocksDB can't auto resume after disk quota exceeded error (#628)
    • Fix don't swallow the error when creating column families failed (#620)
    • Compile lua by C++ compilers to avoid memory leaks (#614)
    • Fix the slave may crash when restoring the db (#606)
    • Fix leaks reported by ASan (#605)
    • Fix can't find jemalloc when building with makefile on MacOSX (#557)
    • Stop sending files if failed in full synchronization (#539)
    • Avoid requirepass and masterauth conflicts with the namespace tokens (#507)
    • Fix slave can't resync with master after password change (#520)
    • Fix rocksdb cmake depends should use list instead of string (#518)
    • Fix spop command should return bulk string instead of array when without count(#515)
    • Fix wrong replication state (#506)
    • Avoid flush all redis db in kvrocks2redis (#498)
    • Fix failing in setting thread name with string more than 16 byte (#496)
    • Avoid accessing slot_migrate_ before it is created (#472)
    • Fix the connection fd was used after free (#479)

    Others

    • Add use case CIRCL.lu (#546)
    • Change license to Apache 2.0 (#533)
    • Add DISCLAIMER (#558)
    • Apply Apache License header for Tcl files (#548)
    • Add .asf.yaml file (#545)
    • Apply Apache License header for source code (#543)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.6(Jan 28, 2022)

    Thanks ChrisZMF who implements slot-based data migration for Kvrocks cluster mode, this authentically makes kvrocks cluster scalable. Even though it already is used in production environments, you still should be cautious of it.

    Here is a comprehensive list of changes in this release compared to 2.0.5

    New features

    • Support slot-based data migration (#430) Biggest feature of this version, authentically makes kvrocks scalable, user can use CLUSTERX MIGRATE command to migrate slot and use CLUSTERX SETSLOT to change cluster slot distribution after finishing migration.
    • Make level0_file_num_compaction_trigger of RocksDB configurable (#455)
    • Support CLUSTERX SETSLOT command (#463)

    Improvements

    • Support executing INFO command during restoring database (#375)
    • Auto Rewrite the config after the namespace was updated (#423)
    • Support to convert sortedint type to zset type in kvrocks2redis (#431)
    • Optimize the seek of RocksDB iterators (#438)
    • Make the bitmap range the same with that of Redis (#465)
    • Support GET command on bitmap type (#447)
    • Support to use row cache of RocksDB (#407)

    Bug fixes

    • Fix the parameter blob_garbage_collection_age_cutoff would be always zero (#425)
    • Fix config rewrite overwrite the first rename-command with the last one (#429)
    • Fix zrevrangebyscore for multi members with the same max score (#434)
    • Fix data race in SetCurrentConnection (#440)
    • Fix failing full replication when enabling slave-empty-db-before-fullsync (#446)
    • Fix memory leak when full replication (#461) Maybe leak dozen kilobytes per full replication
    • Fix the bitmap range is smaller than that of Redis (#465)
    Source code(tar.gz)
    Source code(zip)
    kvrocks-2.0.6-release.x86_64.rpm(129.69 MB)
    kvrocks_2.0.6-release_amd64.deb(118.73 MB)
  • v2.0.5(Dec 7, 2021)

    Here is a comprehensive list of changes in this release compared to 2.0.4

    New features

    • Implement zrevrangebylex command (#381)

    • Use some rocksdb new feature (#395)

      • Support blob db (key-value separation) which requires RocksDB version greater than 6.18
      • Enable partitioned index/filter for SST and hash index for data block
      • Enable whole key filter for memtable
      • Remove dynamic resize block and SST
      • Only apply compression to SST which level is more than 2

      Please note that, if your rocksdb version is lower than 6.6.0, it can't reload the data of new version, you should update cautiously.

    • Implement CAS and CAD commands for string type (#415)

    • Support to use row cache of RocksDB (#407)

    Improvements

    • Add rocksdb ops statistics into the info command (#387) These statistics can more truly reflect the load of rocksdb.
    • Add the maxclients to the info command output (#391)
    • Add write stall/stop stats to info command output (#396)
    • Make kvrocks2redis work (#404)
    • Adjust the max open file limit according to the max clients and rocksdb max open files (#418)

    Related projects

    • Kvrocks exporter, a fork of oliver006/redis_exporter to export the kvrocks metrics, https://github.com/KvrocksLabs/kvrocks_exporter
    Source code(tar.gz)
    Source code(zip)
    kvrocks-2.0.5-release.x86_64.rpm(127.80 MB)
    kvrocks_2.0.5-release_amd64.deb(116.75 MB)
  • v2.0.4(Nov 2, 2021)

    Here is a comprehensive list of changes in this release compared to 2.0.3

    New features

    • Support LUA script commands (#363) (#369) Currently, support EVAL, EVALSHA, and SCRIPT commands, but SCRIPT KILL and DEBUG subcommands are not supported.

    Improvements

    • Allow metadata and subkey CF to share a single block cache (#365) (#373)
    • Add master_repl_offset in INFO command output (#374)

    Bugfixes

    • Fixed dynamic resize target_file_size_base bug (#368) Before we only apply this strategy to the default CF(i.e. subkey CF)
    Source code(tar.gz)
    Source code(zip)
    kvrocks-2.0.4-release.x86_64.rpm(127.47 MB)
    kvrocks_2.0.4-release_amd64.deb(116.46 MB)
  • v2.0.3(Sep 13, 2021)

    If your kvrocks instances are 2.0.2 and use redis-sentinel to implement HA, you should upgrade urgently, please see #352 for details.

    Here is a comprehensive list of changes in this release compared to 2.0.2

    Improvements

    • Use the rocksdb MultiGet interface in the MGET command (#331)
    • Avoid useless space allocation in SetBit (#338)
    • Auto set master/replica replication by cluster topology (#356)
    • Support client kill type subcommands to adapt redis-sentinel (#352) (#361) From 2.0.2, we support MULTI-EXEC, but don't support client kill type $type, so MULTI-EXEC will be rollbacked when failing to parse unknown kill type in client command, and cause redis-sentinel fails to failover.

    Bugfixes

    • Fix SETNX and MSETNX commands can't guarantee atomicity (#337)
    • Fix the replicas won't reconnect when failing to connect with master (#344)
    • Fix master pauses synchronizing with replicas (#347)
    • Fix wrong mater port if enabling master-use-repl-port (#360)
    Source code(tar.gz)
    Source code(zip)
    kvrocks-2.0.3-release.x86_64.rpm(126.69 MB)
    kvrocks_2.0.3-release_amd64.deb(115.68 MB)
  • v2.0.2(Jul 19, 2021)

    We design new cluster solution for kvrocks, but it is experimental, don't recommend to deploy in production environments. And because in cluster mode, to be efficient to migrate keys based on slot when scale, we encode key with slot id, but in order to be compatible with the old version, we don't do that in standalone mode, so data is not compatible between standalone mode with cluster mode, you must migrate data if you want to change mode, otherwise, kvrocks will make data corrupt. In order to better support the cluster mode, we are developing a meta server to manage kvrocks cluster, coming soon!

    Here is a comprehensive list of changes in this release compared to 2.0.1

    New features

    • Support rename command in config file (#272)
    • Support transaction (#285) Support MULTI, EXEC, DISCARD command, but don't support WATCH and UNWATCH commands.
    • Support the auto-resize-block-and-sst config directive (#289)
    • Support to resize rocksdb.block_size automatically (#294)
    • Support cluster mode (#302) Similar with redis cluster mode, but use CLUSTERX command to set cluster topology (#324).

    Data encoding changes

    • Encode key with slot id (#291) (#330) In cluster mode, to be efficient to migrate keys by slot when scale, we encode key with slot id, but in standalone mode, we don't. That is to say, data is not compatible between standalone mode with cluster mode, you must migrate data if you want to change mode, otherwise, kvrocks will make data corrupt.

    Bug fixes

    • Set TCP_KEEPALIVE for replication connection (#284) Replicas can handle network failure with master, before this commit, network failure may result in unrecoverable replication interrupt.
    • Fix ZSET can't add the same member with different scores (#298)
    • Make CONFIG SET command thread safe (#310)
    • Fix LTRIM only delete first item, LREM can't correctly remove repeated items (#314)

    Improvements

    • HSET command supports to set multiple fields (#274)
    • Allow to use host in the master-slave replication (#301)
    • Adapt redis sentinel ping check (#316)
    Source code(tar.gz)
    Source code(zip)
    kvrocks-2.0.2-release.x86_64.rpm(126.28 MB)
    kvrocks_2.0.2-release_amd64.deb(115.27 MB)
  • v2.0.1(May 18, 2021)

    Here is a comprehensive list of changes in this release compared to 2.0.0

    New features

    • Support COMMAND command (#251)
    • Support to purge backups on fullsync to reduce disk space usage (#245)

    Bug fixes

    • Fix uninitialized Rocksdb.level0_stop_writes_trigger (#236)
    • Fix condition race for ReclaimOldDBPtr (#246)
    • Fix data race for accessing database in some commands (#253) (#261) In spop, zremrangebylex, zremrangebyscore commands, the lock guard for accessing the database may not work actually.
    • Fix returning wrong string range in GETRANGE command (#254)
    • Fix skipping wrong CURRENT file when full synchronization (#260) Resuming broken transfer based files may be not available if replicas skip transferred wrong CURRENT file of RocksDB.

    Improvements

    • Replicas can execute publish command (#238)
    • Optimizations for avoiding unnecessary data copy (#239) (#247)
    • Allow to run the kvrocks without assigning the config file (#264)

    Dependencies

    • Upgrade the jemalloc version to the special commit (#265) To be able to compile on macOS, upgrade jemalloc to the production test commit which mentioned in jemalloc/jemalloc#2060.
    Source code(tar.gz)
    Source code(zip)
    kvrocks-2.0.1-release.x86_64.rpm(125.30 MB)
    kvrocks_2.0.1-release_amd64.deb(114.31 MB)
  • v1.3.3(May 18, 2021)

    Here is a comprehensive list of changes in this release compared to 1.3.2

    Bug fixes

    • Fix rocksdb can't auto resume after no space error (#229)
    • Fix uninitialized Rocksdb.level0_stop_writes_trigger (#236)
    • Fix condition race for ReclaimOldDBPtr (#246)
    • Fix returning wrong string range in GETRANGE command (#254)
    • Fix data race for accessing database in some commands (#253) In spop, zremrangebylex, zremrangebyscore commands, the lock guard for accessing the database may not work actually.

    Improvements

    • Replicas can execute publish command (#238)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.0(Apr 23, 2021)

    Kvrocks 2.0 brings some new useful features that solve actual problems in the production environment. Please note that this version instance still can replicate from old version master but old version replicas can't replicate new version master. Otherwise, we start to use new logo, thanks @tianlingyu1997

    Here is a comprehensive list of changes in this release compared to 1.3.2

    New features

    • Don't use extra port for replication (#200)
    • Use checkpoint to implement data snapshot when full replication (#208) Reduce disk space and bandwidth usage and decrease the time of replicas recovery during full synchronization.
    • Use checkpoint to implement kvrocks backup (#232)
    • Replicas can empty db before fullsync (#233) Replicas use less disks space if enabled, but have a risk to lose data.

    Command changes

    • Add QPS and input/output kbps for INFO command (#213)
    • Add pubsub_patterns for INFO command (#234)
    • Slowlog omits args or argument string if too many or big (#215)

    Cluster

    • Remove codis support (#217) From now, we doesn't support codis, you can adopt pre-sharding to implement cluster, and we are developing new cluster solution that is similar with redis cluster mode, please see #219.

    Bugfixes

    • Fix kvrocks can't auto resume after no space error (#229)

    Dependencies

    • Upgrade rocksdb to latest tag v6.19.3 (#226)
    Source code(tar.gz)
    Source code(zip)
  • v1.3.2(Apr 20, 2021)

    Here is a comprehensive list of changes in this release compared to 1.3.1

    Bug fixes

    • Fix incorrect used_db_size in INFO command (#204)
    • Fix the SST file creation time maybe 0 in some unknown conditions (#211)
    • Fix get corruption writebatch data after psync restart (#221)

    Improvements

    • Optimize TCP keepalive detection time (#201)
    Source code(tar.gz)
    Source code(zip)
  • v1.3.0(Feb 8, 2021)

    Here is a comprehensive list of changes in this release compared to 1.2.0

    License

    • Change the license from MIT to BSD-3-Clause

    New features

    • Support to set separate log directory to store server logs (#155)
    • Implement PSETEX command (#156)
    • Implement MSETNX command (#165)

    Improvements

    • Improve the precision of the INCRBYFLOAT/HINCRBYFLOAT command to double (#169)
    • Support +INF/-INF for the score of zset type members (#172)
    • The UNSUBSCRIBE and PUNSUBSCRIBE commands support multiple channels or patterns (#174)

    Behavior changes

    • The UNSUBSCRIBE and PUNSUBSCRIBE commands have the same reply as Redis (#174)
    • The same message format as Redis for pattern subscribers when publishing message (#175) Message format is from 'message channel message' to 'pmessage pattern channel message'

    Bug fixes

    • Fix skipping some SST files when picking compaction files (#154)
    • Handle negative expire time in set command to avoid deleting keys (#156)
    • List type command doesn't rewrite other types of data (#157)
    • Only set positive expire time when migrating key (#160)
    • Don't store an empty key for setrange command (#161)
    • Fix SETBIT/GETBIT bit offset out of range (#164)
    • Fix INCRBY/INCRBYFLOAT command operates on wrong value and precision check (#167)
    • Fix HINCRBY/HINCRBYFLOAT command operates on wrong value and precision check (#168)
    • RPOPLPUSH command doesn't operate non list type (#170)
    • LINSERT command returns -1 if don't find index (#170)
    • Fix LTRIM command range calculation when start is a very large negative number (#170)
    • SMOVE command doesn't operate non set type key (#171)
    • Fix ZREMRANGEBYRANK command corrupts the size of zset (#172)
    • SUBSCRIBE and PSUBSCRIBE commands return the sum of subscription and psubscription (#174)
    • Fix PUNSUBSCRIBE command doesn't take effort (#174)
    • Allow setting slowlog-log-slower-than to -1 to disable the slowlog (#178)
    Source code(tar.gz)
    Source code(zip)
  • v1.1.27(Aug 26, 2020)

  • v1.1.6(Feb 28, 2020)

    • Upgrade the libevent to stable 2.1.11
    • FIX: create the backup at the same time may cause a crash while the CreateNewBackupWithMetadata wasn't thread-safe call
    • FIX: some incorrect implements in redis command
    Source code(tar.gz)
    Source code(zip)
  • v1.0.1(Aug 23, 2019)

Owner
Kvrocks Labs
Home of Kvrocks
Kvrocks Labs
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB

Facebook 24.3k Jan 5, 2023
KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language

KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language. It is designed for persistent memory and provides unified APIs for both volatile and persistent scenarios. It also demonstrates several optimization methods for high performance with persistent memory. Besides providing the basic APIs of key-value store, it offers several advanced features, like transaction, snapshot as well.

Persistent Memory Programming 186 Nov 16, 2022
RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis.

A query and indexing engine for Redis, providing secondary indexing, full-text search, and aggregations.

null 4k Jan 5, 2023
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large-scale graphs with billions of vertices (nodes) and trillions of edges, with milliseconds of latency. It delivers enterprise-grade high performance to simplify the most complex data sets imaginable into meaningful and useful information.

vesoft inc. 8.4k Jan 9, 2023
🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

?? ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

ArangoDB 12.8k Jan 9, 2023
Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB. Scylla embraces a shared-nothing approach that increases throughput and storage capacity to realize order-of-magnitude performance improvements and reduce hardware costs.

ScyllaDB 8.9k Jan 4, 2023
FEDB is a NewSQL database optimised for realtime inference and decisioning application

FEDB is a NewSQL database optimised for realtime inference and decisioning applications. These applications put real-time features extracted from multiple time windows through a pre-trained model to evaluate new data to support decision making. Existing in-memory databases cost hundreds or even thousands of milliseconds so they cannot meet the requirements of inference and decisioning applications.

4Paradigm 1.2k Jan 2, 2023
The MongoDB Database

The MongoDB Database

mongodb 23k Jan 1, 2023
Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.

Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.

Bit Leak 1.9k Jan 8, 2023
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB

Facebook 24.3k Jan 5, 2023
KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language

KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language. It is designed for persistent memory and provides unified APIs for both volatile and persistent scenarios. It also demonstrates several optimization methods for high performance with persistent memory. Besides providing the basic APIs of key-value store, it offers several advanced features, like transaction, snapshot as well.

Persistent Memory Programming 186 Nov 16, 2022
FoundationDB - the open source, distributed, transactional key-value store

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as

Apple 12k Dec 31, 2022
RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis.

A query and indexing engine for Redis, providing secondary indexing, full-text search, and aggregations.

null 4k Jan 5, 2023
redis-cpp is a header-only library in C++17 for Redis (and C++11 backport)

redis-cpp - lightweight C++ client library for Redis redis-cpp is a C++17 library for executing Redis commands with support for pipelines and the publ

null 77 Dec 11, 2022
A redis module, similar to redis zset, but you can set multiple scores for each member to support multi-dimensional sorting

TairZset: Support multi-score sorting zset Introduction Chinese TairZset is a data structure developed based on the redis module. Compared with the na

Alibaba 60 Dec 1, 2022
libmdbx is an extremely fast, compact, powerful, embedded, transactional key-value database, with permissive license

One of the fastest embeddable key-value ACID database without WAL. libmdbx surpasses the legendary LMDB in terms of reliability, features and performance.

Леонид Юрьев (Leonid Yuriev) 1.1k Dec 19, 2022
An Embedded NoSQL, Transactional Database Engine

UnQLite - Transactional Embedded Database Engine

PixLab | Symisc Systems 1.8k Dec 24, 2022
Kunlun distributed DBMS is a NewSQL OLTP relational distributed database management system

Kunlun distributed DBMS is a NewSQL OLTP relational distributed database management system. Application developers can use Kunlun to build IT systems that handles terabytes of data, without any effort on their part to implement data sharding, distributed transaction processing, distributed query processing, crash safety, high availability, strong consistency, horizontal scalability. All these powerful features are provided by Kunlun.

zettadb 114 Dec 26, 2022
John Walker 24 Dec 15, 2022
Kreon is a key-value store library optimized for flash-based storage

Kreon is a key-value store library optimized for flash-based storage, where CPU overhead and I/O amplification are more significant bottlenecks compared to I/O randomness.

Computer Architecture and VLSI Systems (CARV) Laboratory 24 Jul 14, 2022