Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Last update: Oct 19, 2021

Overview

Kvrocks is an open-source key-value database which is based on rocksdb and compatible with Redis protocol. Intention to decrease the cost of memory and increase the capability while compared to Redis. The design of replication and storage was inspired by rocksplicator and blackwidow.

Kvrocks has the following key features:

Redis protocol, user can use redis client to visit the kvrocks
Namespace, similar to redis db but use token per namespace
Replication, async replication using binlog like MySQL
High Available, supports redis sentinel to failover when master or slave was failed

Thanks for designers @田凌宇 and @范世丽 contribute the kvrocks logo for us.

Who uses kvrocks

Tickets a pull reqeust to let us known that you're using kvrocks and add your logo to README

Building kvrocks

requirements

g++ (required by c++11, version >= 4.8)
autoconf automake libtool snappy

Build

NOTE: You should install the snappy first:

# Centos/Redhat
sudo yum install -y epel-release && sudo yum install -y git gcc gcc-c++ make snappy snappy-devel autoconf automake libtool which gtest gtest-devel

# Ubuntu
sudo apt-get install gcc g++ make libsnappy-dev autoconf automake libtool which libgtest-dev

# MACOSX
brew install snappy googletest

It is as simple as:

$ git clone --recursive https://github.com/kvrockslabs/kvrocks.git
$ cd kvrocks
$ make -j4

Running kvrocks

$ ./src/kvrocks -c kvrocks.conf

Running test cases

NOTE: You should install the googletest first

make test

Supported platforms

centos 6/7
ubuntu
macosx

Try kvrocks using Docker

$ docker run -it -p 6666:6666 kvrocks/kvrocks
$ redis-cli -p 6666

127.0.0.1:6666> get a
(nil)

Namespace

namespace was used to isolate data between users. unlike all the redis databases can be visited by requirepass, we use one token per namespace. requirepass was regraded as admin token, only admin token allows to access the namespace command, as well as some commands like config, slaveof, bgsave, etc…

namespace del ns1 OK ">

# add token
127.0.0.1:6666>  namespace add ns1 mytoken
OK

# update token
127.0.0.1:6666> namespace set ns1 new_token
OK

# list namespace
127.0.0.1:6666> namespace get *
1) "ns1"
2) "new_token"
3) "__namespace"
4) "foobared"

# delete namespace
127.0.0.1:6666> namespace del ns1
OK

DOCs

Migrate tools

migrate from redis to kvrocks, use redis-migrate-tool which was developed by vipshop
migrate from kvrocks to redis. use kvrocks2redis in build dir

Performance

Hardware

CPU: 48 cores Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
Memory: 32 GiB
NET: Intel Corporation I350 Gigabit Network Connection
DISK: 2TB NVMe Intel SSD DC P460

Benchmark Client: multi-thread redis-benchmark(unstable branch)

1. Commands QPS

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns / 128 payload

latency: 99.9% < 10ms

2. QPS on different payloads

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns

latency: 99.9% < 10ms

3. QPS on different workers

kvrocks: workers = 16, benchmark: 8 threads/ 512 conns / 128 payload

latency: 99.9% < 10ms

License

Kvrocks is under the BSD-3-Clause license. See the LICENSE file for details.

Issues

同步基本不可用

masterA 有数据 slaveA slaveof过来，数据可以同步，flushall masterA的数据， slaveA 没有同步清空掉。都执行过dbsize scan 2)masterB 没数据 slaveB 同步过来up后，masterB上执行写数据,数据没有同步到slaveB 3)slaveof后 slave节点上经常会报以下错误, 但这个目录是清空后再启动slave的

E0216 13:35:53.319345 26935 storage.cc:703] [storage] Failed to delete dir: IO error: file rmdir: ./kvrocksdatamaster/backup/meta: Directory not empty
E0216 13:35:53.415446 26935 replication.cc:584] [replication] Failed to restore backup while IO error: No such file or directoryWhile opening a file for sequentially reading: ./kvrocksdatamaster/backup/meta/5: No such file or directory
Segmentation fault (core dumped)

need more information

opened by smartlee 23

Enhancement/only generate version when needed

resolved #66

opened by karelrooted 11

deadlock

目前发现一个deadlock，原始代码没有复现，我改写的代码会出现，大概几天会出现一次，看原始代码觉得可能也有这个隐患，以下是deadlock时候的堆栈，如有不正确，请指正谢谢！

Thread 7 (Thread 0x7fe5e47f8700 (LWP 97)):
#0  0x00007fe5f8f4ef4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fe5f8f4ad02 in _L_lock_791 () from /lib64/libpthread.so.0
#2  0x00007fe5f8f4ac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000004437d8 in __gthread_mutex_lock(pthread_mutex_t*) ()
#4  0x0000000000443b10 in std::mutex::lock() ()
#5  0x00000000004db93a in std::lock_guard<std::mutex>::lock_guard(std::mutex&) ()
#6  0x00000000004f7731 in Server::UnSubscribeChannel(std::string const&, Redis::Connection*) ()
#7  0x00000000004a7eee in Redis::Connection::UnSubscribeAll() ()
#8  0x00000000004a723d in Redis::Connection::~Connection() ()
#9  0x00000000005223bd in Worker::FreeConnectionByID(int, unsigned long) ()
#10 0x00000000005235b2 in Worker::KickoutIdleClients(int) ()
#11 0x000000000052158e in Worker::TimerCB(int, short, void*) ()

Thread 2 (Thread 0x7fe5e7fff700 (LWP 129)):
#0  0x00007fe5f8f4ef4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fe5f8f4ad02 in _L_lock_791 () from /lib64/libpthread.so.0
#2  0x00007fe5f8f4ac08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000004437d8 in __gthread_mutex_lock(pthread_mutex_t*) ()
#4  0x0000000000443b10 in std::mutex::lock() ()
#5  0x00000000004efc87 in std::unique_lock<std::mutex>::lock() ()
#6  0x00000000004ee65d in std::unique_lock<std::mutex>::unique_lock(std::mutex&) ()
#7  0x0000000000522665 in Worker::Reply(int, std::string const&) ()
#8  0x00000000004f7288 in Server::PublishMessage(std::string const&, std::string const&) ()
#9  0x000000000048d06f in Redis::CommandPublish::Execute(Server*, Redis::Connection*, std::string*) ()

1）一个worker获得了server的锁 pubsub_channels_mu_ 等待worker的锁conns_mu_ 线程2 另外一个定时器触发，到时间清理timeout的链接线程7 获得了worker的锁conns_mu_在connection析构时会等待server的锁 pubsub_channels_mu_，就卡死在这里

2） KickoutIdleClients 这个函数的行为不是很理解，按理是清理conns_和monitor_conns_的连接数，但这里面并没有遍历monitor_conns_的map,却在FreeConnectionByID里面去找monitor_conns_，此外这个地方难道不是直接map erase掉conns_和monitor_conns_，并将符合条件的加入到to_be_killed_conns里面更好？这样FreeConnectionByID无需加锁。

bug

opened by smartlee 10

提个bug

https://github.com/meitu/kvrocks/blob/2ff46ef91d47f37945354d3a9e77ad49f33c4ce0/src/redis_request.cc#L49 这地方不能return ok,否则可能会导致命令断流，解析出错，我这边可以稳定复现，只能break。
bug need more information

opened by smartlee 10

Fix data race for accessing database in some commands

Describe this PR

Found several data races with -fsanitize=thread flags

1. Data race when starting up the server

Output:

==================
WARNING: ThreadSanitizer: data race (pid=11804)
  Read of size 8 at 0x7b0400004900 by thread T16:
    #0 operator() /home/eagle/github/my/kvrocks/src/worker.cc:362 (kvrocks+0x222b68)
    #1 __invoke_impl<void, WorkerThread::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:60 (kvrocks+0x222b68)
    #2 __invoke<WorkerThread::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:95 (kvrocks+0x222b68)
    #3 _M_invoke<0> /usr/include/c++/9/thread:244 (kvrocks+0x222b68)
    #4 operator() /usr/include/c++/9/thread:251 (kvrocks+0x222b68)
    #5 _M_run /usr/include/c++/9/thread:195 (kvrocks+0x222b68)
    #6 <null> <null> (libstdc++.so.6+0xd6d83)

  Previous write of size 8 at 0x7b0400004900 by main thread:
    #0 std::enable_if<std::__and_<std::__not_<std::__is_tuple_like<std::thread::id> >, std::is_move_constructible<std::thread::id>, std::is_move_assignable<std::thread::id> >::value, void>::type std::swap<std::thread::id>(std::thread::id&, std::thread::id&) /usr/include/c++/9/bits/move.h:194 (kvrocks+0x223742)
    #1 std::thread::swap(std::thread&) /usr/include/c++/9/thread:159 (kvrocks+0x223742)
    #2 std::thread::operator=(std::thread&&) /usr/include/c++/9/thread:153 (kvrocks+0x223742)
    #3 WorkerThread::Start() /home/eagle/github/my/kvrocks/src/worker.cc:363 (kvrocks+0x223742)
    #4 Server::Start() /home/eagle/github/my/kvrocks/src/server.cc:60 (kvrocks+0x1e9427)
    #5 __libc_start_main <null> (libc.so.6+0x270b2)

  Location is heap block of size 16 at 0x7b0400004900 allocated by main thread:
    #0 operator new(unsigned long) <null> (libtsan.so.0+0x8c012)
    #1 Server::Server(Engine::Storage*, Config*) /home/eagle/github/my/kvrocks/src/server.cc:35 (kvrocks+0x1ed2dd)
    #2 main /home/eagle/github/my/kvrocks/src/main.cc:310 (kvrocks+0x9471b)

  Thread T16 'worker' (tid=11882, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x5ea99)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xd7048)
    #2 Server::Start() /home/eagle/github/my/kvrocks/src/server.cc:60 (kvrocks+0x1e9427)
    #3 main /home/eagle/github/my/kvrocks/src/main.cc:317 (kvrocks+0x94839)

SUMMARY: ThreadSanitizer: data race /home/eagle/github/my/kvrocks/src/worker.cc:362 in operator()
==================

2. Thread Sanitizer warning in `make test`

==================
WARNING: ThreadSanitizer: data race (pid=12120)
  Read of size 1 at 0x7b7c00009031 by thread T1:
    #0 memcmp <null> (libtsan.so.0+0x651af)
    #1 rocksdb::Slice::compare(rocksdb::Slice const&) const include/rocksdb/slice.h:250 (unittest+0x56987d)
    #2 Compare util/comparator.cc:27 (unittest+0x56987d)
    #3 Engine::SubKeyFilter::Filter(int, rocksdb::Slice const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool*) const /home/eagle/github/my/kvrocks/src/compact_filter.cc:94 (unittest+0xdb539)
    #4 rocksdb::CompactionFilter::FilterV2(int, rocksdb::Slice const&, rocksdb::CompactionFilter::ValueType, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const include/rocksdb/compaction_filter.h:167 (unittest+0x5c4d32)
    #5 rocksdb::CompactionIterator::InvokeFilterIfNeeded(bool*, rocksdb::Slice*) db/compaction/compaction_iterator.cc:273 (unittest+0x5c4d32)

  Previous write of size 1 at 0x7b7c00009031 by main thread (mutexes: write M668075411208911328):
    #0 memcpy <null> (libtsan.so.0+0x42333)
    #1 memcpy /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34 (unittest+0x3c9539)
    #2 rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::ProtectionInfoKVOTS<unsigned long> const*, bool, rocksdb::MemTablePostProcessInfo*, void**) db/memtable.cc:555 (unittest+0x3c9539)
    #3 Redis::Set::Overwrite(rocksdb::Slice, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/eagle/github/my/kvrocks/src/redis_set.cc:31 (unittest+0x1c0a0b)
    #4 RedisSetTest_Overwrite_Test::TestBody() ../tests/t_set_test.cc:156 (unittest+0x2931b6)
    #5 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) googletest/src/gtest.cc:2433 (unittest+0x6b99a0)
    #6 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) googletest/src/gtest.cc:2469 (unittest+0x6b99a0)
    #7 __libc_start_main <null> (libc.so.6+0x270b2)

  Location is heap block of size 3216 at 0x7b7c00008c00 allocated by main thread:
    #0 operator new(unsigned long) <null> (libtsan.so.0+0x8c012)
    #1 rocksdb::ColumnFamilyData::ConstructNewMemtable(rocksdb::MutableCFOptions const&, unsigned long) db/column_family.cc:1057 (unittest+0x5b30b1)
    #2 Engine::Storage::Open() /home/eagle/github/my/kvrocks/src/storage.cc:227 (unittest+0x222707)
    #3 TestBase::TestBase() ../tests/test_base.h:15 (unittest+0x2958a8)
    #4 RedisSetTest::RedisSetTest() ../tests/t_set_test.cc:7 (unittest+0x2958a8)
    #5 RedisSetTest_Overwrite_Test::RedisSetTest_Overwrite_Test() ../tests/t_set_test.cc:152 (unittest+0x2961a9)
    #6 testing::internal::TestFactoryImpl<RedisSetTest_Overwrite_Test>::CreateTest() /usr/include/gtest/internal/gtest-internal.h:460 (unittest+0x2961a9)
    #7 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) googletest/src/gtest.cc:2433 (unittest+0x6b9b70)
    #8 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) googletest/src/gtest.cc:2469 (unittest+0x6b9b70)
    #9 __libc_start_main <null> (libc.so.6+0x270b2)

  Mutex M668075411208911328 is already destroyed.

  Thread T1 'rocksdb:low0' (tid=12122, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x5ea99)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xd7048)
    #2 Engine::Storage::Open(bool) /home/eagle/github/my/kvrocks/src/storage.cc:165 (unittest+0x2207af)
    #3 Engine::Storage::Open() /home/eagle/github/my/kvrocks/src/storage.cc:227 (unittest+0x222707)
    #4 TestBase::TestBase() ../tests/test_base.h:15 (unittest+0x25bbbd)
    #5 RedisTypeTest::RedisTypeTest() ../tests/t_metadata_test.cc:48 (unittest+0x25bbbd)
    #6 RedisTypeTest_GetMetadata_Test::RedisTypeTest_GetMetadata_Test() ../tests/t_metadata_test.cc:64 (unittest+0x25c249)
    #7 testing::internal::TestFactoryImpl<RedisTypeTest_GetMetadata_Test>::CreateTest() /usr/include/gtest/internal/gtest-internal.h:460 (unittest+0x25c249)
    #8 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) googletest/src/gtest.cc:2433 (unittest+0x6b9b70)
    #9 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) googletest/src/gtest.cc:2469 (unittest+0x6b9b70)
    #10 __libc_start_main <null> (libc.so.6+0x270b2)

SUMMARY: ThreadSanitizer: data race (/lib/x86_64-linux-gnu/libtsan.so.0+0x651af) in memcmp
==================

3. Thread Sanitizer warning when benchmarking

Benchmark command redis-benchmark -p 6666 -n 100000 -q

Full output

Part of the output:

==================
WARNING: ThreadSanitizer: data race (pid=20423)
  Write of size 8 at 0x7f1282ce71c0 by thread T19:
    #0 localtime <null> (libtsan.so.0+0x4701c)
    #1 operator() /home/eagle/github/kvrocks/src/server.cc:78 (kvrocks+0x15d669)
    #2 __invoke_impl<void, Server::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:60 (kvrocks+0x15d669)
    #3 __invoke<Server::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:95 (kvrocks+0x15d669)
    #4 _M_invoke<0> /usr/include/c++/9/thread:244 (kvrocks+0x15d669)
    #5 operator() /usr/include/c++/9/thread:251 (kvrocks+0x15d669)
    #6 _M_run /usr/include/c++/9/thread:195 (kvrocks+0x15d669)
    #7 <null> <null> (libstdc++.so.6+0xd6d83)

  Previous write of size 8 at 0x7f1282ce71c0 by thread T18:
    #0 localtime <null> (libtsan.so.0+0x4701c)
    #1 Server::cron() /home/eagle/github/kvrocks/src/server.cc:463 (kvrocks+0x15ea8a)
    #2 operator() /home/eagle/github/kvrocks/src/server.cc:66 (kvrocks+0x15f3b4)
    #3 __invoke_impl<void, Server::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:60 (kvrocks+0x15f3b4)
    #4 __invoke<Server::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:95 (kvrocks+0x15f3b4)
    #5 _M_invoke<0> /usr/include/c++/9/thread:244 (kvrocks+0x15f3b4)
    #6 operator() /usr/include/c++/9/thread:251 (kvrocks+0x15f3b4)
    #7 _M_run /usr/include/c++/9/thread:195 (kvrocks+0x15f3b4)
    #8 <null> <null> (libstdc++.so.6+0xd6d83)

  As if synchronized via sleep:
    #0 usleep <null> (libtsan.so.0+0x62f5b)
    #1 operator() /home/eagle/github/kvrocks/src/server.cc:94 (kvrocks+0x15d719)
    #2 __invoke_impl<void, Server::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:60 (kvrocks+0x15d719)
    #3 __invoke<Server::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:95 (kvrocks+0x15d719)
    #4 _M_invoke<0> /usr/include/c++/9/thread:244 (kvrocks+0x15d719)
    #5 operator() /usr/include/c++/9/thread:251 (kvrocks+0x15d719)
    #6 _M_run /usr/include/c++/9/thread:195 (kvrocks+0x15d719)
    #7 <null> <null> (libstdc++.so.6+0xd6d83)

  Location is global '<null>' at 0x000000000000 (libc.so.6+0x0000001f11c0)

  Thread T19 'compaction-checker' (tid=20504, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x5ea99)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xd7048)
    #2 main /home/eagle/github/kvrocks/src/main.cc:317 (kvrocks+0x8b0d9)

  Thread T18 'server-cron' (tid=20503, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x5ea99)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xd7048)
    #2 main /home/eagle/github/kvrocks/src/main.cc:317 (kvrocks+0x8b0d9)

SUMMARY: ThreadSanitizer: data race (/lib/x86_64-linux-gnu/libtsan.so.0+0x4701c) in localtime
==================
==================
WARNING: ThreadSanitizer: data race (pid=14056)
  Read of size 1 at 0x7b2400014ebc by thread T9:
    #0 memcpy <null> (libtsan.so.0+0x42333)
    #1 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) <null> (libstdc++.so.6+0x142ec3)
    #2 Redis::Set::Take(rocksdb::Slice const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, int, bool) /home/eagle/github/my/kvrocks/src/redis_set.cc:201 (kvrocks+0x19d8ea)
    #3 Redis::CommandSPop::Execute(Server*, Redis::Connection*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /home/eagle/github/my/kvrocks/src/redis_cmd.cc:1822 (kvrocks+0x14ffb9)
    #4 Redis::Request::ExecuteCommands(Redis::Connection*) /home/eagle/github/my/kvrocks/src/redis_request.cc:211 (kvrocks+0x195d1d)
    #5 Redis::Connection::OnRead(bufferevent*, void*) /home/eagle/github/my/kvrocks/src/redis_connection.cc:64 (kvrocks+0x16a3f9)
    #6 bufferevent_run_deferred_callbacks_unlocked /home/eagle/github/my/kvrocks/external/libevent/bufferevent.c:208 (kvrocks+0x250401)
    #7 operator() /home/eagle/github/my/kvrocks/src/worker.cc:362 (kvrocks+0x222b73)
    #8 __invoke_impl<void, WorkerThread::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:60 (kvrocks+0x222b73)
    #9 __invoke<WorkerThread::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:95 (kvrocks+0x222b73)
    #10 _M_invoke<0> /usr/include/c++/9/thread:244 (kvrocks+0x222b73)
    #11 operator() /usr/include/c++/9/thread:251 (kvrocks+0x222b73)
    #12 _M_run /usr/include/c++/9/thread:195 (kvrocks+0x222b73)
    #13 <null> <null> (libstdc++.so.6+0xd6d83)

  Previous write of size 8 at 0x7b2400014eb8 by thread T13:
    #0 operator new(unsigned long) <null> (libtsan.so.0+0x8c012)
    #1 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) <null> (libstdc++.so.6+0x142e7d)
    #2 Redis::CommandSPop::Execute(Server*, Redis::Connection*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /home/eagle/github/my/kvrocks/src/redis_cmd.cc:1822 (kvrocks+0x14ffb9)
    #3 Redis::Request::ExecuteCommands(Redis::Connection*) /home/eagle/github/my/kvrocks/src/redis_request.cc:211 (kvrocks+0x195d1d)
    #4 Redis::Connection::OnRead(bufferevent*, void*) /home/eagle/github/my/kvrocks/src/redis_connection.cc:64 (kvrocks+0x16a3f9)
    #5 bufferevent_run_deferred_callbacks_unlocked /home/eagle/github/my/kvrocks/external/libevent/bufferevent.c:208 (kvrocks+0x250401)
    #6 operator() /home/eagle/github/my/kvrocks/src/worker.cc:362 (kvrocks+0x222b73)
    #7 __invoke_impl<void, WorkerThread::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:60 (kvrocks+0x222b73)
    #8 __invoke<WorkerThread::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:95 (kvrocks+0x222b73)
    #9 _M_invoke<0> /usr/include/c++/9/thread:244 (kvrocks+0x222b73)
    #10 operator() /usr/include/c++/9/thread:251 (kvrocks+0x222b73)
    #11 _M_run /usr/include/c++/9/thread:195 (kvrocks+0x222b73)
    #12 <null> <null> (libstdc++.so.6+0xd6d83)

  Location is heap block of size 133 at 0x7b2400014eb0 allocated by thread T13:
    #0 operator new(unsigned long) <null> (libtsan.so.0+0x8c012)
    #1 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) <null> (libstdc++.so.6+0x142e7d)
    #2 Redis::CommandSPop::Execute(Server*, Redis::Connection*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /home/eagle/github/my/kvrocks/src/redis_cmd.cc:1822 (kvrocks+0x14ffb9)
    #3 Redis::Request::ExecuteCommands(Redis::Connection*) /home/eagle/github/my/kvrocks/src/redis_request.cc:211 (kvrocks+0x195d1d)
    #4 Redis::Connection::OnRead(bufferevent*, void*) /home/eagle/github/my/kvrocks/src/redis_connection.cc:64 (kvrocks+0x16a3f9)
    #5 bufferevent_run_deferred_callbacks_unlocked /home/eagle/github/my/kvrocks/external/libevent/bufferevent.c:208 (kvrocks+0x250401)
    #6 operator() /home/eagle/github/my/kvrocks/src/worker.cc:362 (kvrocks+0x222b73)
    #7 __invoke_impl<void, WorkerThread::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:60 (kvrocks+0x222b73)
    #8 __invoke<WorkerThread::Start()::<lambda()> > /usr/include/c++/9/bits/invoke.h:95 (kvrocks+0x222b73)
    #9 _M_invoke<0> /usr/include/c++/9/thread:244 (kvrocks+0x222b73)
    #10 operator() /usr/include/c++/9/thread:251 (kvrocks+0x222b73)
    #11 _M_run /usr/include/c++/9/thread:195 (kvrocks+0x222b73)
    #12 <null> <null> (libstdc++.so.6+0xd6d83)

  Thread T9 'worker' (tid=14127, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x5ea99)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xd7048)
    #2 Server::Start() /home/eagle/github/my/kvrocks/src/server.cc:60 (kvrocks+0x1e9427)
    #3 main /home/eagle/github/my/kvrocks/src/main.cc:317 (kvrocks+0x94839)

  Thread T13 'worker' (tid=14131, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x5ea99)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xd7048)
    #2 Server::Start() /home/eagle/github/my/kvrocks/src/server.cc:60 (kvrocks+0x1e9427)
    #3 main /home/eagle/github/my/kvrocks/src/main.cc:317 (kvrocks+0x94839)

SUMMARY: ThreadSanitizer: data race (/lib/x86_64-linux-gnu/libtsan.so.0+0x42333) in memcpy
==================

Steps to reproduce

add -fsanitize=thread to compiler flags
g++ -v

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)

startup command ./src/kvrocks -c kvrocks.conf
test command make test
benchmark command redis-benchmark -p 6666 -n 100000 -q
platform: Windows 10 + WSL2

Questions:

I am not C++ expert, can we use localtime_r?
The destructor of LockGuard is called at the end of if block, is it standard behavior across all C++ versions?

bug enhancement release notes need back patch

opened by calvinxiao 10

Fix LTRIM only delete first item, LREM cannot correctly remove repeated items

Fix LTRIM command only can delete the first index item, the metadata is changed, but items are not delete from RocksDB. Cases: rpush listkey 0 1 2 3 4 3 6 7 3 9 -> listkey: 0 1 2 3 4 3 6 7 3 9 ltrim listkey 2 -3 -> listkey: 2 3 4 3 6 7 lrem listkey 3 3 -> listkey should be: 2 4 6 7, but it is: 2 4 6

Fix LREM cannot remove item correctly from sequence with repeated items, the reason is that the number of deleted elements is not judged, which will cause the repeated items that need to be retained are not copied. Case: rpush listkey a b c d e f a f -> listkey: a b c d e f a f lrem listkey 1 f -> listkey should be: a b c d e a f, but it is: a b c d e a a

Other fixes: Fix wrong read/write attributes of some commands, such as lrem, linsert, flushdb
bug release notes need back patch

opened by ChrisZMF 9
Encode key with slot id

Encode key with slot id, so it will be efficient to migrate keys by slot when scale

Also support CLUSTER KEYSLOT command
release notes major decision

opened by shangxiaoxiong 9
build error with master
Hi,

I'm getting the following build error when trying to build from git.

[ 87%] Building CXX object CMakeFiles/kvrocks2redis.dir/tools/kvrocks2redis/parser.cc.o [ 88%] Linking CXX executable kvrocks2redis /usr/bin/ld: cannot find -lstdc++ collect2: error: ld returned 1 exit status make[2]: *** [kvrocks2redis] Error 1 make[1]: *** [CMakeFiles/kvrocks2redis.dir/all] Error 2 make: *** [all] Error 2

OS: Red Hat Enterprise Linux Server release 7.2 (Maipo)

To Reproduce Steps to reproduce the behavior:

sh build.sh build
opened by tkram01 9
Fix:SUBSCRIBE can't update last interaction time

Act like redis,SUBSCRIBE must update last interaction time while a publish message received, otherwise this channel may be closed for timeout reason by default.
bug release notes need back patch

opened by smartlee 9
Fix slave can't execute publish command

bug release notes need back patch

opened by karelrooted 8
what's the db action when run flushdb in some namespace?

null

opened by ZonghaoWang 1
Support executing INFO command during restoring database
TODO:

[ ] Must add test cases, since it may cause crash.

need tests
opened by ShooterIT 0
Does kvrocks support ROCKSDB's native KEY/VALUE separation, BLOB STORE?

The KVROCKS I use has 100 billion records ， VALUE IS 4KB BYTES ， Reading speed ( GET ) is very slow

opened by hangzhouminghe88 6
[QUESTION] Clarification on Redis version support

I love kvrocks but it would be useful if there was a version vs Redis version support. Whilst supported commands are listed, their Redis version isnt.

E.g. LPOP in Redis 6.2 supports a [count] parameter, allowing for multiple records to be returned whilst kvrocks returns just a single record.

I love would LPOP (etc) and all commands to be in line with Redis release but realise that this isn't going to be the case but it would be great if kvrocks published which Redis version each command is compatible with

opened by stopforumspam 1
[QUESTION] What's the recommended filesystem for kvrocks?

We are using kvrocks in production using ext4 filesystem and having around 4000 SST files. We were wondering if XFS would be better for performance. Did you do some specific tests with the different filesystems? Thank you very much.

opened by adulau 3
Upgrade the version of cpplint from 1.5.0 to 1.5.5.

Here, i've found a misleading cpplint errors generated by cpplint.sh in the progress of CI. And i've made sure that i've format the code by the Google code-style standard, with the necessary double-check by cpplint tool. However, the CI cpplint checking showed that there still exists some style errors in the code. So, i tried to check the difference of cpplint-1.5.0, defaultly used in the CI of kvrocks, between the version 1.5.5. And the difference showed as bellow:

Finally, i've checked the releasing logs of cpplint, and found the following results:

So, could we upgrade the version of cpplint in the CI of kvrocks to 1.5.5 ?

I‘ve verified the upgrade was valid in my local repository CI checking progress.

opened by LykxSassinator 0
possible to implement redisearch?

do you have the implementation guideline on how to implement redisearch? written like this? https://kvrocks.medium.com/design-complex-structure-on-rocksdb-993a00fd4f6d?source=post_internal_links---------0----------------------------

opened by hiqsociety 1
any benchmark against redis?

any benchmark against redis?

opened by hiqsociety 12
[NEW] Allow forcing use the full sync when the lag was too large
Background

Currently, we're using the WAL to do the partial sync like the Redis backlog, as well as the write buffer between worker threads and feed slave thread. Users should tune the WAL size with the key-value size and it's hard in most cases, since the user key-value size may be varied over time. We prefer to give it a generic larger value to the WAL to fit into most cases, but every coin has two sides, it may slow down the partial sync when the key-value size was small and failed to trigger the full sync.

Proposal

The core problem was the generic WAL size may cause the partial sync too slowly when the key-value size was small, and it's hard to adjust the WAL size instance by instance since the size may be changed over time. So we need to find another way to work around this case, and the lag threshold should be a good choice since increment logs were synced by sequences.

TODO

[ ] Add the force-full-sync-lag-threshold configuration to specify and enable the full sync lag threshold, default is 0 which means disabled

[ ] Check the next seq, if it's a new slave and overrun the full sync lag threshold, let's use the full sync mechanism instead of the partial sync

[ ] Add test cases

cool idea from @Alfejik
enhancement new feature
opened by git-hulk 2
Support Watch/Unwatch

I would like to use Watch and Unwatch for full transaction support.

yinqiwen/ardb has a working watch/unwatch impl in case that is useful as a reference.

https://github.com/yinqiwen/ardb
new feature

opened by snowbldr 1

Releases(v2.0.3)

v2.0.3(Sep 13, 2021)
If your kvrocks instances are 2.0.2 and use redis-sentinel to implement HA, you should upgrade urgently, please see #352 for details.

Here is a comprehensive list of changes in this release compared to 2.0.2

Improvements

Use the rocksdb MultiGet interface in the MGET command (#331)

Avoid useless space allocation in SetBit (#338)

Auto set master/replica replication by cluster topology (#356)

Support client kill type subcommands to adapt redis-sentinel (#352) (#361) From 2.0.2, we support MULTI-EXEC, but don't support client kill type $type, so MULTI-EXEC will be rollbacked when failing to parse unknown kill type in client command, and cause redis-sentinel fails to failover.

Bugfixes

Fix SETNX and MSETNX commands can't guarantee atomicity (#337)

Fix the replicas won't reconnect when failing to connect with master (#344)

Fix master pauses synchronizing with replicas (#347)

Fix wrong mater port if enabling master-use-repl-port (#360)

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.3-release.x86_64.rpm(126.69 MB)
kvrocks_2.0.3-release_amd64.deb(115.68 MB)
v2.0.2(Jul 19, 2021)
We design new cluster solution for kvrocks, but it is experimental, don't recommend to deploy in production environments. And because in cluster mode, to be efficient to migrate keys based on slot when scale, we encode key with slot id, but in order to be compatible with the old version, we don't do that in standalone mode, so data is not compatible between standalone mode with cluster mode, you must migrate data if you want to change mode, otherwise, kvrocks will make data corrupt. In order to better support the cluster mode, we are developing a meta server to manage kvrocks cluster, coming soon!

Here is a comprehensive list of changes in this release compared to 2.0.1

New features

Support rename command in config file (#272)

Support transaction (#285) Support MULTI, EXEC, DISCARD command, but don't support WATCH and UNWATCH commands.

Support the auto-resize-block-and-sst config directive (#289)

Support to resize rocksdb.block_size automatically (#294)

Support cluster mode (#302) Similar with redis cluster mode, but use CLUSTERX command to set cluster topology (#324).

Data encoding changes

Encode key with slot id (#291) (#330) In cluster mode, to be efficient to migrate keys by slot when scale, we encode key with slot id, but in standalone mode, we don't. That is to say, data is not compatible between standalone mode with cluster mode, you must migrate data if you want to change mode, otherwise, kvrocks will make data corrupt.

Bug fixes

Set TCP_KEEPALIVE for replication connection (#284) Replicas can handle network failure with master, before this commit, network failure may result in unrecoverable replication interrupt.

Fix ZSET can't add the same member with different scores (#298)

Make CONFIG SET command thread safe (#310)

Fix LTRIM only delete first item, LREM can't correctly remove repeated items (#314)

Improvements

HSET command supports to set multiple fields (#274)

Allow to use host in the master-slave replication (#301)

Adapt redis sentinel ping check (#316)

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.2-release.x86_64.rpm(126.28 MB)
kvrocks_2.0.2-release_amd64.deb(115.27 MB)
v2.0.1(May 18, 2021)
Here is a comprehensive list of changes in this release compared to 2.0.0

New features

Support COMMAND command (#251)

Support to purge backups on fullsync to reduce disk space usage (#245)

Bug fixes

Fix uninitialized Rocksdb.level0_stop_writes_trigger (#236)

Fix condition race for ReclaimOldDBPtr (#246)

Fix data race for accessing database in some commands (#253) (#261) In spop, zremrangebylex, zremrangebyscore commands, the lock guard for accessing the database may not work actually.

Fix returning wrong string range in GETRANGE command (#254)

Fix skipping wrong CURRENT file when full synchronization (#260) Resuming broken transfer based files may be not available if replicas skip transferred wrong CURRENT file of RocksDB.

Improvements

Replicas can execute publish command (#238)

Optimizations for avoiding unnecessary data copy (#239) (#247)

Allow to run the kvrocks without assigning the config file (#264)

Dependencies

Upgrade the jemalloc version to the special commit (#265) To be able to compile on macOS, upgrade jemalloc to the production test commit which mentioned in jemalloc/jemalloc#2060.

Source code(tar.gz)
Source code(zip)
kvrocks-2.0.1-release.x86_64.rpm(125.30 MB)
kvrocks_2.0.1-release_amd64.deb(114.31 MB)
v1.3.3(May 18, 2021)
Here is a comprehensive list of changes in this release compared to 1.3.2

Bug fixes

Fix rocksdb can't auto resume after no space error (#229)

Fix uninitialized Rocksdb.level0_stop_writes_trigger (#236)

Fix condition race for ReclaimOldDBPtr (#246)

Fix returning wrong string range in GETRANGE command (#254)

Fix data race for accessing database in some commands (#253) In spop, zremrangebylex, zremrangebyscore commands, the lock guard for accessing the database may not work actually.

Improvements

Replicas can execute publish command (#238)

Source code(tar.gz)
Source code(zip)
v2.0.0(Apr 23, 2021)
Kvrocks 2.0 brings some new useful features that solve actual problems in the production environment. Please note that this version instance still can replicate from old version master but old version replicas can't replicate new version master. Otherwise, we start to use new logo, thanks @tianlingyu1997

Here is a comprehensive list of changes in this release compared to 1.3.2

New features

Don't use extra port for replication (#200)

Use checkpoint to implement data snapshot when full replication (#208) Reduce disk space and bandwidth usage and decrease the time of replicas recovery during full synchronization.

Use checkpoint to implement kvrocks backup (#232)

Replicas can empty db before fullsync (#233) Replicas use less disks space if enabled, but have a risk to lose data.

Command changes

Add QPS and input/output kbps for INFO command (#213)

Add pubsub_patterns for INFO command (#234)

Slowlog omits args or argument string if too many or big (#215)

Cluster

Remove codis support (#217) From now, we doesn't support codis, you can adopt pre-sharding to implement cluster, and we are developing new cluster solution that is similar with redis cluster mode, please see #219.

Bugfixes

Fix kvrocks can't auto resume after no space error (#229)

Dependencies

Upgrade rocksdb to latest tag v6.19.3 (#226)

Source code(tar.gz)
Source code(zip)
v1.3.2(Apr 20, 2021)
Here is a comprehensive list of changes in this release compared to 1.3.1

Bug fixes

Fix incorrect used_db_size in INFO command (#204)

Fix the SST file creation time maybe 0 in some unknown conditions (#211)

Fix get corruption writebatch data after psync restart (#221)

Improvements

Optimize TCP keepalive detection time (#201)

Source code(tar.gz)
Source code(zip)
v1.3.0(Feb 8, 2021)
Here is a comprehensive list of changes in this release compared to 1.2.0

License

Change the license from MIT to BSD-3-Clause

New features

Support to set separate log directory to store server logs (#155)

Implement PSETEX command (#156)

Implement MSETNX command (#165)

Improvements

Improve the precision of the INCRBYFLOAT/HINCRBYFLOAT command to double (#169)

Support +INF/-INF for the score of zset type members (#172)

The UNSUBSCRIBE and PUNSUBSCRIBE commands support multiple channels or patterns (#174)

Behavior changes

The UNSUBSCRIBE and PUNSUBSCRIBE commands have the same reply as Redis (#174)

The same message format as Redis for pattern subscribers when publishing message (#175) Message format is from 'message channel message' to 'pmessage pattern channel message'

Bug fixes

Fix skipping some SST files when picking compaction files (#154)

Handle negative expire time in set command to avoid deleting keys (#156)

List type command doesn't rewrite other types of data (#157)

Only set positive expire time when migrating key (#160)

Don't store an empty key for setrange command (#161)

Fix SETBIT/GETBIT bit offset out of range (#164)

Fix INCRBY/INCRBYFLOAT command operates on wrong value and precision check (#167)

Fix HINCRBY/HINCRBYFLOAT command operates on wrong value and precision check (#168)

RPOPLPUSH command doesn't operate non list type (#170)

LINSERT command returns -1 if don't find index (#170)

Fix LTRIM command range calculation when start is a very large negative number (#170)

SMOVE command doesn't operate non set type key (#171)

Fix ZREMRANGEBYRANK command corrupts the size of zset (#172)

SUBSCRIBE and PSUBSCRIBE commands return the sum of subscription and psubscription (#174)

Fix PUNSUBSCRIBE command doesn't take effort (#174)

Allow setting slowlog-log-slower-than to -1 to disable the slowlog (#178)

Source code(tar.gz)
Source code(zip)
v1.2.0(Jan 8, 2021)

Source code(tar.gz)
Source code(zip)
kvrocks-1.2.0-167827.release.x86_64.rpm(19.98 MB)
v1.1.32(Oct 28, 2020)

Source code(tar.gz)
Source code(zip)
kvrocks-1.1.32-142122.release.x86_64.rpm(19.96 MB)
v1.1.27(Aug 26, 2020)
FIX: bitcount return wrong number (cause by typo)

Source code(tar.gz)
Source code(zip)
kvrocks-1.1.27-126340.release.x86_64.rpm(19.90 MB)
v1.1.6(Feb 28, 2020)
Upgrade the libevent to stable 2.1.11

FIX: create the backup at the same time may cause a crash while the CreateNewBackupWithMetadata wasn't thread-safe call

FIX: some incorrect implements in redis command

Source code(tar.gz)
Source code(zip)
v.1.1.2(Dec 6, 2019)

Source code(tar.gz)
Source code(zip)
v1.1.0(Nov 14, 2019)

Source code(tar.gz)
Source code(zip)
v1.0.3(Sep 18, 2019)

Source code(tar.gz)
Source code(zip)
v1.0.1(Aug 23, 2019)
ADD: supports flushall/object command

FIX: incorrect command name, flushall => flushdb

FIX: hscan return not found when key not exists

Source code(tar.gz)
Source code(zip)

Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Related tags

Overview

Who uses kvrocks

Building kvrocks

requirements

Build

Running kvrocks

Running test cases

Supported platforms

Try kvrocks using Docker

Namespace

DOCs

Migrate tools

Performance

Hardware

1. Commands QPS

2. QPS on different payloads

3. QPS on different workers

License

Issues

Describe this PR

1. Data race when starting up the server

2. Thread Sanitizer warning in make test

3. Thread Sanitizer warning when benchmarking

Steps to reproduce

Questions:

Background

Proposal

TODO

Releases(v2.0.3)

v2.0.3(Sep 13, 2021)

Improvements

Bugfixes

v2.0.2(Jul 19, 2021)

New features

Data encoding changes

Bug fixes

Improvements

v2.0.1(May 18, 2021)

New features

Bug fixes

Improvements

Dependencies

v1.3.3(May 18, 2021)

Bug fixes

Improvements

v2.0.0(Apr 23, 2021)

New features

Command changes

Cluster

Bugfixes

Dependencies

v1.3.2(Apr 20, 2021)

Bug fixes

Improvements

v1.3.0(Feb 8, 2021)

License

New features

Improvements

Behavior changes

Bug fixes

v1.2.0(Jan 8, 2021)

v1.1.32(Oct 28, 2020)

v1.1.27(Aug 26, 2020)

v1.1.6(Feb 28, 2020)

v.1.1.2(Dec 6, 2019)

v1.1.0(Nov 14, 2019)

v1.0.3(Sep 18, 2019)

v1.0.1(Aug 23, 2019)

Owner

Kvrocks Labs

RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis.

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB

Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

The MongoDB Database

ClickHouse® is a free analytics DBMS for big data

FEDB is a NewSQL database optimised for realtime inference and decisioning application

2. Thread Sanitizer warning in `make test`