Bug Report

KVDK version

14f8778dea099225cd2bb1d9c8b406c7b53b8133

System configuration

Reproduce steps

    kvdk::Configs engine_configs;
    {

      engine_configs.pmem_file_size = 40 * 1024UL * 1024UL * 1024UL;
      engine_configs.pmem_segment_blocks = (1ull << 8);
      engine_configs.hash_bucket_num = (1ull << 10);
      engine_configs.log_level = kvdk::LogLevel::Debug;
    }

    std::string engine_path{kvpmem_data_file_path};

    // Purge old KVDK instance
    system(std::string{"rm -rf " + engine_path + "\n"}.c_str());

    status = kvdk::Engine::Open(engine_path, &engine, engine_configs, stdout);
    assert(status == kvdk::Status::Ok);
    DBUG_PRINT("KVDK",
               ("Successfully created KVDK engine %s", kvpmem_data_file_path));

    status = engine->SSet("hi", "yo", "peeep");
    ASSERT(status == kvdk::Status::Ok, (int)status);
  }

Expect behavior

Should put value in the ordered set.

Current behavior

Assert fails with InvalidDataSize. If I print the size of the string view inside the engine I get following output [ERROR] time 40009 ms: SSET size collection: 1147430597 userkey: 1147430594 value: 1147430588.

I think there is an issue with the string view but I don't understand why since the usage is very similar to the usage guide. I also tried to pass a static string. Any ideas?

Bug Report

Why not use libpmemobj ? I do not think kvdk can support atomic operation only using APIs in libpmem. batchWrite can write multiple KV pairs one by one, but could not roll back if failed. DLinked_list could not update prev/next pointer in one transaction. But it shows "Provide APIs to write multiple key-value pairs in an atomic batch. ", Is there something wrong ?

Why is the reading and writing speed of kvdk not as fast as leveldb? I think this is very strange. Is it the problem of my use or the problem of kvdb itself? If it is the problem of kvdb itself, do you have a good recommendation for kv database, it is best to use B+ tree as the engine, thank you.

In this patch:

1. Move pmem allocator related codes to directory engine/pmem_allocator
2. Reorg free list:

• Use a global free space entry pool to store free space
• Each write thread has a cache structure, which consists of a active free space list and a backup free space list. Each threads store just freed space to and use free space from own active list. To balance free space entries among threads, if too many entries cached by a thread, newly freed entries will be stored in backup list, then move the whole backup list to entry pool which shared by all threads
• If active list is empty, then swap backup list as active list, or fetch a free list from entry pool

What is changed and how it works?

Merge space in background only if there are enough space entry freed since last merge

Check List

Tests

• Unit test
• Integration test

Bug Report

I cannot compile the kvrocks correctly by following the steps mentioned in the example/kvrocks/README I found that the README is updated 9 months ago. However, the patch is updated three days ago.

Current behavior

When I compile with running ‘make’, there's an error:

incubator-kvrocks/external/kvdk/include/types.hpp:7:10: fatal error: libpmemobj++/string_view.hpp: No such file or directory

This error could be fixed by copying libpmemobj++/ directory from kvdk to the kvrocks.

But there is another error:

In file included from redis_bitmap_string.cc:5:
redis_string.h:9:10: fatal error: namespace.hpp: No such file or directory
#include "namespace.hpp"

I found that you are developing kvdk and there is no namespace.h file in the latest version. I would be grateful if you fixing the issue and update the README.

The empty string as key acts normally in KVDK. No need to check for zero-sized keys. Added a file debug.cpp under tests directory and is compiled to dbdebug. This file is almost a copy of tutorial.cpp but for debugging purpose, because it's very simple and easy to change.

"Set error" happened when using sorted-type and latency enable, option type inclues "Fill, update, Insert"，for example：

Insert new sorted-type kv Write latency overflow: 12191390 us Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error sh: line 1: 71030 Segmentation fault (core dumped) numactl --cpunodebind=0 --membind=0 ./bench -latency=1 -populate=1 -value_size=128 -threads=96 -time=10 -path=/mnt/pmem0/kvdk -num=789516047 -space=536870912000 -max_write_threads=96 -fill=0 -type=sorted -read_ratio =0 -existing_keys_ratio=0 > ./results/sorted_vs128_insert_thread96

Signed-off-by: Yu, Peng [email protected]

What problem does this PR solve?

Problem Summary: Add submodule jemalloc and numactl for volatile memory management

What is changed and how it works?

What's Changed:

• Add submodule jemalloc and numactl under extern directory
• Update cmake configurations to build above submodules
• Small improvement to Java module

Check List

Tests

• Unit test

Side effects

• None

What problem does this PR solve?

Problem Summary: compare_string_view() function works inefficiently by comparing characters one by one .

What is changed and how it works?

When the Key is more than 8B, compare two uint64_t instead of 8 bytes.

What's Changed: static inline int compare_string_view(const StringView& src, const StringView& target) { auto size = std::min(src.size(), target.size()); auto batchNum = size & ~(sizeof(uint64_t)-1);
uint32_t i = 0; for( i =0; i< batchNum ; ){ // if size is more than 8, compared by type of uint64_t. auto nEq = (uint64_t)(&src[i]) - (uint64_t)(&target[i]); if ( nEq ) return nEq; i += sizeof(uint64_t); } for (; i < size; i++) { if (src[i] != target[i]) { return src[i] - target[i];
} } return src.size() - target.size(); }

Tests

• No code

Side effects none

What problem does this PR solve?

Problem Summary: Introducing optional volatile KV storage on DRAM

What is changed and how it works?

What's Changed:

• add compile option WITH_PMEM, which is ON by default
• add enable_pmem in KVDK::Configs, which is true by default
• cmake with -DWITH_PMEM=OFF to remove dependency to libpmem
• even built with -DWITH_PMEM=ON, setting Configs.enable_pmem = false can make KVDK store data on DRAM

Check List

Tests

• Unit test
• Integration test

Side effects

• None

Signed-off-by: Yu, Peng [email protected]

What problem does this PR solve?

Problem Summary: Adapt benchmarks for volatile KV storage

What is changed and how it works?

What's Changed: volatile/benchmark/bench.cpp

Check List

Tests

• Unit test
• Integration test

Side effects

• None

What is changed and how it works?

Implement new hashmap with rehash function. Reduce memory usage. Add VHash(Volatile Hash) data type.

TODO list

Add automatic rehashing logic.

Check List

Tests

• Unit test

The problem/use-case that the feature addresses

If one PM disk space is not enough, how can multiple PM disks be mapped into one KVDK?

Description of the feature

KVDK can take use of multiple PM disk.

Alternatives you've considered

Additional information

Any additional information that is relevant to the feature request.