In Memory Property Graph Server using the Shared Nothing design from Seastar.

Overview

RageDB

In Memory Property Graph Server using the Shared Nothing design from Seastar.

The RageDB server can host multiple Graphs. The graphs are accessible via a REST API (see below). Each Graph is split into multiple Shards. One Shard per Core of the server. Shards communicate by explicit message passing. Nodes and Relationships have internal and external ids. The external ids embed the type as well as which Shard they belong to. The internal ids are pointers into vectors that hold the data of each Node and Relationship. The Relationship ids are replicated to both incoming and outgoing Nodes. The Relationship Object (and properties) belong to the Outgoing Node (and shard). Each Node must have a singular Type and unique Key on creation which the server stores in a map for retrieval. External and Internal Ids are assigned upon creation for both Nodes and Relationships.

Along side an HTTP API, RageDB also has a Lua http endpoint that allows users to send complex queries. These queries are interpreted by LuaJIT, compiled and executed within a Seastar Thread that allows blocking. By not having a "query language" we avoid parsing, query planning, query execution and a host of problems.

HTTP API

Schema

Get Node Types

:GET /db/{graph}/schema/nodes

Get a Node Type

:GET /db/{graph}/schema/nodes/{type}

Create a Node Type

:POST /db/{graph}/schema/nodes/{type}

Delete a Node Type

:POST /db/{graph}/schema/nodes/{type}

Get Relationship Types

:GET /db/{graph}/schema/relationships

Get a Relationship Type

:GET /db/{graph}/schema/relationships/{type}

Create a Relationship Type

:POST /db/{graph}/schema/relationships/{type}

Delete a Relationship Type

:DELETE /db/{graph}/schema/relationships/{type}

RageDB currently supports booleans, 64-bit integers, 64-bit doubles, strings and lists of the preceding data types:

boolean, integer, double, string, boolean_list, integer_list, double_list, string_list

Get a Node Property Type

:GET /db/{graph}/schema/nodes/{type}/properties/{property}

Create a Node Property Type

:POST /db/{graph}/schema/nodes/{type}/properties/{property}/{data_type}

Delete a Node Property Type

:DELETE /db/{graph}/schema/nodes/{type}/properties/{property}

Get a Relationship Property Type

:GET /db/{graph}/schema/relationships/{type}/properties/{property}

Create a Relationship Property Type

:POST /db/{graph}/schema/relationships/{type}/properties/{property}/{data_type}

Delete a Relationship Property Type

:DELETE /db/{graph}/schema/relationships/{type}/properties/{property}

Nodes

Get All Nodes

:GET /db/{graph}/nodes?limit=100&offset=0

Get All Nodes of a Type

:GET /db/{graph}/nodes/{type}?limit=100&offset=0

Get A Node By Type and Key

:GET /db/{graph}/node/{type}/{key}

Get A Node By Id

:GET /db/{graph}/node/{id}

Create A Node

:POST /db/{graph}/node/{type}/{key}
JSON formatted Body: {properties}

Delete A Node By Type and Key

:DELETE /db/{graph}/node/{type}/{key}

Delete A Node By Id

:DELETE /db/{graph}/node/{id}

Node Properties

Get the Properties of a Node By Type and Key

:GET /db/{graph}/node/{type}/{key}/properties

Get the Properties of a Node By Id

:GET /db/{graph}/node/{id}/properties

Reset the Properties of a Node By Type and Key

:POST /db/{graph}/node/{type}/{key}/properties
JSON formatted Body: {properties}

Reset the Properties of a Node By Id

:POST /db/{graph}/node/{id}/properties
JSON formatted Body: {properties}

Set some Properties of a Node By Type and Key

:PUT /db/{graph}/node/{type}/{key}/properties
JSON formatted Body: {properties}

Set some Properties of a Node By Id

:PUT /db/{graph}/node/{id}/properties
JSON formatted Body: {properties}

Delete the Properties of a Node By Type and Key

:DELETE /db/{graph}/node/{type}/{key}/properties

Delete the Properties of a Node By Id

:DELETE /db/{graph}/node/{id}/properties

Get a Property of a Node By Type and Key

:GET /db/{graph}/node/{type}/{key}/property/{property}

Get a Property of a Node By Id

:GET /db/{graph}/node/{id}/property/{property}

Create a Property of a Node By Type and Key

:PUT /db/{graph}/node/{type}/{key}/property/{property}
JSON formatted Body: {property}

Create a Property of a Node By Id

:PUT /db/{graph}/node/{id}/property/{property}
JSON formatted Body: {property}

Delete a Property of a Node By Type and Key

:DELETE /db/{graph}/node/{type}/{key}/property/{property}

Delete a Property of a Node By Id

:DELETE /db/{graph}/node/{id}/property/{property}

Relationships

Get A Relationship

:GET /db/{graph}/relationship/{id}

Create A Relationship By Node Types

:POST /db/{graph}/node/{type_1}/{key_1}/relationship/{type_2}/{key_2}/{rel_type}
JSON formatted Body: {properties}

Create A Relationship By Node Ids

:POST /db/{graph}/node/{id_1}/relationship/{id_2}/{rel_type}
JSON formatted Body: {properties}

Delete A Relationship

:DELETE /db/{graph}/relationship/{id}

Get the Relationships of a Node By Node Type

:GET /db/{graph}/node/{type}/{key}/relationships
:GET /db/{graph}/node/{type}/{key}/relationships/{direction [all, in, out]} 
:GET /db/{graph}/node/{type}/{key}/relationships/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{type}/{key}/relationships/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Get the Relationships of a Node By Node Id

:GET /db/{graph}/node/{id}/relationships
:GET /db/{graph}/node/{id}/relationships/{direction [all, in, out]} 
:GET /db/{graph}/node/{id}/relationships/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{id}/relationships/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Relationship Properties

Get the Properties of a Relationship

:GET /db/{graph}/relationship/{id}/properties

Reset the Properties of a Relationship

:POST /db/{graph}/relationship/{id}/properties
JSON formatted Body: {properties}

Set some Properties of a Relationship

:PUT /db/{graph}/relationship/{id}/properties
JSON formatted Body: {properties}

Delete the Properties of a Relationship

:DELETE /db/{graph}/relationship/{id}/properties

Get a Property of a Relationship

:GET /db/{graph}/relationship/{id}/property/{property}

Create a Property of a Relationship

:PUT /db/{graph}/relationship/{id}/property/{property}
JSON formatted Body: {property}

Delete a Property of a Relationship

:DELETE /db/{graph}/relationship/{id}/property/{property}

Node Degrees

Get the Degree of a Node By Node Type

:GET /db/{graph}/node/{type}/{key}/degree
:GET /db/{graph}/node/{type}/{key}/degree/{direction [all, in, out]} 
:GET /db/{graph}/node/{type}/{key}/degree/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{type}/{key}/degree/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Get the Degree of a Node By Node Id

:GET /db/{graph}/node/{id}/degree
:GET /db/{graph}/node/{id}/degree/{direction [all, in, out]} 
:GET /db/{graph}/node/{id}/degree/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{id}/degree/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Node Neighbors

Get the Neighbors of a Node By Node Type

:GET /db/{graph}/node/{type}/{key}/neighbors
:GET /db/{graph}/node/{type}/{key}/neighbors/{direction [all, in, out]} 
:GET /db/{graph}/node/{type}/{key}/neighbors/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{type}/{key}/neighbors/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Get the Neighbors of a Node By Node Id

:GET /db/{graph}/node/{id}/neighbors
:GET /db/{graph}/node/{id}/neighbors/{direction [all, in, out]} 
:GET /db/{graph}/node/{id}/neighbors/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{id}/neighbors/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Lua

:POST db/{graph}/lua
STRING formatted Body: {script}

The script must end in one or more values that will be returned in JSON format inside an Array. Within the script the user can access to graph functions. For example:

-- Get some things about a node
a = NodeGetId("Node","Max")
b = NodeTypesGetCount()
c = NodeTypesGetCountByType("Node")
d = NodePropertyGet("Node", "Max", "name")
e = NodePropertyGetById(a, "name")
a, b, c, d, e

A second example:

-- get the names of nodes I have relationships with
names = {}
ids = NodeGetRelationshipsIds("Node", "Max")
for k=1,#ids do
    v = ids[k]
    table.insert(names, NodePropertyGetById(v.node_id, "name"))
end
names

Building

RageDB uses Seastar which only runs on *nix servers (no windows or mac) so use your local linux desktop or use EC2.

On EC2 launch an instance:

Step 1: Choose an Amazon Machine Image
Ubuntu Server 20.04 LTS(HVM), SSD Volume Type - ami-09e67e426f25ce0d7

Step 2: Choose Instance Type
r5.2xlarge

Step 3: Configure Instance
Specify CPU options
Threads per core = 1

Step 4: Add Storage
100 GiB

Launch

Once the instance is running, connect to it and start a "screen" session, then follow these steps:

First let's update and upgrade to the latest versions of local software:

sudo apt-get update && sudo apt-get dist-upgrade

Install Seastar (this will take a while, that's why we are using screen):

git clone https://github.com/scylladb/seastar.git
cd seastar
sudo ./install_dependencies.sh
./configure.py --mode=release --prefix=/usr/local
sudo ninja -C build/release install

Install Additional Dependencies

sudo apt-get install -y ccache python3-pip

Install conan

pip install --user conan
sudo ln -s ~/.local/bin/conan /usr/bin/conan

Install LuaJIT

sudo apt-get install -y luajit luajit-5.1-dev

Troubleshooting

If you get errors regarding conan locks, run:

conan remove --locks

Missing Features that can be added "easily"

- Allow Node and Relationship Type handlers to take a json map defining the property keys and data types
- Allow additional data types: 8, 16 and 32 bit integers, 32 bit floats, byte, list of bytes, nested types 
- NodeTypes and RelationshipTypes should allow type deletion and type id reuse
- Allow property type conversions (int to double, string to int, int to int array, etc).

PVS Commands

pvs-studio-analyzer trace -- make
pvs-studio-analyzer analyze
plog-converter -a GA:1,2 -t tasklist  PVS-Studio.log
Issues
  • Deal with date parsing slowdown

    Deal with date parsing slowdown

    date(message.creationDate):todouble() takes a long time. It doubles import time for LDBC 10. Handle it in the c++ side by adding real datetime type or move date parsing to c++

    small enhancement 
    opened by maxdemarzi 2
  • Add Shell Interface

    Add Shell Interface

    Once version 2.0 of https://github.com/daniele77/cli gets pushed on conan, consider adding a shell interface much like the Crash shell was added on the lower half of https://maxdemarzi.com/2017/01/18/our-own-multi-model-database-part-5/

    large enhancement 
    opened by maxdemarzi 1
  • Use-after-free

    Use-after-free

    NOTE: not even compile tested. btw, if you don't store a ref to shared ptr in lambda, p2 may be destroyed after when_all_succeed() defers. note that seastar projects like scylladb use do_with() to handle this, or just store a ref to shared ptr in lambda like done by this fix.

    opened by raphaelsc 1
  • Create a read only sandbox

    Create a read only sandbox

    Create a sandbox that only had read only graph methods. We can try using it without taking a lock (maybe). Gives us a read only environment to deploy for testing or demos.

    opened by maxdemarzi 0
  • Test NodesGet

    Test NodesGet

    I don't think this works due to overloaded vectors see https://github.com/ThePhD/sol2/issues/1357 , break out if we have to:

        lua.set_function("NodesGet", sol::overload(
              [this](std::vector<uint64_t> ids) { return this->NodesGetViaLua(ids); },
              [this](std::vector<Link> links) { return this->NodesGetByLinksViaLua(links); }
             ));
          lua.set_function("NodesGetKey", sol::overload(
              [this](std::vector<uint64_t> ids) { return this->NodesGetKeyViaLua(ids); },
              [this](std::vector<Link> links) { return this->NodesGetKeyByLinksViaLua(links); }
             ));
          lua.set_function("NodesGetType", sol::overload(
              [this](std::vector<uint64_t> ids) { return this->NodesGetTypeViaLua(ids); },
              [this](std::vector<Link> links) { return this->NodesGetTypeByLinksViaLua(links); }
             ));
          lua.set_function("NodesGetProperty", sol::overload(
              [this](std::vector<uint64_t> ids, const std::string& property) { return this->NodesGetPropertyViaLua(ids, property); },
              [this](std::vector<Link> links, const std::string& property) { return this->NodesGetPropertyByLinksViaLua(links, property); }
             ));
          lua.set_function("NodesGetProperties", sol::overload(
              [this](std::vector<uint64_t> ids) { return this->NodesGetPropertiesViaLua(ids); },
              [this](std::vector<Link> links) { return this->NodesGetPropertiesByLinksViaLua(links); }
             ));
    
    opened by maxdemarzi 0
Owner
Rage DB
Rage DB
MillenniumDB is a graph oriented database management system

Millennium DB MillenniumDB is a graph oriented database management system developed by the Millennium Institute for Foundational Research on Data (IMF

null 21 Jun 20, 2022
DB Browser for SQLite (DB4S) is a high quality, visual, open source tool to create, design, and edit database files compatible with SQLite.

DB Browser for SQLite What it is DB Browser for SQLite (DB4S) is a high quality, visual, open source tool to create, design, and edit database files c

null 16.7k Jun 27, 2022
prometheus exporter using workflow HTTP server

wf-prometheus This is a light prometheus exporter using workflow HTTP server. This project is currently in the development stage, and the first versio

C++ Workflow Project and Ecosystem 9 Oct 23, 2021
C++ embedded memory database

ShadowDB 一个C++嵌入式内存数据库 语法极简风 支持自定义索引、复合条件查询('<','<=','==','>=','>','!=',&&,||) 能够快速fork出一份数据副本 // ShadowDB简单示例 // ShadowDB是一个可以创建索引、能够快速fork出一份数据分支的C+

null 8 Jun 21, 2022
MySQL Server, the world's most popular open source database, and MySQL Cluster, a real-time, open source transactional database.

Copyright (c) 2000, 2021, Oracle and/or its affiliates. This is a release of MySQL, an SQL database server. License information can be found in the

MySQL 7.9k Jun 24, 2022
Beryl-cli is a client for the BerylDB database server

Beryl-cli is a client for the BerylDB database server. It offers multiple commands and is designed to be fast and user-friendly.

BerylDB 11 Apr 21, 2022
A proxy server for OceanBase Database.

OceanBase Database Proxy TODO: some badges here OceanBase Database Proxy (ODP for short) is a dedicated proxy server for OceanBase Database. OceanBase

OceanBase 76 Apr 6, 2022
A PostgreSQL extension providing an async networking interface accessible via SQL using a background worker and curl.

pg_net is a PostgreSQL extension exposing a SQL interface for async networking with a focus on scalability and UX.

Supabase 41 Jun 19, 2022
A framework to monitor and improve the performance of PostgreSQL using Machine Learning methods.

pg_plan_inspector pg_plan_inspector is being developed as a framework to monitor and improve the performance of PostgreSQL using Machine Learning meth

suzuki hironobu 166 Jun 20, 2022
Modern cryptography for PostgreSQL using libsodium.

pgsodium pgsodium is an encryption library extension for PostgreSQL using the libsodium library for high level cryptographic algorithms. pgsodium can

Michel Pelletier 252 May 17, 2022
✔️The smallest header-only GUI library(4 KLOC) for all platforms

Welcome to GUI-lite The smallest header-only GUI library (4 KLOC) for all platforms. 中文 Lightweight ✂️ Small: 4,000+ lines of C++ code, zero dependenc

null 6.3k Jun 27, 2022
Shared-Memory Parallel Graph Partitioning for Large K

KaMinPar The graph partitioning software KaMinPar -- Karlsruhe Minimal Graph Partitioning. KaMinPar is a shared-memory parallel tool to heuristically

Karlsruhe High Quality Graph Partitioning 13 Jun 14, 2022
C++ library to create dynamic structures in plain memory of shared-memory segments

Ищи описание на хабре @mrlolthe1st. #define _CRT_SECURE_NO_WARNINGS #include "shared_structures.h" #include <iostream> #include <fstream> #include <ca

Александр Новожилов 4 Mar 30, 2022
Arduino library for sending email and SMS from nothing but the ESP8266!

Did you know your ESP8266 could send Email and SMS without any special hardware or paid services like Twilio? With AlertMe, your ESP8266 project can:

Lixie Labs 61 Feb 24, 2022
Fast C++ IPC using shared memory

Fast C++ IPC using shared memory

Dheeraj R Reddy 320 Jun 25, 2022
Using shared memory to communicate between two executables or processes, for Windows, Linux and MacOS (posix). Can also be useful for remote visualization/debugging.

shared-memory-example Using shared memory to communicate between two executables or processes, for Windows, Linux and MacOS (posix). Can also be usefu

null 8 Mar 18, 2022
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large-scale graphs with billions of vertices (nodes) and trillions of edges, with milliseconds of latency. It delivers enterprise-grade high performance to simplify the most complex data sets imaginable into meaningful and useful information.

vesoft inc. 7.6k Jun 24, 2022
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large scale graphs with dozens of billions of vertices (nodes) and trillions of edges, with milliseconds of latency.

vesoft inc. 803 Jun 18, 2022
Memgraph is a streaming graph application platform that helps you wrangle your streaming data, build sophisticated models that you can query in real-time, and develop graph applications.

Memgraph is a streaming graph application platform that helps you wrangle your streaming data, build sophisticated models that you can query in real-time, and develop graph applications.

Memgraph 406 Jun 22, 2022
By putting in a lot of speed, the speed sequence is sorted and divided, three types of speed interval distribution maps are generated.(including broken line graph,histogram and curve graph)

Auto-drawing-speed-range-map By putting in a lot of speed, the speed sequence is sorted and divided, three types of speed interval distribution maps a

wellwellAllwen 4 May 14, 2022
LLVM meets Code Property Graphs

llvm2cpg llvm2cpg is a tool that converts LLVM Bitcode into Code Property Graph (CPG). The CPG can be further analyzed via Joern or Ocular. To get sta

ShiftLeft Inc. 56 Jun 10, 2022
Header-only C++11 library for property-based testing.

autocheck Header-only C++11 library for QuickCheck (and later, SmallCheck) testing. Please consult the wiki for documentation. Install conan remote ad

John Freeman 119 Apr 18, 2022
Serialization framework for Unreal Engine Property System that just works!

DataConfig Serialization framework for Unreal Engine Property System that just works! Unreal Engine features a powerful Property System which implemen

null 56 Jun 12, 2022
Clang plugin to find method or property directable.

ObjCDirectFinder Clang had provided objc_direct attribute for us to write this: @property (nonatomic, assign, direct) BOOL isLaunchFinished; - (BOOL)i

Kam-To 3 Jun 2, 2022
A library to handle Apple Property List format in binary or XML

libplist A small portable C library to handle Apple Property List files in binary or XML format. Features The project provides an interface to read an

libimobiledevice 407 Jun 13, 2022
C/C++ Application to solve irrigation rotation whatever two-turn rotation or three-turn rotation, longitudinal section design, hydraulic calculations, and design of hydraulic structures like weirs and tail escape.

Irrigation works C/C++ Application to solve irrigation rotation whatever two-turn rotation or three-turn rotation, longitudinal section design, hydrau

Mohamed Jamal Ghayyad 1 Jun 24, 2022
Implementation of System V shared memory (a type of inter process communication) in xv6 operating system.

NOTE: we have stopped maintaining the x86 version of xv6, and switched our efforts to the RISC-V version (https://github.com/mit-pdos/xv6-riscv.git)

Viraj Jadhav 5 Feb 21, 2022
Cross-platform shared memory stream/buffer, header-only library for IPC in C/C++.

libsharedmemory libsharedmemory is a small C++11 header-only library for using shared memory on Windows, Linux and macOS. libsharedmemory makes it eas

Aron Homberg 8 May 23, 2022
A high performance, shared memory, lock free, cross platform, single file, no dependencies, C++11 key-value store

SimDB A high performance, shared memory, lock free, cross platform, single file, no dependencies, C++11 key-value store. SimDB is part of LAVA (Live A

null 441 Jun 8, 2022