In Memory Property Graph Server using the Shared Nothing design from Seastar.

Overview

RageDB

In Memory Property Graph Server using the Shared Nothing design from Seastar.

The RageDB server can host multiple Graphs. The graphs are accessible via a REST API (see below). Each Graph is split into multiple Shards. One Shard per Core of the server. Shards communicate by explicit message passing. Nodes and Relationships have internal and external ids. The external ids embed the type as well as which Shard they belong to. The internal ids are pointers into vectors that hold the data of each Node and Relationship. The Relationship ids are replicated to both incoming and outgoing Nodes. The Relationship Object (and properties) belong to the Outgoing Node (and shard). Each Node must have a singular Type and unique Key on creation which the server stores in a map for retrieval. External and Internal Ids are assigned upon creation for both Nodes and Relationships.

Along side an HTTP API, RageDB also has a Lua http endpoint that allows users to send complex queries. These queries are interpreted by LuaJIT, compiled and executed within a Seastar Thread that allows blocking. By not having a "query language" we avoid parsing, query planning, query execution and a host of problems.

HTTP API

Schema

Get Node Types

:GET /db/{graph}/schema/nodes

Get a Node Type

:GET /db/{graph}/schema/nodes/{type}

Create a Node Type

:POST /db/{graph}/schema/nodes/{type}

Delete a Node Type

:POST /db/{graph}/schema/nodes/{type}

Get Relationship Types

:GET /db/{graph}/schema/relationships

Get a Relationship Type

:GET /db/{graph}/schema/relationships/{type}

Create a Relationship Type

:POST /db/{graph}/schema/relationships/{type}

Delete a Relationship Type

:DELETE /db/{graph}/schema/relationships/{type}

RageDB currently supports booleans, 64-bit integers, 64-bit doubles, strings and lists of the preceding data types:

boolean, integer, double, string, boolean_list, integer_list, double_list, string_list

Get a Node Property Type

:GET /db/{graph}/schema/nodes/{type}/properties/{property}

Create a Node Property Type

:POST /db/{graph}/schema/nodes/{type}/properties/{property}/{data_type}

Delete a Node Property Type

:DELETE /db/{graph}/schema/nodes/{type}/properties/{property}

Get a Relationship Property Type

:GET /db/{graph}/schema/relationships/{type}/properties/{property}

Create a Relationship Property Type

:POST /db/{graph}/schema/relationships/{type}/properties/{property}/{data_type}

Delete a Relationship Property Type

:DELETE /db/{graph}/schema/relationships/{type}/properties/{property}

Nodes

Get All Nodes

:GET /db/{graph}/nodes?limit=100&offset=0

Get All Nodes of a Type

:GET /db/{graph}/nodes/{type}?limit=100&offset=0

Get A Node By Type and Key

:GET /db/{graph}/node/{type}/{key}

Get A Node By Id

:GET /db/{graph}/node/{id}

Create A Node

:POST /db/{graph}/node/{type}/{key}
JSON formatted Body: {properties}

Delete A Node By Type and Key

:DELETE /db/{graph}/node/{type}/{key}

Delete A Node By Id

:DELETE /db/{graph}/node/{id}

Node Properties

Get the Properties of a Node By Type and Key

:GET /db/{graph}/node/{type}/{key}/properties

Get the Properties of a Node By Id

:GET /db/{graph}/node/{id}/properties

Reset the Properties of a Node By Type and Key

:POST /db/{graph}/node/{type}/{key}/properties
JSON formatted Body: {properties}

Reset the Properties of a Node By Id

:POST /db/{graph}/node/{id}/properties
JSON formatted Body: {properties}

Set some Properties of a Node By Type and Key

:PUT /db/{graph}/node/{type}/{key}/properties
JSON formatted Body: {properties}

Set some Properties of a Node By Id

:PUT /db/{graph}/node/{id}/properties
JSON formatted Body: {properties}

Delete the Properties of a Node By Type and Key

:DELETE /db/{graph}/node/{type}/{key}/properties

Delete the Properties of a Node By Id

:DELETE /db/{graph}/node/{id}/properties

Get a Property of a Node By Type and Key

:GET /db/{graph}/node/{type}/{key}/property/{property}

Get a Property of a Node By Id

:GET /db/{graph}/node/{id}/property/{property}

Create a Property of a Node By Type and Key

:PUT /db/{graph}/node/{type}/{key}/property/{property}
JSON formatted Body: {property}

Create a Property of a Node By Id

:PUT /db/{graph}/node/{id}/property/{property}
JSON formatted Body: {property}

Delete a Property of a Node By Type and Key

:DELETE /db/{graph}/node/{type}/{key}/property/{property}

Delete a Property of a Node By Id

:DELETE /db/{graph}/node/{id}/property/{property}

Relationships

Get A Relationship

:GET /db/{graph}/relationship/{id}

Create A Relationship By Node Types

:POST /db/{graph}/node/{type_1}/{key_1}/relationship/{type_2}/{key_2}/{rel_type}
JSON formatted Body: {properties}

Create A Relationship By Node Ids

:POST /db/{graph}/node/{id_1}/relationship/{id_2}/{rel_type}
JSON formatted Body: {properties}

Delete A Relationship

:DELETE /db/{graph}/relationship/{id}

Get the Relationships of a Node By Node Type

:GET /db/{graph}/node/{type}/{key}/relationships
:GET /db/{graph}/node/{type}/{key}/relationships/{direction [all, in, out]} 
:GET /db/{graph}/node/{type}/{key}/relationships/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{type}/{key}/relationships/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Get the Relationships of a Node By Node Id

:GET /db/{graph}/node/{id}/relationships
:GET /db/{graph}/node/{id}/relationships/{direction [all, in, out]} 
:GET /db/{graph}/node/{id}/relationships/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{id}/relationships/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Relationship Properties

Get the Properties of a Relationship

:GET /db/{graph}/relationship/{id}/properties

Reset the Properties of a Relationship

:POST /db/{graph}/relationship/{id}/properties
JSON formatted Body: {properties}

Set some Properties of a Relationship

:PUT /db/{graph}/relationship/{id}/properties
JSON formatted Body: {properties}

Delete the Properties of a Relationship

:DELETE /db/{graph}/relationship/{id}/properties

Get a Property of a Relationship

:GET /db/{graph}/relationship/{id}/property/{property}

Create a Property of a Relationship

:PUT /db/{graph}/relationship/{id}/property/{property}
JSON formatted Body: {property}

Delete a Property of a Relationship

:DELETE /db/{graph}/relationship/{id}/property/{property}

Node Degrees

Get the Degree of a Node By Node Type

:GET /db/{graph}/node/{type}/{key}/degree
:GET /db/{graph}/node/{type}/{key}/degree/{direction [all, in, out]} 
:GET /db/{graph}/node/{type}/{key}/degree/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{type}/{key}/degree/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Get the Degree of a Node By Node Id

:GET /db/{graph}/node/{id}/degree
:GET /db/{graph}/node/{id}/degree/{direction [all, in, out]} 
:GET /db/{graph}/node/{id}/degree/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{id}/degree/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Node Neighbors

Get the Neighbors of a Node By Node Type

:GET /db/{graph}/node/{type}/{key}/neighbors
:GET /db/{graph}/node/{type}/{key}/neighbors/{direction [all, in, out]} 
:GET /db/{graph}/node/{type}/{key}/neighbors/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{type}/{key}/neighbors/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Get the Neighbors of a Node By Node Id

:GET /db/{graph}/node/{id}/neighbors
:GET /db/{graph}/node/{id}/neighbors/{direction [all, in, out]} 
:GET /db/{graph}/node/{id}/neighbors/{direction [all, in, out]}/{type TYPE_ONE}
:GET /db/{graph}/node/{id}/neighbors/{direction [all, in, out]}/{type(s) TYPE_ONE&TYPE_TWO}

Lua

:POST db/{graph}/lua
STRING formatted Body: {script}

The script must end in one or more values that will be returned in JSON format inside an Array. Within the script the user can access to graph functions. For example:

-- Get some things about a node
a = NodeGetId("Node","Max")
b = NodeTypesGetCount()
c = NodeTypesGetCountByType("Node")
d = NodePropertyGet("Node", "Max", "name")
e = NodePropertyGetById(a, "name")
a, b, c, d, e

A second example:

-- get the names of nodes I have relationships with
names = {}
ids = NodeGetRelationshipsIds("Node", "Max")
for k=1,#ids do
    v = ids[k]
    table.insert(names, NodePropertyGetById(v.node_id, "name"))
end
names

Building

RageDB uses Seastar which only runs on *nix servers (no windows or mac) so use your local linux desktop or use EC2.

On EC2 launch an instance:

Step 1: Choose an Amazon Machine Image
Ubuntu Server 20.04 LTS(HVM), SSD Volume Type - ami-09e67e426f25ce0d7

Step 2: Choose Instance Type
r5.2xlarge

Step 3: Configure Instance
Specify CPU options
Threads per core = 1

Step 4: Add Storage
100 GiB

Launch

Once the instance is running, connect to it and start a "screen" session, then follow these steps:

First let's update and upgrade to the latest versions of local software:

sudo apt-get update && sudo apt-get dist-upgrade

Install Seastar (this will take a while, that's why we are using screen):

git clone https://github.com/scylladb/seastar.git
cd seastar
sudo ./install_dependencies.sh
./configure.py --mode=release --prefix=/usr/local
sudo ninja -C build/release install

Install Additional Dependencies

sudo apt-get install -y ccache python3-pip

Install conan

pip install --user conan
sudo ln -s ~/.local/bin/conan /usr/bin/conan

Install LuaJIT

sudo apt-get install -y luajit luajit-5.1-dev

Troubleshooting

If you get errors regarding conan locks, run:

conan remove --locks

Missing Features that can be added "easily"

- Allow Node and Relationship Type handlers to take a json map defining the property keys and data types
- Allow additional data types: 8, 16 and 32 bit integers, 32 bit floats, byte, list of bytes, nested types 
- NodeTypes and RelationshipTypes should allow type deletion and type id reuse
- Allow property type conversions (int to double, string to int, int to int array, etc).

PVS Commands

pvs-studio-analyzer trace -- make
pvs-studio-analyzer analyze
plog-converter -a GA:1,2 -t tasklist  PVS-Studio.log
Comments
  • Running Docker image on Mac M1

    Running Docker image on Mac M1

    I do try to run the docker on MacBook with Apple M1 Max chip

    The result is bellow. How to approach it?

    Mention: I intend only use the DB, not develop, so may be there a simplier (or already compiled) DB engine

    docker run -u 0 -p 127.0.0.1:7243:7243 --name ragedb -t dockerhub.ragedb.com/ragedb/ragedb:latest --cap-add=sys_nice
    WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
    Killed
    

    Thanks

    opened by danve 9
  • Clarification on RageDB

    Clarification on RageDB

    Hello,

    I have been reading about your RageDB project since I am searching for a core system that I can build on for a project that will be using MSYS2/MinGW64 but am a bit confused.

    Is RageDB an independent property graph database or is it a layer on this other project that was referred here called Seastar?

    I am not sure what Seastar is.

    I need to locate a simple in-memory property graph library and am wondering if RageDB could be converted for that purpose as it would be integrated with some other libraries for a particular graph database project.

    Also, can RageDB be ported to MinGW?

    opened by lonnietc 2
  • Deal with date parsing slowdown

    Deal with date parsing slowdown

    date(message.creationDate):todouble() takes a long time. It doubles import time for LDBC 10. Handle it in the c++ side by adding real datetime type or move date parsing to c++

    small enhancement 
    opened by maxdemarzi 2
  • Add Shell Interface

    Add Shell Interface

    Once version 2.0 of https://github.com/daniele77/cli gets pushed on conan, consider adding a shell interface much like the Crash shell was added on the lower half of https://maxdemarzi.com/2017/01/18/our-own-multi-model-database-part-5/

    large enhancement 
    opened by maxdemarzi 1
  • Use-after-free

    Use-after-free

    NOTE: not even compile tested. btw, if you don't store a ref to shared ptr in lambda, p2 may be destroyed after when_all_succeed() defers. note that seastar projects like scylladb use do_with() to handle this, or just store a ref to shared ptr in lambda like done by this fix.

    opened by raphaelsc 1
  • Create a read only sandbox

    Create a read only sandbox

    Create a sandbox that only had read only graph methods. We can try using it without taking a lock (maybe). Gives us a read only environment to deploy for testing or demos.

    opened by maxdemarzi 0
  • make sure c++ classes are implemented correctly

    make sure c++ classes are implemented correctly

    https://www.fluentcpp.com/2019/04/23/the-rule-of-zero-zero-constructor-zero-calorie/ https://en.cppreference.com/w/cpp/language/rule_of_three https://www.codementor.io/@sandesh87/the-rule-of-five-in-c-1pdgpzb04f

    opened by maxdemarzi 0
  • Overload vectors once Sol2 supports it

    Overload vectors once Sol2 supports it

    I don't think this works due to overloaded vectors see https://github.com/ThePhD/sol2/issues/1357 , break out if we have to:

        lua.set_function("NodesGet", sol::overload(
              [this](std::vector<uint64_t> ids) { return this->NodesGetViaLua(ids); },
              [this](std::vector<Link> links) { return this->NodesGetByLinksViaLua(links); }
             ));
          lua.set_function("NodesGetKey", sol::overload(
              [this](std::vector<uint64_t> ids) { return this->NodesGetKeyViaLua(ids); },
              [this](std::vector<Link> links) { return this->NodesGetKeyByLinksViaLua(links); }
             ));
          lua.set_function("NodesGetType", sol::overload(
              [this](std::vector<uint64_t> ids) { return this->NodesGetTypeViaLua(ids); },
              [this](std::vector<Link> links) { return this->NodesGetTypeByLinksViaLua(links); }
             ));
          lua.set_function("NodesGetProperty", sol::overload(
              [this](std::vector<uint64_t> ids, const std::string& property) { return this->NodesGetPropertyViaLua(ids, property); },
              [this](std::vector<Link> links, const std::string& property) { return this->NodesGetPropertyByLinksViaLua(links, property); }
             ));
          lua.set_function("NodesGetProperties", sol::overload(
              [this](std::vector<uint64_t> ids) { return this->NodesGetPropertiesViaLua(ids); },
              [this](std::vector<Link> links) { return this->NodesGetPropertiesByLinksViaLua(links); }
             ));
    
    opened by maxdemarzi 2
Owner
Rage DB
Rage DB
MillenniumDB is a graph oriented database management system

Millennium DB MillenniumDB is a graph oriented database management system developed by the Millennium Institute for Foundational Research on Data (IMF

null 27 Nov 26, 2022
DB Browser for SQLite (DB4S) is a high quality, visual, open source tool to create, design, and edit database files compatible with SQLite.

DB Browser for SQLite What it is DB Browser for SQLite (DB4S) is a high quality, visual, open source tool to create, design, and edit database files c

null 17.5k Jan 2, 2023
prometheus exporter using workflow HTTP server

wf-prometheus This is a light prometheus exporter using workflow HTTP server. This project is currently in the development stage, and the first versio

C++ Workflow Project and Ecosystem 10 Jan 4, 2023
C++ embedded memory database

ShadowDB 一个C++嵌入式内存数据库 语法极简风 支持自定义索引、复合条件查询('<','<=','==','>=','>','!=',&&,||) 能够快速fork出一份数据副本 // ShadowDB简单示例 // ShadowDB是一个可以创建索引、能够快速fork出一份数据分支的C+

null 13 Nov 10, 2022
MySQL Server, the world's most popular open source database, and MySQL Cluster, a real-time, open source transactional database.

Copyright (c) 2000, 2021, Oracle and/or its affiliates. This is a release of MySQL, an SQL database server. License information can be found in the

MySQL 8.6k Dec 26, 2022
Beryl-cli is a client for the BerylDB database server

Beryl-cli is a client for the BerylDB database server. It offers multiple commands and is designed to be fast and user-friendly.

BerylDB 11 Oct 9, 2022
A proxy server for OceanBase Database.

OceanBase Database Proxy TODO: some badges here OceanBase Database Proxy (ODP for short) is a dedicated proxy server for OceanBase Database. OceanBase

OceanBase 79 Dec 9, 2022
A PostgreSQL extension providing an async networking interface accessible via SQL using a background worker and curl.

pg_net is a PostgreSQL extension exposing a SQL interface for async networking with a focus on scalability and UX.

Supabase 49 Dec 14, 2022
A framework to monitor and improve the performance of PostgreSQL using Machine Learning methods.

pg_plan_inspector pg_plan_inspector is being developed as a framework to monitor and improve the performance of PostgreSQL using Machine Learning meth

suzuki hironobu 198 Dec 27, 2022
Modern cryptography for PostgreSQL using libsodium.

pgsodium pgsodium is an encryption library extension for PostgreSQL using the libsodium library for high level cryptographic algorithms. pgsodium can

Michel Pelletier 386 Dec 23, 2022
✔️The smallest header-only GUI library(4 KLOC) for all platforms

Welcome to GUI-lite The smallest header-only GUI library (4 KLOC) for all platforms. 中文 Lightweight ✂️ Small: 4,000+ lines of C++ code, zero dependenc

null 6.6k Jan 8, 2023
Shared-Memory Parallel Graph Partitioning for Large K

KaMinPar The graph partitioning software KaMinPar -- Karlsruhe Minimal Graph Partitioning. KaMinPar is a shared-memory parallel tool to heuristically

Karlsruhe High Quality Graph Partitioning 17 Nov 10, 2022
C++ library to create dynamic structures in plain memory of shared-memory segments

Ищи описание на хабре @mrlolthe1st. #define _CRT_SECURE_NO_WARNINGS #include "shared_structures.h" #include <iostream> #include <fstream> #include <ca

Александр Новожилов 4 Mar 30, 2022
Arduino library for sending email and SMS from nothing but the ESP8266!

Did you know your ESP8266 could send Email and SMS without any special hardware or paid services like Twilio? With AlertMe, your ESP8266 project can:

Lixie Labs 61 Feb 24, 2022
Fast C++ IPC using shared memory

Fast C++ IPC using shared memory

Dheeraj R Reddy 397 Dec 14, 2022
Using shared memory to communicate between two executables or processes, for Windows, Linux and MacOS (posix). Can also be useful for remote visualization/debugging.

shared-memory-example Using shared memory to communicate between two executables or processes, for Windows, Linux and MacOS (posix). Can also be usefu

null 9 Aug 17, 2022
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large-scale graphs with billions of vertices (nodes) and trillions of edges, with milliseconds of latency. It delivers enterprise-grade high performance to simplify the most complex data sets imaginable into meaningful and useful information.

vesoft inc. 8.4k Jan 9, 2023
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large scale graphs with dozens of billions of vertices (nodes) and trillions of edges, with milliseconds of latency.

vesoft inc. 834 Dec 24, 2022
Memgraph is a streaming graph application platform that helps you wrangle your streaming data, build sophisticated models that you can query in real-time, and develop graph applications.

Memgraph is a streaming graph application platform that helps you wrangle your streaming data, build sophisticated models that you can query in real-time, and develop graph applications.

Memgraph 797 Dec 29, 2022
By putting in a lot of speed, the speed sequence is sorted and divided, three types of speed interval distribution maps are generated.(including broken line graph,histogram and curve graph)

Auto-drawing-speed-range-map By putting in a lot of speed, the speed sequence is sorted and divided, three types of speed interval distribution maps a

wellwellAllwen 4 May 14, 2022