Introduction
FEDB is a NewSQL database optimised for realtime inference and decisioning applications. These applications put real-time features extracted from multiple time windows through a pre-trained model to evaluate new data to support decision making. Existing in-memory databases cost hundreds or even thousands of milliseconds so they cannot meet the requirements of inference and decisioning applications.
FEDB use a double-layered skiplist as the core data structure to store all the data in memory and optimize sql compilation to improve the execution performance.
-
High Performance
The benchmark shows that FEDB can be one to two orders of magnitude faster than SingleStore and SAP HANA.
-
SQL Compatible
FEDB is compatible with most of ANSI SQL syntax. You can implement your aplications with sqlalchemy or JDBC.
-
Online-offline Consistency
Machine learning applications developed by FEDB can be launched simply and ensure online and offline consistency, which greatly reduces the cost.
-
High Availability
Support auto failover and scaling horizontally.
Note: The latest released FEDB is unstable and not recommend to be used in production environment.
Architecture
Quick Start
Build on Linux
docker pull 4pdosc/centos6_gcc7_hybridsql:0.1.1
git clone https://github.com/4paradigm/fedb.git
cd fedb
docker run -v `pwd`:/fedb -it 4pdosc/centos6_gcc7_hybridsql:0.1.1
cd /fedb
sh steps/init_env.sh
mkdir -p build && cd build && cmake ../ && make -j5 fedb
Demo
- Predict taxi trip duration
- Detect the healthy of online transaction and make an alert -oncoming
- Online real-time transaction fraud detection -oncoming
Performance
In AI scenarios most real-time features are time-related and required to be computed over multiple time windows. So we use computation TopN queries as benchmark scenario.
Server Specification
The server spec is as follows:
| Item | Spec |
|---|---|
| CPU | Intel Xeon Platinum 8280L Processor |
| Memory | 384 GB |
| OS | CentOS-7 with kernel 5.1.9-1.el7 |
Benchmark Results
The benchmark result shows that FEDB can be one to two orders of magnitude faster than SingleStore and SAP HANA.
Roadmap
ANSI SQL Compatibility
FEDB is currently compatible with mainstream DDL and DML syntax, and will gradually enhances the compatibility of ANSI SQL syntax.
- [2021H1] Support the standard syntax of Window, Where, Group By and Join ect.
- [2021H1&H2] Expand AI-oriented SQL syntax and UDAF functions.
Features
In order to meet the high performance requirements of realtime inference and decisioning scenarios, FEDB chooses memory as the storage engine medium. At present, the memory storage engine used in the industry has memory fragmentation and recovery efficiency problems. FEDB plans to optimize the memory allocation algorithm to reduce fragmentation and accelerate data recovery with PMEM(Intel Optane DC Persistent Memory Module).
- [2021H1]Provide a new strategy of memory allocation to reduce memory fragmentation.
- [2021H2]Support PMEM-based storage engine.
Build Ecosystem
FEDB has python client and java client which support most of JDBC API. FEDB will make a connection with big data ecosystem for integrating with Flink/Kafka/Spark simplily.
- [2021H1&H2]Support Flink/Kafka/Spark connector.
Feedback and Getting involved
- Report bugs, ask questions or give suggestions by Github Issues.
- Cannot find what you are looking for? Have a question or idea? Please post your questions or comments on our slack.
License
Apache License 2.0
A segmentation fault occurs during the operation of Demo with The Standalone Mode ../openmldb/bin/openmldb --host 127.0.0.1 --port 6527, which may be caused by insufficient memory of the machine.
8.4k Jan 9, 2023
12.8k Jan 9, 2023
1.9k Jan 8, 2023
1.9k Jan 9, 2023
23k Jan 1, 2023
24.3k Jan 5, 2023
4k Jan 5, 2023
114 Dec 26, 2022
384 Jan 2, 2023
313 Dec 31, 2022
25.9k Jan 9, 2023
17 Sep 17, 2021
8.6k Dec 26, 2022
25 May 19, 2022
4 Nov 14, 2022
6.6k Jan 1, 2023
33 Nov 6, 2022
20.4k Jan 5, 2023
90 Oct 10, 2022
3 Nov 7, 2022