SiriDB is a highly-scalable, robust and super fast time series database

Overview

SiriDB Server

SiriDB is a highly-scalable, robust and super fast time series database.



Installation

Ubuntu

For Ubuntu we have a deb package available which can be downloaded here.

Note: SiriDB requires libexpat1, libuv1, libpcre2-8-0 and libcleri0 these libraries can be easily installed using apt:

apt install libexpat1 libuv1 libpcre2-8-0 libcleri0

Library libcleri0 is available from Ubuntu 18.04, for older versions a deb package can be found here: https://github.com/transceptor-technology/libcleri/releases/latest

The .deb package installs a configuration file at /etc/siridb/siridb.conf. You might want to view or change this file before starting SiriDB.

Compile from source

From version 2.0.19 libcleri is not included as part of this source anymore and needs to be installed separately. libcleri can be found here: https://github.com/transceptor-technology/libcleri or can be installed using apt.

Linux

Install the following requirements: (Ubuntu 18.04)

sudo apt install libcleri-dev
sudo apt install libpcre2-dev
sudo apt install libuv1-dev
sudo apt install libyajl-dev
sudo apt install uuid-dev

Compile (replace Release with Debug for a debug build):

cd ./Release
make clean
make test
make

Install

sudo make install

OSX

Make sure libcleri is installed!

Install the following requirements:

brew install pcre2
brew install libuv
brew install yajl
brew install ossp-uuid

Compile (replace Release with Debug for a debug build):

cd ./Release
export CFLAGS="-I/usr/local/include"
export LDFLAGS="-L/usr/local/lib"
make clean
make test
make

Install

sudo make install

Configuration

SiriDB requires a configuration file to run. By default SiriDB will search for the configuration file in /etc/siridb/siridb.conf but alternatively you can specify a custom path by using the -c/--config argument.

$ siridb-server -c /my/path/siridb.conf

An example configuration file can be found here: https://github.com/SiriDB/siridb-server/blob/master/siridb.conf

Build Debian package:

Install required packages (autopkgtest is required for running the tests)

apt-get install devscripts lintian help2man autopkgtest

Create archive

git archive -o ../siridb-server_2.0.31.orig.tar.gz master

Run tests

autopkgtest -B -- null

Build package

debuild -us -uc

Run integration tests

The simplest way to run the integration tests is to use docker.

Build integration test image

docker build -t siridb/itest -f itest/Dockerfile .

Run integration tests

docker run siridb/itest:latest

Create or expand a database

SiriDB Admin can be used for creating a new database or expanding an existing database with a new server. Documentation on how to install and use the admin tool can be found at the siridb-admin github project. Binaries are available for most platforms and can be downloaded from here. As an alternative it is possible to use a simple HTTP API for creating or expanding a SiriDB database.

Using SiriDB

SiriDB has several tools available to connect to a SiriDB database.

SiriDB Connectors

The following native connectors are available:

When not using one of the above, you can still communicate to SiriDB using SiriDB HTTP.

SiriDB HTTP

SiriDB HTTP provides a HTTP API for SiriDB and has support for JSON, MsgPack, Qpack, CSV and Socket.io. SiriDB HTTP also has an optional web interface and SSL support.

SiriDB Prompt

SiriDB Prompt can be used as a command line SiriDB client with auto-completion support and can be used to load json or csv data into a SiriDB database. Click here for more information about SiriDB Prompt.

Grafana

SiriDB Grafana Datasource is a plugin for Grafana. See the following blog article on how to configure and use this plugin: https://github.com/SiriDB/grafana-siridb-http-example.

Query language

Documentation about the query language can be found at https://siridb.net/documentation.

Issues
  • HTTP Pipelining issue in embedded api service

    HTTP Pipelining issue in embedded api service

    When I was testing query sql from one Firefox extension which called RESTer, the server is crashed on following code:

    src/siri/api.c

    static int api__headers_complete_cb(http_parser * parser)
    {
        siri_api_request_t * ar = parser->data;
    
        assert (!ar->buf);
    

    The reason is that, that Firefox extension sending multiple HTTP requests via same socket connection. Thus the headers_complete_cb called when server is executing message_complete_cb.

    bug 
    opened by srdgame 17
  • Siridb fails when inserting both floats and ints in same series

    Siridb fails when inserting both floats and ints in same series

    Tested on Ubuntu 16.04 amd64 Steps to reproduce error:

    1. Install release deb packages:
    • siridb-http_1.0.3_amd64.deb
    • siridb-manage_2.0.1_amd64.deb
    • siridb-prompt_2.0.3_amd64.deb
    • siridb-server_2.0.14_amd64.deb
    1. Start the server and create new database with siridb-manage
    • Time precision = s
    • Number (float and integer) sharding duration = 1h
    • Log (string) sharding duration = 1d
    • Buffer size 1024
    1. Start the http interface and insert the following values
    {
    	"series-001": [
    		[1488889532, 100],
    		[1488889542, 101]
    	]
    }
    

    Making a query at this stage will return the correct values as inserted. As long as we keep inserting integer values or floats ending with .0 (eg. 10.0 or 12.0) everything is fine. However something goes wrong after below insertion:

    {
    	"series-001": [
    		[1488889552, 100.12]
    	]
    }
    

    Making a query at this stage will return a wrong value for the last insertion, as seen below:

    {"series-001": [[1488889532, 100], [1488889542, 101], [1488889552, 4636745735603937608]]}

    Going the other way and starting with floats also fails, when an integer is eventually inserted or a float ending with .0. For instance:

    {
    	"series-002": [
    		[1488889667, 234.34],
    		[1488889677, 23.23456],
    		[1488889687, 34.134]
    	]
    }
    
    {
    	"series-002": [
    		[1488889767, 2123.0]
    	]
    }
    

    Doing a query now will return a wrong value for the last insertion, as seen below:

    {"series-002": [[1488889667, 234.34], [1488889677, 23.23456], [1488889687, 34.134], [1488889767, 1.049e-320]]}
    

    Seems like inserting floats after integer values results in a very large number when reading it back out, and inserting integers after floats results in a very small number close to 0.

    opened by AlmiS 14
  • Possible memory corruption

    Possible memory corruption

    First of all, I am not sure about this one. We have a couple of instances where SiriDB takes way more CPU and looks stuck (no points were inserted since it got into this state). The health-check is working ok :man_shrugging:.

    Stack trace:

    pcre2_match_8 0x00007f0f41d8f51c
    async_series_re listener.c:5763
    uv__async_io 0x00007f0f41de57b3
    uv__io_poll 0x00007f0f41df343b
    uv_run 0x00007f0f41de5d7a
    siri_start siri.c:342
    main main.c:76
    libc_start_main_stage2 0x00007f0f41e1d1ef
    _start 0x000055cdd1852e15
    

    GDB says: Program terminated with signal SIGSEGV, Segmentation fault. I think that's not true because I took core dump from the running process and I think in the case of SIGSEGV it would be terminated.

    The query is this one: "select max() from /gateways\\.speed\\..*/ between now - 1d and now + 1h". I think escaping is not correct, but the query otherwise works and probably is not a culprit. That said I don't have a repro for this yet, but we are seeing this often so it's hopefully a matter of time.

    I can provide the core dump if needed.

    opened by ubnt-michals 7
  • Implement named pipe support

    Implement named pipe support

    Support pipes via uv_pipe_t. They should be significantly faster than TCP loopback: https://stackoverflow.com/questions/14973942/tcp-loopback-connection-vs-unix-domain-socket-performance

    This is still WIP and needs testing. I've only tested connection/disconnection to SiriDB via pipe on Linux (ARM).

    opened by pavelxdd 7
  • Native Client for NodeJS

    Native Client for NodeJS

    Is there planned support for a native nodejs client utilizing libsiridb as a C addon?

    I can make due with the HTTP interface for now, but you would love to have a native nodejs client to reduce the HTTP overhead.

    enhancement 
    opened by beegmon 7
  • Server stuck in libuv

    Server stuck in libuv

    Some of our customers recently reported SiriDB server as being stuck. I've managed to get hold of one of those servers a took a core dump.

    It looks like the server is unable to accept any connection and from the backtrace, it looks like it's stuck somewhere in libuv.

    The situation is more confusing by the fact that the health check at GET /status is working fine and the container is not restarted.

    The docker image: https://hub.docker.com/layers/ubnt/unms-siridb/1.3.3/images/sha256-1f194131d97ae00595fccc0e212f0ef46a5b18a41ca530bc2c004b40879cf96f?context=explore Core dump: https://drive.google.com/file/d/1L-TfgJe4rQImBz6nG4tHvkoc7ILDi4R3/view?usp=sharing OS: Linux unms 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux Docker: 20.10.0, build 7287ab3

    Core was generated by `siridb-server'.
    #0  0x00007f8f2b8a2516 in epoll_pwait () from /lib/ld-musl-x86_64.so.1
    [Current thread is 1 (LWP 8)]
    
    #0  epoll_pwait (fd=3, [email protected]=0x7ffc3a951328, [email protected]=1024, [email protected]=215, [email protected]=0x0) at ./arch/x86_64/syscall_arch.h:61
    #1  0x00007f8f2b8a253d in epoll_wait (fd=<optimized out>, [email protected]=0x7ffc3a951328, [email protected]=1024, [email protected]=215) at src/linux/epoll.c:36
    #2  0x00007f8f2b8751fd in uv__io_poll ([email protected]=0x564df7866d00, timeout=215) at src/unix/linux-core.c:307
    #3  0x00007f8f2b867d7a in uv_run (loop=0x564df7866d00, [email protected]=UV_RUN_DEFAULT) at src/unix/core.c:385
    #4  0x0000564df6069d5a in siri_start () at ../src/siri/siri.c:342
    #5  0x0000564df601ed7c in main ([email protected]=1, [email protected]=0x7ffc3a954808) at ../main.c:76
    #6  0x00007f8f2b89f1ef in libc_start_main_stage2 (main=0x564df601ec10 <main>, argc=1, argv=0x7ffc3a954808) at src/env/__libc_start_main.c:94
    #7  0x0000564df601ee15 in _start ()
    
    opened by ubnt-michals 5
  • Inserting series in a loop causes fatal crash

    Inserting series in a loop causes fatal crash

    I modified the Python example slightly to quickly insert a few large series:

    import asyncio
    from datetime import datetime, timedelta
    import random
    import uuid
    from siridb.connector import SiriDBClient
    
    async def example(siri):
        await siri.connect()
    
        try:
            inc = timedelta(minutes=15)
            start = datetime(2020,1,1)
            end = datetime(2021,1,1) - inc
            quarters = int((end - start) / inc)
            for _ in range(100):
                values = [ (int((start + i * inc).timestamp()), random.random()) for i in range(quarters) ]
    
                await siri.insert({f'{str(uuid.uuid4())}': values})
    
        finally:
            # Close all SiriDB connections.
            siri.close()
    
    
    siri = SiriDBClient(
        username='iris',
        password='siri',
        dbname='dbtest',
        hostlist=[('localhost', 9000)],  # Multiple connections are supported
        keepalive=True)
    
    loop = asyncio.get_event_loop()
    loop.run_until_complete(example(siri))
    

    This causes a reproducible crash on version 687835d3b764406b2d2a933e308f0352c6ce2b8c on macOS 10.14.6: crashlog.txt

    If I restart the server and try again, occasionally I get a different crashlog: crashlog2.txt All mentioned files exist, so I'm guessing this is some kind of race condition with the filesystem?

    opened by Botje 5
  • Provide better support for building

    Provide better support for building

    Hello, trying to build on Mac OS X.

    Is it possible to regenerate *.mk files with custom gcc options ? Or may be you can update all *.mk files in your repo to include standard env vars CFLAGS and LDFLAGS ?

    For example to build on mac os x I need to edit Build/src/test/subdir.mk like this:

    # Each subdirectory must supply rules for building sources it contributes
    src/test/%.o: ../src/test/%.c
            @echo 'Building file: $<'
            @echo 'Invoking: GCC C Compiler'
            gcc -I../include $(CFLAGS) -O3 -Wall -c -fmessage-length=0 -MMD -MP -MF"$(@:%.o=%.d)" -MT"$(@)" -o "[email protected]" "$<"
            @echo 'Finished building: $<'
            @echo ' '
    

    Before make I just set CFLAGS like this: export CFLAGS="-I/usr/local/include -I/usr/local/Cellar/pcre/8.39/include"

    Thanks

    opened by deem0n 5
  • Simple insert

    Simple insert

    Hi i can't find how to send a simple insert cmd (no clients) , docs talk about all high levels stuff but a simple insert query is nowhere to be found. can we do multi value insert, array like value? is so how to format the data because trying to do multi insert with python

    for a in range(500):
                await siri.insert({'send1': [[ts,value]],'sens2': [[ts,value]],'sens3': [[ts, value]],'sens4': [[ts, value]],'sens5': [[ts, 343*value]]})
    

    this toke 23.13 sec to execute, i suspected that something is wrong, i can run same write to mariadb in 0.53 sec

    bug 
    opened by ovresko 4
  • Confused about database duration and shards expiration

    Confused about database duration and shards expiration

    e.g I created the database with 4 weeks duration, and then set the expiration_num to 0. Does it means that the data will never been dropped? or will be dropped when reach database's duration?

    BTW: why the duration cannot be bigger than 99w? is this limited by design?

    opened by srdgame 4
  • siridb-server error at startup

    siridb-server error at startup

    siridb-server version: 2.0.38 Installer: siridb-server_2.0.38-0.tt1_amd64.deb

    At the startup the program shows the following error: siridb-server: symbol lookup error: siridb-server: undefined symbol: cleri_parse

    opened by cbiale 4
  • Disk Failure

    Disk Failure

    Hello,

    We have an IoT SiriDB database that has 2 servers and 4 pools. Disk of Server1-Pool0 crashed and lost all data yesterday. Database is working stable using Pool0-Server2.

    image

    We chanced the failure disk to new disk on Server1. How can we readjust cluster structure on Database and sync two servers-pools?

    Best Regards, Enes

    question 
    opened by uzunenes 1
  • Drop dulicate values

    Drop dulicate values

    Is your feature request related to a problem? Please describe. some times, series have overlaping values for the same timestame (multiple node writing to same serie) or backfilling after a failure

    Describe the solution you'd like Drop duplicates periodicly with retention policy (last,first,throw) or at write

    enhancement 
    opened by ovresko 0
  • please document workflow for contributors

    please document workflow for contributors

    While working with the source code, I was wondering what the workflow is. Could you please document that somewhere in the README.md file?

    E.g. how are the makefiles generated?

    E.g. What is the connection between src, Release/src and Debug/src (and if they need syncing, how is that done)? Why isn't the delta between Release and Debug not just a configure option/make variable?

    opened by paulgevers 0
Releases(2.0.48)
Owner
SiriDB
SiriDB Time Series Database, maintained by Transceptor Technology.
SiriDB
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

Overview GridDB is Database for IoT with both NoSQL interface and SQL Interface. Please refer to GridDB Features Reference for functionality. This rep

GridDB 1.8k Aug 10, 2022
ObjectBox C and C++: super-fast database for objects and structs

ObjectBox Embedded Database for C and C++ ObjectBox is a superfast C and C++ database for embedded devices (mobile and IoT), desktop and server apps.

ObjectBox 137 Aug 7, 2022
以简单、易用、高性能为目标、开源的时序数据库,支持Linux和Windows, Time Series Database

松果时序数据库(pinusdb) 松果时序数据库是一款针对中小规模(设备数少于10万台,每天产生的数据量少于10亿条)场景设计的时序数据库。以简单、易用、高性能为设计目标。使用SQL语句进行交互,拥有极低的学习、使用成本, 提供了丰富的功能、较高的性能。 我们的目标是成为最简单、易用、健壮的单机时序

null 95 Aug 5, 2022
MySQL Server, the world's most popular open source database, and MySQL Cluster, a real-time, open source transactional database.

Copyright (c) 2000, 2021, Oracle and/or its affiliates. This is a release of MySQL, an SQL database server. License information can be found in the

MySQL 8.1k Aug 16, 2022
A mini database for learning database

A mini database for learning database

Chuckie Tan 4 Jul 21, 2022
High-performance time-series aggregation for PostgreSQL

PipelineDB has joined Confluent, read the blog post here. PipelineDB will not have new releases beyond 1.0.0, although critical bugs will still be fix

PipelineDB 2.5k Aug 3, 2022
StarRocks is a next-gen sub-second MPP database for full analysis senarios, including multi-dimensional analytics, real-time analytics and ad-hoc query, formerly known as DorisDB.

StarRocks is a next-gen sub-second MPP database for full analysis senarios, including multi-dimensional analytics, real-time analytics and ad-hoc query, formerly known as DorisDB.

StarRocks 3.1k Aug 10, 2022
The database built for IoT streaming data storage and real-time stream processing.

The database built for IoT streaming data storage and real-time stream processing.

HStreamDB 522 Aug 11, 2022
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large scale graphs with dozens of billions of vertices (nodes) and trillions of edges, with milliseconds of latency.

vesoft inc. 817 Aug 14, 2022
A very fast lightweight embedded database engine with a built-in query language.

upscaledb 2.2.1 Fr 10. Mär 21:33:03 CET 2017 (C) Christoph Rupp, [email protected]; http://www.upscaledb.com This is t

Christoph Rupp 539 Aug 12, 2022
libmdbx is an extremely fast, compact, powerful, embedded, transactional key-value database, with permissive license

One of the fastest embeddable key-value ACID database without WAL. libmdbx surpasses the legendary LMDB in terms of reliability, features and performance.

Леонид Юрьев (Leonid Yuriev) 1k Apr 13, 2022
TrailDB is an efficient tool for storing and querying series of events

TrailDB is an efficient tool for storing and querying series of events. This repository contains the core C library and the tdb command line tool.

TrailDB 1.1k Jul 27, 2022
A friendly and lightweight C++ database library for MySQL, PostgreSQL, SQLite and ODBC.

QTL QTL is a C ++ library for accessing SQL databases and currently supports MySQL, SQLite, PostgreSQL and ODBC. QTL is a lightweight library that con

null 157 Aug 4, 2022
dqlite is a C library that implements an embeddable and replicated SQL database engine with high-availability and automatic failover

dqlite dqlite is a C library that implements an embeddable and replicated SQL database engine with high-availability and automatic failover. The acron

Canonical 3.1k Aug 10, 2022
ESE is an embedded / ISAM-based database engine, that provides rudimentary table and indexed access.

Extensible-Storage-Engine A Non-SQL Database Engine The Extensible Storage Engine (ESE) is one of those rare codebases having proven to have a more th

Microsoft 782 Jul 29, 2022
OceanBase is an enterprise distributed relational database with high availability, high performance, horizontal scalability, and compatibility with SQL standards.

What is OceanBase database OceanBase Database is a native distributed relational database. It is developed entirely by Alibaba and Ant Group. OceanBas

OceanBase 4.5k Aug 9, 2022
Velox is a new C++ vectorized database acceleration library aimed to optimizing query engines and data processing systems.

Velox is a C++ database acceleration library which provides reusable, extensible, and high-performance data processing components

Facebook Incubator 1k Aug 9, 2022
Config and tools for config of tasmota devices from mysql database

tasmota-sql Tools for management of tasmota devices based on mysql. The tasconfig command can load config from tasmota and store in sql, or load from

RevK 3 Jan 8, 2022
Serverless SQLite database read from and write to Object Storage Service, run on FaaS platform.

serverless-sqlite Serverless SQLite database read from and write to Object Storage Service, run on FaaS platform. NOTES: This repository is still in t

老雷 7 May 12, 2022