When I was testing query sql from one Firefox extension which called RESTer, the server is crashed on following code:

src/siri/api.c

static int api__headers_complete_cb(http_parser * parser)
{
    siri_api_request_t * ar = parser->data;

    assert (!ar->buf);

The reason is that, that Firefox extension sending multiple HTTP requests via same socket connection. Thus the headers_complete_cb called when server is executing message_complete_cb.

Tested on Ubuntu 16.04 amd64 Steps to reproduce error:

1. Install release deb packages:

• siridb-http_1.0.3_amd64.deb
• siridb-manage_2.0.1_amd64.deb
• siridb-prompt_2.0.3_amd64.deb
• siridb-server_2.0.14_amd64.deb

2. Start the server and create new database with siridb-manage

• Time precision = s
• Number (float and integer) sharding duration = 1h
• Log (string) sharding duration = 1d
• Buffer size 1024

3. Start the http interface and insert the following values

{
	"series-001": [
		[1488889532, 100],
		[1488889542, 101]
	]
}

Making a query at this stage will return the correct values as inserted. As long as we keep inserting integer values or floats ending with .0 (eg. 10.0 or 12.0) everything is fine. However something goes wrong after below insertion:

{
	"series-001": [
		[1488889552, 100.12]
	]
}

Making a query at this stage will return a wrong value for the last insertion, as seen below:

{"series-001": [[1488889532, 100], [1488889542, 101], [1488889552, 4636745735603937608]]}

Going the other way and starting with floats also fails, when an integer is eventually inserted or a float ending with .0. For instance:

{
	"series-002": [
		[1488889667, 234.34],
		[1488889677, 23.23456],
		[1488889687, 34.134]
	]
}

{
	"series-002": [
		[1488889767, 2123.0]
	]
}

Doing a query now will return a wrong value for the last insertion, as seen below:

{"series-002": [[1488889667, 234.34], [1488889677, 23.23456], [1488889687, 34.134], [1488889767, 1.049e-320]]}

Seems like inserting floats after integer values results in a very large number when reading it back out, and inserting integers after floats results in a very small number close to 0.

First of all, I am not sure about this one. We have a couple of instances where SiriDB takes way more CPU and looks stuck (no points were inserted since it got into this state). The health-check is working ok :man_shrugging:.

Stack trace:

pcre2_match_8 0x00007f0f41d8f51c
async_series_re listener.c:5763
uv__async_io 0x00007f0f41de57b3
uv__io_poll 0x00007f0f41df343b
uv_run 0x00007f0f41de5d7a
siri_start siri.c:342
main main.c:76
libc_start_main_stage2 0x00007f0f41e1d1ef
_start 0x000055cdd1852e15

GDB says: Program terminated with signal SIGSEGV, Segmentation fault. I think that's not true because I took core dump from the running process and I think in the case of SIGSEGV it would be terminated.

The query is this one: "select max() from /gateways\\.speed\\..*/ between now - 1d and now + 1h". I think escaping is not correct, but the query otherwise works and probably is not a culprit. That said I don't have a repro for this yet, but we are seeing this often so it's hopefully a matter of time.

I can provide the core dump if needed.

Support pipes via uv_pipe_t. They should be significantly faster than TCP loopback: https://stackoverflow.com/questions/14973942/tcp-loopback-connection-vs-unix-domain-socket-performance

This is still WIP and needs testing. I've only tested connection/disconnection to SiriDB via pipe on Linux (ARM).

Is there planned support for a native nodejs client utilizing libsiridb as a C addon?

I can make due with the HTTP interface for now, but you would love to have a native nodejs client to reduce the HTTP overhead.

When built against libuv1 1.44.2, siridb_query_free may call siridb_tasks_dec on a NULL siridb, causing a segfault. Add a NULL check on siridb before calling siridb_tasks_dec to avoid this.

For more background, see the related Debian bug, and Ubuntu autopkgtest failures.

When using gdb to take a closer look, this is what I found:

Thread 1 "siridb-server" received signal SIGSEGV, Segmentation fault.
siridb_query_free (handle=0x55d2a8de6bf0) at ../src/siri/db/query.c:159
159	../src/siri/db/query.c: No such file or directory.
(gdb) bt
#0  siridb_query_free (handle=0x55d2a8de6bf0) at ../src/siri/db/query.c:159
#1  0x00007ff439a4ab65 in uv__finish_close (handle=0x55d2a8de6bf0) at ./src/unix/core.c:319
#2  uv__run_closing_handles (loop=0x55d2a8dd6760) at ./src/unix/core.c:333
#3  uv_run (loop=0x55d2a8dd6760, mode=mode@entry=UV_RUN_DEFAULT) at ./src/unix/core.c:421
#4  0x000055d2a884f981 in siri_start () at ../src/siri/siri.c:342
#5  0x000055d2a8800deb in main (argc=<optimized out>, argv=<optimized out>) at ../main.c:76

I have not been able to figure out exactly what changed in libuv 1.44.2 that caused this, but this should be a safe change to make regardless.

Some of our customers recently reported SiriDB server as being stuck. I've managed to get hold of one of those servers a took a core dump.

It looks like the server is unable to accept any connection and from the backtrace, it looks like it's stuck somewhere in libuv.

The situation is more confusing by the fact that the health check at GET /status is working fine and the container is not restarted.

The docker image: https://hub.docker.com/layers/ubnt/unms-siridb/1.3.3/images/sha256-1f194131d97ae00595fccc0e212f0ef46a5b18a41ca530bc2c004b40879cf96f?context=explore Core dump: https://drive.google.com/file/d/1L-TfgJe4rQImBz6nG4tHvkoc7ILDi4R3/view?usp=sharing OS: Linux unms 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux Docker: 20.10.0, build 7287ab3

Core was generated by `siridb-server'.
#0  0x00007f8f2b8a2516 in epoll_pwait () from /lib/ld-musl-x86_64.so.1
[Current thread is 1 (LWP 8)]

#0  epoll_pwait (fd=3, ev=ev@entry=0x7ffc3a951328, cnt=cnt@entry=1024, to=to@entry=215, sigs=sigs@entry=0x0) at ./arch/x86_64/syscall_arch.h:61
#1  0x00007f8f2b8a253d in epoll_wait (fd=<optimized out>, ev=ev@entry=0x7ffc3a951328, cnt=cnt@entry=1024, to=to@entry=215) at src/linux/epoll.c:36
#2  0x00007f8f2b8751fd in uv__io_poll (loop=loop@entry=0x564df7866d00, timeout=215) at src/unix/linux-core.c:307
#3  0x00007f8f2b867d7a in uv_run (loop=0x564df7866d00, mode=mode@entry=UV_RUN_DEFAULT) at src/unix/core.c:385
#4  0x0000564df6069d5a in siri_start () at ../src/siri/siri.c:342
#5  0x0000564df601ed7c in main (argc=argc@entry=1, argv=argv@entry=0x7ffc3a954808) at ../main.c:76
#6  0x00007f8f2b89f1ef in libc_start_main_stage2 (main=0x564df601ec10 <main>, argc=1, argv=0x7ffc3a954808) at src/env/__libc_start_main.c:94
#7  0x0000564df601ee15 in _start ()

I modified the Python example slightly to quickly insert a few large series:

import asyncio
from datetime import datetime, timedelta
import random
import uuid
from siridb.connector import SiriDBClient

async def example(siri):
    await siri.connect()

    try:
        inc = timedelta(minutes=15)
        start = datetime(2020,1,1)
        end = datetime(2021,1,1) - inc
        quarters = int((end - start) / inc)
        for _ in range(100):
            values = [ (int((start + i * inc).timestamp()), random.random()) for i in range(quarters) ]

            await siri.insert({f'{str(uuid.uuid4())}': values})

    finally:
        # Close all SiriDB connections.
        siri.close()


siri = SiriDBClient(
    username='iris',
    password='siri',
    dbname='dbtest',
    hostlist=[('localhost', 9000)],  # Multiple connections are supported
    keepalive=True)

loop = asyncio.get_event_loop()
loop.run_until_complete(example(siri))

This causes a reproducible crash on version 687835d3b764406b2d2a933e308f0352c6ce2b8c on macOS 10.14.6: crashlog.txt

If I restart the server and try again, occasionally I get a different crashlog: crashlog2.txt All mentioned files exist, so I'm guessing this is some kind of race condition with the filesystem?

Hello, trying to build on Mac OS X.

Is it possible to regenerate *.mk files with custom gcc options ? Or may be you can update all *.mk files in your repo to include standard env vars CFLAGS and LDFLAGS ?

For example to build on mac os x I need to edit Build/src/test/subdir.mk like this:

# Each subdirectory must supply rules for building sources it contributes
src/test/%.o: ../src/test/%.c
        @echo 'Building file: $<'
        @echo 'Invoking: GCC C Compiler'
        gcc -I../include $(CFLAGS) -O3 -Wall -c -fmessage-length=0 -MMD -MP -MF"$(@:%.o=%.d)" -MT"$(@)" -o "$@" "$<"
        @echo 'Finished building: $<'
        @echo ' '

Before make I just set CFLAGS like this: export CFLAGS="-I/usr/local/include -I/usr/local/Cellar/pcre/8.39/include"

Thanks

Hi i can't find how to send a simple insert cmd (no clients) , docs talk about all high levels stuff but a simple insert query is nowhere to be found. can we do multi value insert, array like value? is so how to format the data because trying to do multi insert with python

for a in range(500):
            await siri.insert({'send1': [[ts,value]],'sens2': [[ts,value]],'sens3': [[ts, value]],'sens4': [[ts, value]],'sens5': [[ts, 343*value]]})

this toke 23.13 sec to execute, i suspected that something is wrong, i can run same write to mariadb in 0.53 sec

e.g I created the database with 4 weeks duration, and then set the expiration_num to 0. Does it means that the data will never been dropped? or will be dropped when reach database's duration?

BTW: why the duration cannot be bigger than 99w? is this limited by design?

Is your feature request related to a problem? Please describe. some times, series have overlaping values for the same timestame (multiple node writing to same serie) or backfilling after a failure

Describe the solution you'd like Drop duplicates periodicly with retention policy (last,first,throw) or at write

While working with the source code, I was wondering what the workflow is. Could you please document that somewhere in the README.md file?

E.g. how are the makefiles generated?

E.g. What is the connection between src, Release/src and Debug/src (and if they need syncing, how is that done)? Why isn't the delta between Release and Debug not just a configure option/make variable?