Serverless SQLite database read from and write to Object Storage Service, run on FaaS platform.

Overview

CodeQL GitHub license

serverless-sqlite

Serverless SQLite database read from and write to Object Storage Service, run on FaaS platform.

NOTES: This repository is still in the early development stage.

Goals

  • Use Object Storage Service as the underlying storage engine, such as AWS S3, Aliyun OSS, etc.
  • Run on FaaS platform, multiple query instances can share the same database, can be scaled horizontally.
  • Provide query interface via HTTP protocol.

Why?

I'm studying how to build a serverless database recently. I have created this repository to practice my C++ skills, and for fun.

Development

Requirements

  • C++ 17 compiler, such as GCC 9, Clang 10.
  • Unix-like operating system, such as Linux, FreeBSD, macOS.
  • Development tools:
    • clang-format: used to format the source code. Run the following command to install it:
      • Ubuntu or Debian: apt install clang-format
      • macOS: brew install clang-format

Setup

  • Initialize git submodule: git submodule init && git submodule update
  • Install dependencies: ./install-deps.sh

Reference

libcephsqlite

This SQLite VFS may be used for storing and accessing a SQLite database backed by RADOS. This allows you to fully decentralize your database using Ceph’s object store for improved availability, accessibility, and use of storage. Link: https://docs.ceph.com/en/latest/rados/api/libcephsqlite/

  • PRAGMA page_size = 65536 - bigger page size
  • PRAGMA cache_size = 4096 - cache 4096 pages or 256MB (with 64K page_cache).
  • PRAGMA journal_mode = PERSIST - don't deletes the journal for every transaction
  • PRAGMA temp_store=memory - put the temporary tables on memory

sqlite-s3vfs

Python virtual filesystem for SQLite to read from and write to S3. Link: https://github.com/uktrade/sqlite-s3vfs

sqlite-s3vfs stores the SQLite database in fixed-sized blocks, and each is stored as a separate object in S3. SQLite stores its data in fixed-size pages, and always writes exactly a page at a time. This virtual filesystem translates page reads and writes to block reads and writes. In the case of SQLite pages being the same size as blocks, which is the case by default, each page write results in exactly one block write.

Separate objects are required since S3 does not support the partial replace of an object; to change even 1 byte, it must be re-uploaded in full.

Owner
老雷
Software Engineer
老雷
React-native-quick-sqlite - ⚡️ The fastest SQLite implementation for react-native.

React Native Quick SQLite The **fastest** SQLite implementation for react-native. Copy typeORM patch-package from example dir npm i react-nati

Oscar Franco 310 Aug 15, 2022
A friendly and lightweight C++ database library for MySQL, PostgreSQL, SQLite and ODBC.

QTL QTL is a C ++ library for accessing SQL databases and currently supports MySQL, SQLite, PostgreSQL and ODBC. QTL is a lightweight library that con

null 157 Aug 4, 2022
Simple constant key/value storage library, for read-heavy systems with infrequent large bulk inserts.

Sparkey is a simple constant key/value storage library. It is mostly suited for read heavy systems with infrequent large bulk inserts. It includes bot

Spotify 967 Jul 16, 2022
The database built for IoT streaming data storage and real-time stream processing.

The database built for IoT streaming data storage and real-time stream processing.

HStreamDB 522 Aug 11, 2022
MySQL Server, the world's most popular open source database, and MySQL Cluster, a real-time, open source transactional database.

Copyright (c) 2000, 2021, Oracle and/or its affiliates. This is a release of MySQL, an SQL database server. License information can be found in the

MySQL 8.1k Aug 16, 2022
A mini database for learning database

A mini database for learning database

Chuckie Tan 4 Jul 21, 2022
An SQLite binding for node.js with built-in encryption, focused on simplicity and (async) performance

Description An SQLite (more accurately SQLite3MultipleCiphers) binding for node.js focused on simplicity and (async) performance. When dealing with en

mscdex 14 May 15, 2022
Fork of sqlite4java with updated SQLite and very basic compiler hardening enabled.

Download latest version: sqlite4java-392 with SQLite 3.8.7, Windows/Linux/Mac OS X/Android binaries OSGi bundle 1.0.392 with sqlite4java-392 Files for

GrapheneOS 5 Jun 8, 2022
A hook for Project Zomboid that intercepts files access for savegames and puts them in an SQLite DB instead.

ZomboidDB This project consists of a library and patcher that results in file calls for your savegame(s) being transparently intercepted and redirecte

Oliver 6 May 6, 2022
❤️ SQLite ORM light header only library for modern C++

SQLite ORM SQLite ORM light header only library for modern C++ Status Branch Travis Appveyor master dev Advantages No raw string queries Intuitive syn

Yevgeniy Zakharov 1.6k Aug 10, 2022
SQLean: all the missing SQLite functions

SQLite has very few functions compared to other DBMS. SQLite authors see this as a feature rather than a bug, because SQLite has extension mechanism in place.

Anton Zhiyanov 1.4k Aug 8, 2022
Yet another SQLite wrapper for Nim

Yet another SQLite wrapper for Nim Features: Design for ARC/ORC, you don’t need to close the connection manually Use importdb macro to create helper f

Code Hz 8 Jul 4, 2022
An updated fork of sqlite_protobuf, a SQLite extension for extracting values from serialized Protobuf messages.

This fork of sqlite_protobuf fixes some issues (e.g., #15) and removes the test suite that we do not use. It also comes with proto_table, a C library

Backtrace Labs 17 Jun 27, 2022
Verneuil is a VFS extension for SQLite that asynchronously replicates databases to S3-compatible blob stores.

Verneuil: streaming replication for sqlite Verneuil1 [vɛʁnœj] is a VFS (OS abstraction layer) for sqlite that accesses local database files like the d

Backtrace Labs 278 Jul 27, 2022
C++ ORM for SQLite

Hiberlite ORM C++ object-relational mapping with API inspired by the awesome Boost.Serialization - that means almost no API to learn. Usage Just compi

Paul Korzhyk 622 Aug 10, 2022
The C++14 wrapper around sqlite library

sqlite modern cpp wrapper This library is a lightweight modern wrapper around sqlite C api . #include<iostream> #include <sqlite_modern_cpp.h> using n

null 689 Aug 14, 2022
Unofficial git mirror of SQLite sources (see link for build instructions)

SQLite Source Repository This repository contains the complete source code for the SQLite database engine. Some test scripts are also included. Howeve

null 2k Aug 6, 2022
Lightweight C++ wrapper for SQLite

NLDatabase Lightweight C++ wrapper for SQLite. Requirements C++11 compiler SQLite 3 Usage Let's open a database file and read some rows: #include "NLD

Raven 0 Sep 20, 2019
Writing a sqlite clone from scratch in C++

如何用C++实现一个简易数据库 基于cstack/db_tutorial C语言版本 KCNyu 2022/2/2 作为笔者写的第一个系列型教程,还是选择基于前人的教程经验以及添加一些自己个人的探索。也许有很多纰漏之处,希望大家指正。 1. 数据库是什么? 数据库是“按照数据结构来组织、存储和管理数

shengyu.li 45 Aug 9, 2022