Backup / restore solution for PostgreSQL

Overview

pgmoneta

pgmoneta is a backup / restore solution for PostgreSQL.

pgmoneta is named after the Roman Goddess of Memory.

Features

  • Full backup
  • Restore
  • Symlink support
  • Prometheus support
  • Remote management
  • Transport Layer Security (TLS) v1.2+ support
  • Daemon mode
  • User vault

See Getting Started on how to get started with pgmoneta.

See Configuration on how to configure pgmoneta.

Overview

pgmoneta makes use of

  • Process model
  • Shared memory model across processes
  • libev for fast network interactions
  • Atomic operations are used to keep track of state
  • The PostgreSQL command line tools

See Architecture for the architecture of pgmoneta.

Tested platforms

Compiling the source

pgmoneta requires

dnf install git gcc cmake make libev libev-devel openssl openssl-devel systemd systemd-devel zlib zlib-devel libzstd libzstd-devel python3-docutils

Alternative clang 8+ can be used.

Release build

The following commands will install pgmoneta in the /usr/local hierarchy.

git clone https://github.com/pgmoneta/pgmoneta.git
cd pgmoneta
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr/local ..
make
sudo make install

Note, that pgmoneta requires PostgreSQL 10 or later to run as the command line tools are required. These can be installed from the PostgreSQL YUM repository, or from the official distribution repository, if supported, like

dnf install -y postgresql

See RPM for how to build a RPM of pgmoneta.

Debug build

The following commands will create a DEBUG version of pgmoneta.

git clone https://github.com/pgmoneta/pgmoneta.git
cd pgmoneta
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make

Remember to set the log_level configuration option to debug5.

Contributing

Contributions to pgmoneta are managed on GitHub.com

Contributions are most welcome !

Please, consult our Code of Conduct policies for interacting in our community.

Consider giving the project a star on GitHub if you find it useful. And, feel free to follow the project on Twitter as well.

License

BSD-3-Clause

Comments
  • feat: on-disk encryption

    feat: on-disk encryption

    Code Refactoring: move aes related code from security.h(c) to aes.h and aes.c, modified includes.

    New feature: More encryption algorithms supported. User can now choose from [ 128 | 192 | 256 ] and [ cbc | ctr ]

    Refs: #56

    feature 
    opened by solokyo 7
  • pgmoneta: error while loading shared libraries: /usr/local/lib64/libpgmoneta.so.0: invalid ELF header

    pgmoneta: error while loading shared libraries: /usr/local/lib64/libpgmoneta.so.0: invalid ELF header

    Today I cloned the pgmoneta frome github. Then I did like this in my docker container(centos8): cd pgmoneta mkdir build cd build/ cmake -DCMAKE_INSTALL_PREFIX=/usr/local .. make make install When I run pgmoneta, I met this: pgmoneta: error while loading shared libraries: /usr/local/lib64/libpgmoneta.so.0: invalid ELF header. I run git log. It showed the last commmit id is a3a673274fdf921ba82bed08ec27e66cd825e818. Then I tried the release version 0.56. This problem disappeared.

    bug 
    opened by skpupil 7
  • Implement On-disk Encryption

    Implement On-disk Encryption

    We can exploit existing aes functionality used by pgmoneta-admin to encrypt wal and other backup data.

    • [x] Refactoring the code, move AES related code from security.h(c) to aes.h(c)
    • [x] Add encryption parameter to configuration
    • [x] Implement encryption API
    • [x] Implement encryption workflow
    • [x] Documentation
    • [x] Implement pgmoneta-cli [encrypt | decrypt] command

    Possible follow-ups:
    1. Support more encryption algorithms
    2. More benchmarks
    feature 
    opened by solokyo 2
  • how to solve this “No connection to pgmoneta on /tmp”

    how to solve this “No connection to pgmoneta on /tmp”

    the pg_hba.conf is

    local   all             all                                     peer
    host    all             all             127.0.0.1/32            trust
    host    all             all             ::1/128                 trust
    local   replication     all                                     peer
    host    replication     all             127.0.0.1/32            ident
    host    replication     all             ::1/128                 ident
    local   postgres        repl                                   scram-sha-256
    host    postgres        repl           127.0.0.1/32            scram-sha-256
    host    postgres        repl           ::1/128                 scram-sha-256
    host    replication     repl           127.0.0.1/32            scram-sha-256
    host    replication     repl           ::1/128                 scram-sha-256
    

    I run this

    /usr/pgsql-10/bin/createdb -U postgres -h localhost -p 5432 kklx
    

    In postgres I run this

    SELECT pg_create_physical_replication_slot('repl');
    CREATE ROLE repl WITH LOGIN REPLICATION PASSWORD 'abc123';
    

    In postgres.conf I set this unix_socket_directories = '/tmp'

    the pgmoneta.conf

    [pgmoneta]
    host = *
    metrics = 5001
    
    base_dir = /home/kk/GSoC/pgmoneta/
    
    compression = zstd
    
    retention = 7
    
    log_type = file
    log_level = info
    log_path = /tmp/pgmoneta.log
    
    unix_socket_dir = /tmp
    #unix_socket_dir = /var/run/postgresql
    pgsql_dir = /usr/bin/
    
    [kklx]
    host = localhost
    port = 5432
    user = repl
    wal_slot = repl
    

    I find something under /tmp:

    [[email protected] kk]# find /tmp  -name '*5432*' 
    /tmp/.s.PGSQL.5432.lock
    /tmp/.s.PGSQL.5432
    

    When I run ./pgmoneta-cli -c ./configs/pgmoneta.conf backup primary , I met this problem No connection to pgmoneta on /tmp

    bug 
    opened by skpupil 1
  • [#44] Handle configuration file with no newline at the end.

    [#44] Handle configuration file with no newline at the end.

    In the case the last line of the configuration file does not have a new line, such line is not parsed at all. This means the key/value pair is not extracted at all. This could cause misleading errors.

    Close #44

    opened by fluca1978 1
  • [#42] Automatic PID file path definition

    [#42] Automatic PID file path definition

    In the case the user does not set the pidfile configuration parameter, the software should define one path itself. In particular the change makes the pidfile equal to the concatenation of the unix_socket_dir (mandatory) and the string pgmoneta.<host>.pid where <host> is the value of the host configuration parameter or all if host = *.

    Changed the pidfile variable size.

    Updated the documentation and the AUTHORS file.

    Close #42.

    opened by fluca1978 1
  • Disk space usage

    Disk space usage

    Based on pgexporter_ext extension giving

    pgexporter_used_disk_space_data
    pgexporter_free_disk_space_data
    pgexporter_total_disk_space_data
    pgexporter_used_disk_space_wal
    pgexporter_free_disk_space_wal
    pgexporter_total_disk_space_wal
    

    enabled by data_dir and wal_dir properties

    feature 
    opened by jesperpedersen 1
  • Implementing a SSH storage engine.

    Implementing a SSH storage engine.

    Implementing a SSH storage engine that supports the following commands:

    • [x] backup: It is used to create a backup for a server.
    • [ ] restore: It is used to restore a backup from a server.
    • [ ] archive: It is used to archive a backup from a server.
    • [ ] delete: It is used to delete a backup from a server.
    • [ ] retain: It is used to retain a backup from a server.
    • [ ] expunge: It is used to expunge a backup from a server.
    • [ ] list-backups: It is used to list the backups for a server.
    • [x] Adding the symlink functionality.

    libssh will be used to implement the SSH storage engine, specifically the SFTP api to copy files and directories to the remote system, in addition to deleting backup remote directories.

    feature 
    opened by MariamFahmy98 2
  • Splitting commands into workflows

    Splitting commands into workflows

    Creating a workflow for the following commands:

    • [x] restore: It is used to restore a backup from a server.
    • [x] archive: It is used to archive a backup from a server.
    • [x] delete: It is used to delete a backup from a server.
    • [x] retain: It is used to retain a backup from a server.
    • [ ] expunge: It is used to expunge a backup from a server.
    • [ ] list-backups: It is used to list the backups for a server.

    Splitting the above commands to a workflow will make it easier to support them in the storage engines.

    opened by MariamFahmy98 0
Releases(0.6.0)
Owner
Backup / restore solution for PostgreSQL
null
PolarDB for PostgreSQL (PolarDB for short) is an open source database system based on PostgreSQL.

PolarDB for PostgreSQL (PolarDB for short) is an open source database system based on PostgreSQL. It extends PostgreSQL to become a share-nothing distributed database, which supports global data consistency and ACID across database nodes, distributed SQL processing, and data redundancy and high availability through Paxos based replication. PolarDB is designed to add values and new features to PostgreSQL in dimensions of high performance, scalability, high availability, and elasticity. At the same time, PolarDB remains SQL compatibility to single-node PostgreSQL with best effort.

Alibaba 2.4k Sep 24, 2022
Incremental backup with strong cryptographic confidentiality baked into the data model.

Incremental backup with strong cryptographic confidentiality baked into the data model.

Rich Felker 109 Sep 22, 2022
The official C++ client API for PostgreSQL.

libpqxx Welcome to libpqxx, the C++ API to the PostgreSQL database management system. Home page: http://pqxx.org/development/libpqxx/ Find libpqxx on

Jeroen Vermeulen 673 Sep 21, 2022
YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features

YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features. It is best to fit for cloud-native OLTP (i.e. real-time, business-critical) applications that need absolute data correctness and require at least one of the following: scalability, high tolerance to failures, or globally-distributed deployments.

yugabyte 6.9k Sep 24, 2022
A PostgreSQL extension providing an async networking interface accessible via SQL using a background worker and curl.

pg_net is a PostgreSQL extension exposing a SQL interface for async networking with a focus on scalability and UX.

Supabase 47 Sep 16, 2022
A framework to monitor and improve the performance of PostgreSQL using Machine Learning methods.

pg_plan_inspector pg_plan_inspector is being developed as a framework to monitor and improve the performance of PostgreSQL using Machine Learning meth

suzuki hironobu 184 Sep 14, 2022
Prometheus exporter for PostgreSQL

pgexporter pgexporter is a Prometheus exporter for PostgreSQL. pgexporter will connect to one or more PostgreSQL instances and let you monitor their o

null 15 Apr 17, 2022
PostgreSQL extension for pgexporter

pgexporter_ext pgexporter_ext is an extension for PostgreSQL to provide additional Prometheus metrics for pgexporter. Features Disk space metrics See

null 4 Apr 13, 2022
The PostgreSQL client API in modern C++

C++ client API to PostgreSQL {#mainpage} Dmitigr Pgfe (PostGres FrontEnd, hereinafter referred to as Pgfe) - is a C++ client API to PostgreSQL servers

Dmitry Igrishin 138 Sep 16, 2022
A friendly and lightweight C++ database library for MySQL, PostgreSQL, SQLite and ODBC.

QTL QTL is a C ++ library for accessing SQL databases and currently supports MySQL, SQLite, PostgreSQL and ODBC. QTL is a lightweight library that con

null 163 Sep 6, 2022
C++ client library for PostgreSQL

Welcome to taoPQ taoPQ is a lightweight C++ client library for accessing a PostgreSQL➚ database. It has no dependencies beyond libpq➚, the C applicati

The Art of C++ 220 Sep 6, 2022
recovery postgresql table data by update/delete/rollback/dropcolumn command

recovery postgresql table data by update/delete/rollback/dropcolumn command

RadonDB 6 Aug 4, 2022
pgagroal is a high-performance protocol-native connection pool for PostgreSQL.

pgagroal is a high-performance protocol-native connection pool for PostgreSQL.

Agroal 540 Sep 15, 2022
xxhash functions for PostgreSQL

pg_xxhash PostgreSQL ❤️ xxhash Tested with xxhash 0.8.1 and PostgreSQL 14.1 on Linux and macOS. Think twice before even considering to use it in any s

Igor Hatarist 5 Mar 11, 2022
Distributed PostgreSQL as an extension

What is Citus? Citus is a PostgreSQL extension that transforms Postgres into a distributed database—so you can achieve high performance at any scale.

Citus Data 7.3k Sep 21, 2022
High-performance time-series aggregation for PostgreSQL

PipelineDB has joined Confluent, read the blog post here. PipelineDB will not have new releases beyond 1.0.0, although critical bugs will still be fix

PipelineDB 2.5k Sep 17, 2022
upstream module that allows nginx to communicate directly with PostgreSQL database.

About ngx_postgres is an upstream module that allows nginx to communicate directly with PostgreSQL database. Configuration directives postgres_server

RekGRpth 1 Apr 29, 2022
Modern cryptography for PostgreSQL using libsodium.

pgsodium pgsodium is an encryption library extension for PostgreSQL using the libsodium library for high level cryptographic algorithms. pgsodium can

Michel Pelletier 339 Sep 16, 2022
Open Source Oracle Compatible PostgreSQL.

IvorySQL is advanced, fully featured, open source Oracle compatible PostgreSQL with a firm commitment to always remain 100% compatible and a Drop-in r

null 103 Sep 20, 2022