Backup / restore solution for PostgreSQL

Overview

pgmoneta

pgmoneta is a backup / restore solution for PostgreSQL.

pgmoneta is named after the Roman Goddess of Memory.

Features

  • Full backup
  • Restore
  • Symlink support
  • Prometheus support
  • Remote management
  • Transport Layer Security (TLS) v1.2+ support
  • Daemon mode
  • User vault

See Getting Started on how to get started with pgmoneta.

See Configuration on how to configure pgmoneta.

Overview

pgmoneta makes use of

  • Process model
  • Shared memory model across processes
  • libev for fast network interactions
  • Atomic operations are used to keep track of state
  • The PostgreSQL command line tools

See Architecture for the architecture of pgmoneta.

Tested platforms

Compiling the source

pgmoneta requires

dnf install git gcc cmake make libev libev-devel openssl openssl-devel systemd systemd-devel zlib zlib-devel libzstd libzstd-devel python3-docutils

Alternative clang 8+ can be used.

Release build

The following commands will install pgmoneta in the /usr/local hierarchy.

git clone https://github.com/pgmoneta/pgmoneta.git
cd pgmoneta
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr/local ..
make
sudo make install

Note, that pgmoneta requires PostgreSQL 10 or later to run as the command line tools are required. These can be installed from the PostgreSQL YUM repository, or from the official distribution repository, if supported, like

dnf install -y postgresql

See RPM for how to build a RPM of pgmoneta.

Debug build

The following commands will create a DEBUG version of pgmoneta.

git clone https://github.com/pgmoneta/pgmoneta.git
cd pgmoneta
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make

Remember to set the log_level configuration option to debug5.

Contributing

Contributions to pgmoneta are managed on GitHub.com

Contributions are most welcome !

Please, consult our Code of Conduct policies for interacting in our community.

Consider giving the project a star on GitHub if you find it useful. And, feel free to follow the project on Twitter as well.

License

BSD-3-Clause

Comments
  • feat: on-disk encryption

    feat: on-disk encryption

    Code Refactoring: move aes related code from security.h(c) to aes.h and aes.c, modified includes.

    New feature: More encryption algorithms supported. User can now choose from [ 128 | 192 | 256 ] and [ cbc | ctr ]

    Refs: #56

    feature 
    opened by solokyo 7
  • pgmoneta: error while loading shared libraries: /usr/local/lib64/libpgmoneta.so.0: invalid ELF header

    pgmoneta: error while loading shared libraries: /usr/local/lib64/libpgmoneta.so.0: invalid ELF header

    Today I cloned the pgmoneta frome github. Then I did like this in my docker container(centos8): cd pgmoneta mkdir build cd build/ cmake -DCMAKE_INSTALL_PREFIX=/usr/local .. make make install When I run pgmoneta, I met this: pgmoneta: error while loading shared libraries: /usr/local/lib64/libpgmoneta.so.0: invalid ELF header. I run git log. It showed the last commmit id is a3a673274fdf921ba82bed08ec27e66cd825e818. Then I tried the release version 0.56. This problem disappeared.

    bug 
    opened by skpupil 7
  • Implement On-disk Encryption

    Implement On-disk Encryption

    We can exploit existing aes functionality used by pgmoneta-admin to encrypt wal and other backup data.

    • [x] Refactoring the code, move AES related code from security.h(c) to aes.h(c)
    • [x] Add encryption parameter to configuration
    • [x] Implement encryption API
    • [x] Implement encryption workflow
    • [x] Documentation
    • [x] Implement pgmoneta-cli [encrypt | decrypt] command

    Possible follow-ups:
    1. Support more encryption algorithms
    2. More benchmarks
    feature 
    opened by solokyo 2
  • how to solve this “No connection to pgmoneta on /tmp”

    how to solve this “No connection to pgmoneta on /tmp”

    the pg_hba.conf is

    local   all             all                                     peer
    host    all             all             127.0.0.1/32            trust
    host    all             all             ::1/128                 trust
    local   replication     all                                     peer
    host    replication     all             127.0.0.1/32            ident
    host    replication     all             ::1/128                 ident
    local   postgres        repl                                   scram-sha-256
    host    postgres        repl           127.0.0.1/32            scram-sha-256
    host    postgres        repl           ::1/128                 scram-sha-256
    host    replication     repl           127.0.0.1/32            scram-sha-256
    host    replication     repl           ::1/128                 scram-sha-256
    

    I run this

    /usr/pgsql-10/bin/createdb -U postgres -h localhost -p 5432 kklx
    

    In postgres I run this

    SELECT pg_create_physical_replication_slot('repl');
    CREATE ROLE repl WITH LOGIN REPLICATION PASSWORD 'abc123';
    

    In postgres.conf I set this unix_socket_directories = '/tmp'

    the pgmoneta.conf

    [pgmoneta]
    host = *
    metrics = 5001
    
    base_dir = /home/kk/GSoC/pgmoneta/
    
    compression = zstd
    
    retention = 7
    
    log_type = file
    log_level = info
    log_path = /tmp/pgmoneta.log
    
    unix_socket_dir = /tmp
    #unix_socket_dir = /var/run/postgresql
    pgsql_dir = /usr/bin/
    
    [kklx]
    host = localhost
    port = 5432
    user = repl
    wal_slot = repl
    

    I find something under /tmp:

    [[email protected] kk]# find /tmp  -name '*5432*' 
    /tmp/.s.PGSQL.5432.lock
    /tmp/.s.PGSQL.5432
    

    When I run ./pgmoneta-cli -c ./configs/pgmoneta.conf backup primary , I met this problem No connection to pgmoneta on /tmp

    bug 
    opened by skpupil 1
  • [#44] Handle configuration file with no newline at the end.

    [#44] Handle configuration file with no newline at the end.

    In the case the last line of the configuration file does not have a new line, such line is not parsed at all. This means the key/value pair is not extracted at all. This could cause misleading errors.

    Close #44

    opened by fluca1978 1
  • [#42] Automatic PID file path definition

    [#42] Automatic PID file path definition

    In the case the user does not set the pidfile configuration parameter, the software should define one path itself. In particular the change makes the pidfile equal to the concatenation of the unix_socket_dir (mandatory) and the string pgmoneta.<host>.pid where <host> is the value of the host configuration parameter or all if host = *.

    Changed the pidfile variable size.

    Updated the documentation and the AUTHORS file.

    Close #42.

    opened by fluca1978 1
  • Disk space usage

    Disk space usage

    Based on pgexporter_ext extension giving

    pgexporter_used_disk_space_data
    pgexporter_free_disk_space_data
    pgexporter_total_disk_space_data
    pgexporter_used_disk_space_wal
    pgexporter_free_disk_space_wal
    pgexporter_total_disk_space_wal
    

    enabled by data_dir and wal_dir properties

    feature 
    opened by jesperpedersen 1
  • Implementing Azure Storage Engine

    Implementing Azure Storage Engine

    Implementing Azure storage engine that supports the following commands:

    • [x] backup: It is used to create a backup for a server.
    • [ ] restore: It is used to restore a backup from a server.
    • [ ] archive: It is used to archive a backup from a server.
    • [ ] delete: It is used to delete a backup from a server.
    • [ ] retain: It is used to retain a backup from a server.
    • [ ] expunge: It is used to expunge a backup from a server.
    • [ ] list-backups: It is used to list the backups for a server.

    HTTP requests will be sent using libcurl.

    feature 
    opened by MariamFahmy98 0
  • pgmoneta fork proceses and no close

    pgmoneta fork proceses and no close

    Describe the bug

    5293 "pgmoneta: main" ├─ 5361 "pgmoneta: main" ├─ 5434 "pgmoneta: main" ├─ 5601 "pgmoneta: main" ├─ 5677 "pgmoneta: main" ├─ 5779 "pgmoneta: main" ├─ 5859 "pgmoneta: main" ├─ 5958 "pgmoneta: main" ├─ 6078 "pgmoneta: main" ├─ 6177 "pgmoneta: main" ├─ 6235 "pgmoneta: main" ├─ 6290 "pgmoneta: main" ├─ 6347 "pgmoneta: main" ├─ 6485 "pgmoneta: main" ├─ 6587 "pgmoneta: main" ├─ 6653 "pgmoneta: main" ├─ 6754 "pgmoneta: main" ├─ 6882 "pgmoneta: main" ├─ 7038 "pgmoneta: main" ├─ 7113 "pgmoneta: main" ├─ 7245 "pgmoneta: main" ├─ 7382 "pgmoneta: main" ├─ 7542 "pgmoneta: main"

    ################################### Oct 20 15:31:43 pgmoneta kernel: traps: pgmoneta[5163] trap divide error ip:7f5f747722bc sp:7ffcb78eff50 error:0 in libpgmoneta.so.0.6.0[7f5f74756000+25000] Oct 20 15:31:43 pgmoneta systemd-coredump[5168]: Not enough arguments passed by the kernel (6, expected 7). Oct 20 15:31:43 pgmoneta kernel: traps: pgmoneta[5163] trap divide error ip:7f5f747722bc sp:7ffcb78eff50 error:0 in libpgmoneta.so.0.6.0[7f5f74756000+25000] Oct 20 15:31:43 pgmoneta systemd-coredump[5168]: Not enough arguments passed by the kernel (6, expected 7).

    To Reproduce

    Steps to reproduce the behavior.

    Version

    version 0.6.0

    PostgreSQL

    gsql-14

    libev

    What is the version of libev ?

    OpenSSL

    What is the version of OpenSSL ?

    OS

    Almalinux 9

    Configuration

    Can you provide the configuration pgmoneta ?

    host = 0.0.0.0 metrics = 5001

    base_dir = /home/pgmoneta

    compression = zstd #compression_level = 0

    retention = 7

    log_type = file log_level = debug log_path = /var/log/pgmoneta/pgmoneta.log

    #unix_socket_dir = /var/lib/pgsql/data/base unix_socket_dir = /tmp #pgsql_dir = /usr/bin/ pgsql_dir = /usr/pgsql-14/bin/

    [nextcloud] host = 192.168.53.156 port = 5432 user = replica #wal_slot= replica

    [demodb3] host = 192.168.51.214 port = 5432 user = replica #wal_slot = replica

    Debug l2022-10-20 15:02:47 INFO main.c:620 pgmoneta: started on 0.0.0.0 2022-10-20 15:02:47 DEBUG main.c:621 Management: 4 2022-10-20 15:02:47 DEBUG main.c:624 Metrics: 7 2022-10-20 15:02:47 DEBUG utils.c:343 libev available: select 2022-10-20 15:02:47 DEBUG utils.c:347 libev available: poll 2022-10-20 15:02:47 DEBUG utils.c:351 libev available: epoll 2022-10-20 15:02:47 DEBUG utils.c:359 libev available: iouring 2022-10-20 15:02:47 DEBUG main.c:631 libev engine: epoll 2022-10-20 15:02:47 DEBUG main.c:635 OpenSSL 3.0.1 14 Dec 2021 2022-10-20 15:02:47 DEBUG main.c:637 Configuration size: 144768 2022-10-20 15:02:47 DEBUG main.c:638 Known users: 1 2022-10-20 15:02:47 DEBUG main.c:639 Known admins: 0 2022-10-20 15:02:47 INFO wal.c:120 WAL: demodb3 2022-10-20 15:02:47 INFO wal.c:120 WAL: nextcloud 2022-10-20 15:03:03 DEBUG main.c:754 pgmoneta: Management backup: all 2022-10-20 15:03:11 DEBUG wf_basebackup.c:173 Base: nextcloud/20221020150303 (Elapsed: 00:00:08) 2022-10-20 15:03:11 INFO se_local.c:85 Local storage engine: nextcloud/20221020150303 (Elapsed: 00:00:00) 2022-10-20 15:03:12 DEBUG wf_zstd.c:122 Compression: nextcloud/20221020150303 (Elapsed: 00:00:01) 2022-10-20 15:03:12 INFO backup.c:139 Backup: nextcloud/20221020150303 (Elapsed: 00:00:09) 2022-10-20 15:17:56 DEBUG wf_basebackup.c:173 Base: demodb3/20221020150303 (Elapsed: 00:14:53) 2022-10-20 15:17:56 INFO se_local.c:85 Local storage engine: demodb3/20221020150303 (Elapsed: 00:00:00) ogs

    Tip

    Use ``` before and after the text to keep the output as is.

    bug 
    opened by lubomarinov 1
  • Implementing AWS S3 storage engine

    Implementing AWS S3 storage engine

    Implementing an AWS S3 storage engine that supports the following commands:

    • [x] backup: It is used to create a backup for a server.
    • [ ] restore: It is used to restore a backup from a server.
    • [ ] archive: It is used to archive a backup from a server.
    • [ ] delete: It is used to delete a backup from a server.
    • [ ] retain: It is used to retain a backup from a server.
    • [ ] expunge: It is used to expunge a backup from a server.
    • [ ] list-backups: It is used to list the backups for a server.

    HTTP requests will be sent using libcurl.

    feature 
    opened by MariamFahmy98 0
  • Consolidate compiler generated dependencies of target pgmoneta

    Consolidate compiler generated dependencies of target pgmoneta

    Describe the bug

    Whene I install pgmoneta whit make command in macos. I am getting error like this

    [ 0%] Built target man Consolidate compiler generated dependencies of target pgmoneta [ 2%] Building C object src/CMakeFiles/pgmoneta.dir/libpgmoneta/aes.c.o /usr/local/pgmoneta/src/libpgmoneta/aes.c:60:28: error: use of undeclared identifier 'DT_DIR' if (entry->d_type == DT_DIR) ^ /usr/local/pgmoneta/src/libpgmoneta/aes.c:144:28: error: use of undeclared identifier 'DT_REG' if (entry->d_type == DT_REG) ^ /usr/local/pgmoneta/src/libpgmoneta/aes.c:223:28: error: use of undeclared identifier 'DT_DIR' if (entry->d_type == DT_DIR) ^ 3 errors generated. make[2]: *** [src/CMakeFiles/pgmoneta.dir/libpgmoneta/aes.c.o] Error 1 make[1]: *** [src/CMakeFiles/pgmoneta.dir/all] Error 2 make: *** [all] Error 2

    How can ı solve this problem. thenks

    bug 
    opened by omerfruk 3
  • Implementing a SSH storage engine.

    Implementing a SSH storage engine.

    Implementing a SSH storage engine that supports the following commands:

    • [x] backup: It is used to create a backup for a server.
    • [ ] restore: It is used to restore a backup from a server.
    • [ ] archive: It is used to archive a backup from a server.
    • [ ] delete: It is used to delete a backup from a server.
    • [ ] retain: It is used to retain a backup from a server.
    • [ ] expunge: It is used to expunge a backup from a server.
    • [ ] list-backups: It is used to list the backups for a server.
    • [x] Adding the symlink functionality.

    libssh will be used to implement the SSH storage engine, specifically the SFTP api to copy files and directories to the remote system, in addition to deleting backup remote directories.

    feature 
    opened by MariamFahmy98 2
Releases(0.6.0)
Owner
Backup / restore solution for PostgreSQL
null
PolarDB for PostgreSQL (PolarDB for short) is an open source database system based on PostgreSQL.

PolarDB for PostgreSQL (PolarDB for short) is an open source database system based on PostgreSQL. It extends PostgreSQL to become a share-nothing distributed database, which supports global data consistency and ACID across database nodes, distributed SQL processing, and data redundancy and high availability through Paxos based replication. PolarDB is designed to add values and new features to PostgreSQL in dimensions of high performance, scalability, high availability, and elasticity. At the same time, PolarDB remains SQL compatibility to single-node PostgreSQL with best effort.

Alibaba 2.5k Dec 31, 2022
Incremental backup with strong cryptographic confidentiality baked into the data model.

Incremental backup with strong cryptographic confidentiality baked into the data model.

Rich Felker 117 Dec 29, 2022
The official C++ client API for PostgreSQL.

libpqxx Welcome to libpqxx, the C++ API to the PostgreSQL database management system. Home page: http://pqxx.org/development/libpqxx/ Find libpqxx on

Jeroen Vermeulen 718 Jan 3, 2023
YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features

YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features. It is best to fit for cloud-native OLTP (i.e. real-time, business-critical) applications that need absolute data correctness and require at least one of the following: scalability, high tolerance to failures, or globally-distributed deployments.

yugabyte 7.4k Jan 7, 2023
A PostgreSQL extension providing an async networking interface accessible via SQL using a background worker and curl.

pg_net is a PostgreSQL extension exposing a SQL interface for async networking with a focus on scalability and UX.

Supabase 49 Dec 14, 2022
A framework to monitor and improve the performance of PostgreSQL using Machine Learning methods.

pg_plan_inspector pg_plan_inspector is being developed as a framework to monitor and improve the performance of PostgreSQL using Machine Learning meth

suzuki hironobu 198 Dec 27, 2022
Prometheus exporter for PostgreSQL

pgexporter pgexporter is a Prometheus exporter for PostgreSQL. pgexporter will connect to one or more PostgreSQL instances and let you monitor their o

null 19 Dec 22, 2022
PostgreSQL extension for pgexporter

pgexporter_ext pgexporter_ext is an extension for PostgreSQL to provide additional Prometheus metrics for pgexporter. Features Disk space metrics See

null 4 Apr 13, 2022
The PostgreSQL client API in modern C++

C++ client API to PostgreSQL {#mainpage} Dmitigr Pgfe (PostGres FrontEnd, hereinafter referred to as Pgfe) - is a C++ client API to PostgreSQL servers

Dmitry Igrishin 137 Dec 14, 2022
A friendly and lightweight C++ database library for MySQL, PostgreSQL, SQLite and ODBC.

QTL QTL is a C ++ library for accessing SQL databases and currently supports MySQL, SQLite, PostgreSQL and ODBC. QTL is a lightweight library that con

null 173 Dec 12, 2022
C++ client library for PostgreSQL

Welcome to taoPQ taoPQ is a lightweight C++ client library for accessing a PostgreSQL➚ database. It has no dependencies beyond libpq➚, the C applicati

The Art of C++ 232 Dec 22, 2022
recovery postgresql table data by update/delete/rollback/dropcolumn command

recovery postgresql table data by update/delete/rollback/dropcolumn command

RadonDB 6 Aug 4, 2022
pgagroal is a high-performance protocol-native connection pool for PostgreSQL.

pgagroal is a high-performance protocol-native connection pool for PostgreSQL.

Agroal 555 Dec 27, 2022
xxhash functions for PostgreSQL

pg_xxhash PostgreSQL ❤️ xxhash Tested with xxhash 0.8.1 and PostgreSQL 14.1 on Linux and macOS. Think twice before even considering to use it in any s

Igor Hatarist 6 Oct 27, 2022
Distributed PostgreSQL as an extension

What is Citus? Citus is a PostgreSQL extension that transforms Postgres into a distributed database—so you can achieve high performance at any scale.

Citus Data 7.7k Dec 30, 2022
High-performance time-series aggregation for PostgreSQL

PipelineDB has joined Confluent, read the blog post here. PipelineDB will not have new releases beyond 1.0.0, although critical bugs will still be fix

PipelineDB 2.5k Dec 26, 2022
upstream module that allows nginx to communicate directly with PostgreSQL database.

About ngx_postgres is an upstream module that allows nginx to communicate directly with PostgreSQL database. Configuration directives postgres_server

RekGRpth 1 Apr 29, 2022
Modern cryptography for PostgreSQL using libsodium.

pgsodium pgsodium is an encryption library extension for PostgreSQL using the libsodium library for high level cryptographic algorithms. pgsodium can

Michel Pelletier 386 Dec 23, 2022
Open Source Oracle Compatible PostgreSQL.

IvorySQL is advanced, fully featured, open source Oracle compatible PostgreSQL with a firm commitment to always remain 100% compatible and a Drop-in r

null 420 Dec 28, 2022