MinIO C++ Client SDK for Amazon S3 Compatible Cloud Storage

Overview

NOTE: This project is a work in progress

MinIO C++ Client SDK for Amazon S3 Compatible Cloud Storage Slack Sourcegraph Apache V2 License

The MinIO C++ Client SDK provides simple APIs to access any Amazon S3 compatible object storage.

This quickstart guide will show you how to install the MinIO client SDK, connect to MinIO, and provide a walkthrough for a simple file uploader. For a complete list of APIs and examples, please take a look at the C++ Client API Reference.

This document assumes that you have a working C++ development environment.

NOTE: This library is based on original work https://github.com/cjameshuff/s3tools, but has been completely re-implemented since then.

Build Instructions

In order to build this project, you need the Cross-Platform Make CMake 3.10 or higher. You can download it from http://www.cmake.org/. In order to build miniocpp you need to have the following libraries and their development headers installed.

  • libcurl-dev
  • libssl-dev (OpenSSL 1.1.x, preferably)
  • doxygen
    • dnf install doxygen -y on CentOS 8.x
    • apt install doxygen -y on Ubuntu 20.04
  • pugixml
    • dnf install pugixml-devel -y on CentOS 8.x
    • apt install libpugixml-dev -y on Ubuntu 20.04
git clone https://github.com/minio/minio-cpp
cd minio-cpp; git submodule init; git submodule update; mkdir build; cd build; cmake ../;
make

If building pugixml from source, then in the pugixml directory e.g. /src/pugixml-1.11

cmake .
make

Then you may point PUGIXML to point to custom folders.

git clone https://github.com/minio/minio-cpp
cd minio-cpp; git submodule init; git submodule update; mkdir build; cd build;
cmake -DPUGIXML_INCLUDE_DIR=/src/pugixml-1.11/src \
      -DCMAKE_PREFIX_PATH=/src/pugixml-1.11 ../;
make

Example code

#include <iostream>
#include <fstream>
#include <stdlib.h>
#include <getopt.h>
#include <s3.h>

using namespace Minio;

int
main ( int argc, char** argv )
{
  S3Client s3("https://play.min.io:9000", "minioadmin", "minioadmin");
  S3ClientIO io;
  s3.MakeBucket("newbucket", io);
  if(io.Failure()) {
    std::cerr << "ERROR: failed to create bucket" << endl;
    std::cerr << "response:\n" << io << endl;
    std::cerr << "response body:\n" << io.response.str() << endl;
    return -1;
  }
  return 0;
}

Run an example

Following example runs 'multipart' upload, uploads a single part. You would have to choose a local file to upload for -f, and also remote bucket to upload the object to as -n and final object name in the bucket as -k.

\ -n -k ">
export ACTION="multipart"
export ACCESS_KEY=minioadmin
export SECRET_KEY=minioadmin
export ENDPOINT="https://play.min.io:9000"

./examples/s3 -a ${ACTION} -f 
     
       \
              -n 
      
        -k 
       

       
      
     

Please choose a that exists.

License

This SDK is distributed under the Apache License, Version 2.0, see LICENSE for more information.

Comments
  • bugfix: after using curl_multi_fdset(), selcet() timeout minimum is recommended to be set 100ms, rather than NULL, which may cause current thread always waiting if 'maxfd' equal -1 on some platform.

    bugfix: after using curl_multi_fdset(), selcet() timeout minimum is recommended to be set 100ms, rather than NULL, which may cause current thread always waiting if 'maxfd' equal -1 on some platform.

    see https://github.com/curl/curl/blob/5ccddf64398c1186deb5769dac086d738e150e09/docs/examples/multi-legacy.c

    /On success the value of maxfd is guaranteed to be >= -1. We call select(maxfd + 1, ...); specially in case of (maxfd == -1) there are no fds ready yet so we call select(0, ...) --or Sleep() on Windows-- to sleep 100ms, which is the minimum suggested value in the curl_multi_fdset() doc./

    opened by yanxiuzi 8
  • SDK problems.

    SDK problems.

    Installed latest sdk on updated Ubuntu 18.04.6 LTS, 5.4.0-52-generic via vcpkg install minio-cpp. Compiling the example OK, but running it results in nothing and no error no matter what credentials, however - wrong url results in error. Populated storage according to https://github.com/minio/mc/releases without any issues. Any ideas?

    opened by cumafo 6
  • I have an SDK request problem

    I have an SDK request problem

    Hi,I have an SDK request problem,I use the following code, return “MakeBucket(): invalid response received; status code: 400; content-type: text/xml; charset=utf-8”,You know why?

    code: minio::s3::MakeBucketArgs args1; args1.bucket = "test"; minio::s3::MakeBucketResponse resp1 = client.MakeBucket(args1); if (!resp1) { throw std::runtime_error("MakeBucket(): " + resp1.Error().String()); }

    opened by xiaonaiquan 4
  • is vcpkg arm64-osx version outdated?

    is vcpkg arm64-osx version outdated?

    I tried to install miniocpp with vcpkg on my mac M1 and when I look at the include folder in miniocpp (~/vcpkg/installed/arm64-osx/include/miniocpp) I can only see the following:

    ├── s3.h ├── s3_headers.h ├── s3_http.h ├── s3_io.h ├── s3_signature_v2.h └── s3_types.h

    This doesn't match with what's on the repository. Is it because the version is outdated?

    opened by Clement-Jean 4
  • should reuse the http connetion in multi operation?

    should reuse the http connetion in multi operation?

    Dear, I see that minio::s3::Client create new http connections for each operation. That will consume much ports in high concurrency scenario.

    So, I want to reuse the http connection to avoid that problem. How can I do?

    many thanks.

    opened by coregiu 4
  • Build on ubuntu18.04

    Build on ubuntu18.04

    Unable to build due to: CMake Error at CMakeLists.txt:57 (find_package): Could not find a package configuration file provided by "unofficial-curlpp" Cannot find package on linux

    opened by cumafo 3
  • Whether the library can compile on the Windows platform?

    Whether the library can compile on the Windows platform?

    I compiled the library in Windows 10 vs 2019;

    Error message is displayed: “arpa/inet.h”: No such file or directory

    This file appears to be for the Linux platform, whether the library can be compiled on Windows

    opened by cumtb-edu-loo 1
  • Any plans for C bindings ?

    Any plans for C bindings ?

    Hello,

    I am reaching out in regards to support of C language, how viable is it in your opinion? Are there any plans of having C SDK of minio or at least a header wrapper for C that will be linked to CXX library via minio-cpp?

    Thank you !

    opened by clibequilibrium 1
  • PutObject error

    PutObject error

    https://github.com/minio/minio-cpp/blob/main/examples/PutObject.cc

    I refer to the above file and I get an error when I execute PutObject": server failed with HTTP status code 0

    opened by gocpplua 1
  • potential stack buffer overflow

    potential stack buffer overflow

    In http.h line 127 there is a sockaddr_in, which is an IPv4 sockaddr, but in the following line it gets filled with an IPv6 address, which needs more space than an IPv4 address It should be sockaddr_in6 for that purpose.

    opened by ticso24 1
  • adapt to windows (msvc) and use vcpkg for 3rd libs (include curlpp)

    adapt to windows (msvc) and use vcpkg for 3rd libs (include curlpp)

    Recommended to use vcpkg as a 3rd lib manager.

    1. adapt to windows msvc (2019+): add src/win/getopt.h
    2. use vcpkg instead of traditional find_package
    3. remove curlpp submodule, also use vcpkg
    opened by kkzi 1
  • segfault ListObjects if objects in bucket more then max_keys of 1000 keys

    segfault ListObjects if objects in bucket more then max_keys of 1000 keys

    just replace 'i < 3' to 'i < 1010' in tests/tests.cc:390 at method test.ListObjects and you get it: Program terminated with signal SIGSEGV, Segmentation fault.

    #0  0x00007f65929e9241 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
       from /lib/x86_64-linux-gnu/libstdc++.so.6
    (gdb) bt
    #0  0x00007f65929e9241 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
       from /lib/x86_64-linux-gnu/libstdc++.so.6
    #1  0x00007f65929e95fe in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
       from /lib/x86_64-linux-gnu/libstdc++.so.6
    #2  0x000055cf8340c963 in minio::s3::ListObjectsResult::Populate (this=0x7ffc43bd7000) at src/client.cc:38
    #3  0x000055cf8339c738 in minio::s3::ListObjectsResult::operator++ (this=0x7ffc43bd7000) at include/client.h:49
    #4  0x000055cf8339cced in minio::s3::ListObjectsResult::operator++ (this=0x7ffc43bd7000) at include/client.h:55
    #5  0x000055cf833a328d in Tests::ListObjects (this=0x7ffc43bd7800, quantity=1010) at tests/tests.cc:406
    #6  0x000055cf8339a08b in main (argc=1, argv=0x7ffc43bd7b18) at tests/tests.cc:724
    

    Trouble at

    sed -n "36,40p" src/client.cc

    36    args_->marker = resp_.next_marker;
    37  } else {
    38    args_->start_after = resp_.start_after; // <- trouble at this line
    39    args_->continuation_token = resp_.next_continuation_token;
    40  }
    

    The reason of trouble at src/client.cc:669:

    minio::s3::ListObjectsResult minio::s3::Client::ListObjects(
        ListObjectsArgs args) {
       if (error::Error err = args.Validate()) return err;
      return ListObjectsResult(this, args);
     }
    

    minio::s3::Client::ListObjects method accepts args as structure of ListObjectsArgs, then get its address and init ListObjectsResult with pointer to it. Then this pointer used again for each request at ListObjectsRsult.Populate() to continue list next part of objects at minio storage bucket. But argument args in constructor ListObjects is not the original object, it is a copy of ListObjectsArgs created on function stack and pointer to this copy of object is transit to ListObjectsResult. Copy of object is destroyed on return from client.ListObjects. After that ListOjbectsResult in Populate() has freed pointer and when uses it this leads to segfault.

    opened by AlexBurnes 0
Owner
Multi-Cloud Object Storage
Multi-Cloud Object Storage
A simple console client for pCloud cloud storage.

pCloud Console Client A simple console client for pCloud cloud storage. Project Information pCloud Console Client was forked from the project initiall

Serghei Iakovlev 24 Oct 28, 2022
Spacex Storage is an offchain storage work inspector of Mannheim Network running inside TEE enclave.

Spacex Storage Spacex Storage is an offchain storage work inspector of Mannheim Network running inside TEE enclave. Contribution Thank you for conside

Mannheim Network 10 Oct 21, 2022
Cloud Native Data Plane (CNDP) is a collection of user space libraries to accelerate packet processing for cloud applications.

CNDP - Cloud Native Data Plane Overview Cloud Native Data Plane (CNDP) is a collection of userspace libraries for accelerating packet processing for c

Cloud Native Data Plane 35 Dec 28, 2022
Sensory Cloud C++ SDK

Sensory Cloud C++ SDK This repository contains the source code for the Sensory Cloud C++ SDK. Requirements This project uses CMake as the primary buil

SensoryCloud 2 Jan 10, 2022
The Raspberry Pi Pico SDK (henceforth the SDK) provides the headers, libraries and build system necessary

The Raspberry Pi Pico SDK (henceforth the SDK) provides the headers, libraries and build system necessary to write programs for the RP2040-based devices such as the Raspberry Pi Pico in C, C++ or assembly language.

Raspberry Pi 1.9k Jan 5, 2023
The Gecko SDK (GSDK) combines all Silicon Labs 32-bit IoT product software development kits (SDKs) based on Gecko Platform into a single, integrated SDK.

Silicon Labs Gecko SDK (GSDK) The Gecko SDK (GSDK) combines Silicon Labs wireless software development kits (SDKs) and Gecko Platform into a single, i

Silicon Labs 163 Dec 28, 2022
Poseidon OS (POS) is a light-weight storage OS

Poseidon OS Poseidon OS (POS) is a light-weight storage OS that offers the best performance and valuable features over storage network. POS exploits t

null 59 Jan 5, 2023
XTAO Unified Distributed Storage

Anna - A branch project from CEPH Anna is a XTAO project branched from CEPH distributed storage. CEPH is a nice opensource project for unified distrib

XTAO Technolgy 3 Nov 12, 2021
An efficient, small mobile key-value storage framework developed by WeChat. Works on Android, iOS, macOS, Windows, and POSIX.

中文版本请参看这里 MMKV is an efficient, small, easy-to-use mobile key-value storage framework used in the WeChat application. It's currently available on Andr

Tencent 15.4k Jan 8, 2023
Modern transactional key-value/row storage library.

Sophia is advanced transactional MVCC key-value/row storage library. How does it differ from other storages? Sophia is RAM-Disk hybrid storage. It is

Dmitry Simonenko 1.8k Dec 15, 2022
The OpenEXR project provides the specification and reference implementation of the EXR file format, the professional-grade image storage format of the motion picture industry.

OpenEXR OpenEXR provides the specification and reference implementation of the EXR file format, the professional-grade image storage format of the mot

Academy Software Foundation 1.3k Jan 6, 2023
A demonstration PoC for CVE-2022-21877 (storage spaces controller memory leak)

POC CVE-2022-21877 This repository contains a POC for the CVE-2022-21877, found by Quang Linh, working at STAR Labs. This is an information leak found

null 4 Mar 8, 2022
PlotFS is a fuse filesystem for efficient storage of Chia plot files.

PlotFS PlotFS is a fuse filesystem for efficient storage of Chia plot files. PlotFS is not a traditional filesystem. It is mounted read only for farmi

Matthew Szatmary 49 Nov 8, 2022
C++ implementation for a binary data storage format.

bsmlib- A C++ library for loading and writing binary data to and from files. bsmlib provides functions for loading, modifying, and saving BSM (Binary

Colleen 4 Oct 9, 2022
This repository was created in order to keep local data with code in the cloud.

Airplane Ino Данный репозиторий был создан для совсместной комфортной работы над проектом. В данном файле(README.md) будет размещена основная полезная

surpri6e 0 Aug 11, 2022
This code converts a point cloud obtained by a Velodyne VLP16 3D-Lidar sensor into a depth image mono16.

pc2image This code converts a point cloud obtained by a Velodyne VLP16 3D-Lidar sensor into a depth image mono16. Requisites ROS Kinetic or Melodic Ve

Edison Velasco Sánchez 6 May 18, 2022
This repository uses a ROS node to subscribe to camera (hikvision) and lidar (livox) data. After the node merges the data, it publishes the colored point cloud and displays it in rviz.

fusion-lidar-camera-ROS 一、介绍 本仓库是一个ROS工作空间,其中ws_fusion_camera/src有一个工具包color_pc ws_fusion_camera │ README.md │ └───src │ └───package: c

hongyu wang 23 Dec 7, 2022
This project is used for lidar point cloud undistortion.

livox_cloud_undistortion This project is used for lidar point cloud undistortion. During the recording process, the lidar point cloud has naturally th

livox 74 Dec 20, 2022
Cloud-native high-performance edge/middle/service proxy

Cloud-native high-performance edge/middle/service proxy Envoy is hosted by the Cloud Native Computing Foundation (CNCF). If you are a company that wan

Envoy Proxy - CNCF 21.2k Jan 9, 2023