Dependency manager library that supports decentralization

Overview

Arbiter Build Status

Arbiter is a cross-platform C library1 which implements the baseline functionality that should be expected of any dependency manager or package manager, without being coupled to any particular use case, so that many more specific tools can be built on top.

In other words, Arbiter does not prescribe any one user experience—it just tries to solve those backend concerns which are common to all dependency managers.

1 Note that Arbiter is actually implemented in C++14, but currently only exposes a plain C API to minimize surface area and maximize interoperability.

Motivation

Arbiter’s main contributors originally built Carthage, a dependency manager for iOS and macOS frameworks. Carthage introduced some novel ideas which aren’t commonly found in other dependency managers, like a focus on decentralization. Unfortunately, Carthage is fairly coupled to Apple’s development tools and process.

Arbiter was conceived, in part, to generalize the best of Carthage’s features for use by any dependency manager, on a variety of platforms.

There are also some baseline features that we believe any dependency manager should support in order to be taken seriously, including:

  • Dependency “freezing.” Different tools have different names for this concept, but the basic idea is the same: once dependencies have been resolved, all projects and the versions selected for each should be saved to disk so that anyone else collaborating on the parent project can reproduce the dependency checkouts exactly.
  • Automatic conflict resolution. If two projects in the dependency graph specify mutually exclusive requirements for a shared dependency, the dependency resolver should still be able to back up (e.g., to different versions of the two projects) and try another configuration that might result in success.
  • Safe uninstallation. The tool should understand when uninstalling one package would break another, and surface this information to the user.

Since there are many dependency managers and package managers which do not meet all of the above criteria, a generic library like Arbiter could be used to fill in the gaps.

Functionality

Some major features of Arbiter include:

Compliance with Semantic Versioning

Semantic Versioning, or SemVer, is a specification for what software version numbers mean, and how they should be used to convey compatibility (and the lack thereof).

Arbiter implements SemVer and incorporates it into its dependency resolution algorithm, so that complex versioning and compatibility logic does not have to be reinvented from scratch for each new tool.

Lazy, decentralized dependency resolution

Most package managers require a centralized server which has knowledge of all packages and versions in the system.

However, Arbiter resolves individual dependencies on demand, allowing them to be loaded from anywhere—even different places for different versions! This doesn’t preclude using a centralized server, but means that it is not a requirement.

Parallelizable dependency installation

Arbiter does not itself determine what “installing” a package means, but can provide information to the package manager about the installation process.

Specifically, Arbiter understands when one package must be installed before another, and conversely when certain packages have no implicit relationship to each other. The package manager can use this information to download and install multiple packages concurrently, potentially reducing wait times for the end user.

… and more to come

For a full list of planned features, check out our backlog. If you’d be interested in making any of these a reality, please consider contributing!

Documentation

The Arbiter API is extensively documented in header comments, from which we periodically generate Doxygen pages. For the public C API, look at headers under include/arbiter/ in the file list of the documentation.

Examples

This repository contains not-production-strength examples for demonstrating how the Arbiter API can be used to build different functionality.

To compile all included examples, run make examples.

For more information about individual examples, see the README in each folder. Of course, there are almost certainly other possible uses that we assuredly haven’t thought of or implemented, so this shouldn’t be taken as an exhaustive showcase!

Bindings

Because the functionality of Arbiter is exposed in a C interface, it’s easy to build bindings into other languages. Currently, Arbiter already has Swift bindings, with more planned!

To compile all included bindings, run make bindings.

If you’d like to implement your own bindings, please let us know about them in a GitHub issue, and we can include a link here in the README.

License

Arbiter is released under the MIT license.

I am providing code in this repository to you under an open source license. Because this is my personal repository, the license you receive to my code is from me and not from my employer (Facebook).

Comments
  • Use CMake with a Makefile wrapper, instead of vanilla Make

    Use CMake with a Makefile wrapper, instead of vanilla Make

    This makes good on #28 and #29.

    I removed some of the .PHONY targets from your Makefile. I don't really know what those do, either... 😅 If you have any links handy, that'd be great!

    TODO

    • [ ] Get make check passing
    • [ ] Decide which version of CMake to use/install for CI
    • [ ] Decide which version of CMake to set as the project minimum
    • [ ] Get CI builds working
    • [ ] Review for parity with previous Makefile
    • [x] Test that header changes cause rebuilds (#28)
    • [ ] Test that tests can be built in parallel (#29)
    opened by modocache 8
  • Expose full resolved graph information in public API, split

    Expose full resolved graph information in public API, split "depths" logic into Installer type

    #45 and #46 will be easier if users are able to reconstitute a fully resolved dependency graph and pass it into Arbiter, instead of just consuming them. To do that, we basically move all of the DependencyGraph API which was only in Resolver.cpp into the public interface, and move the previous "install-ordered graph" logic into a more properly named ArbiterResolvedDependencyInstaller type.

    To do:

    • [x] Update bindings
    opened by jspahrsummers 6
  • Allow requirements to carry a priority

    Allow requirements to carry a priority

    Priority is used to determine which requirements are eligible for intersection. Requirements in two different priority bands will never be intersected.

    This can be used to implement unpinned versions like a branch or commit hash, by creating a requirement which is higher priority (a lower numeric index) than the default.

    cc @mdiep

    opened by jspahrsummers 4
  • Addition of a build system generator

    Addition of a build system generator

    opened by elfring 3
  • Add support for requirements and selected versions without a semantic version

    Add support for requirements and selected versions without a semantic version

    This is useful to match upon and provide versions using only metadata, part of implementing "unpinned" versions (like picking a branch).

    cc @mdiep

    opened by jspahrsummers 2
  • Test that numeric version identifiers have lower precedence than non-numeric identifiers

    Test that numeric version identifiers have lower precedence than non-numeric identifiers

    From the SemVer spec (emphasis mine):

    … identifiers consisting of only digits are compared numerically and identifiers with letters or hyphens are compared lexically in ASCII sort order. Numeric identifiers always have lower precedence than non-numeric identifiers. A larger set of pre-release fields has a higher precedence than a smaller set, if all of the preceding identifiers are equal. Example: 1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-alpha.beta < 1.0.0-beta < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0.

    enhancement 
    opened by jspahrsummers 1
  • Add futures/promises which can fire a callback when satisfied

    Add futures/promises which can fire a callback when satisfied

    std::future composes so poorly that it's amazing it even made it into the standard. It basically requires that any user block while work is performed.

    We should write something similar which allows a callback to be invoked instead.

    enhancement 
    opened by jspahrsummers 1
  • Support installation manifests and uninstalling

    Support installation manifests and uninstalling

    For some kinds of clients, like package managers for system binaries, uninstalling a package is a very common use case. However, unless the package manager knows which other packages depend upon it, this can be a very unsafe operation.

    We should offer hooks for recording the complete dependency trees associated with each installation. The client would be expected to provide storage and lookup, but could avoid reinventing our dependency tracking as long as we hand over all the necessary information.

    enhancement 
    opened by jspahrsummers 1
  • Support using cached builds

    Support using cached builds

    Once building is part of the Arbiter API, we could offer some generalized hooks that support looking up and reusing cached builds (instead of repeating a build from scratch).

    For example, Carthage supports downloading pre-built frameworks uploaded with GitHub releases.

    We probably don't need to get that specific, but we can add a generic API that allows the Arbiter client to install a cached build before we actually trigger our build step.

    enhancement 
    opened by jspahrsummers 1
  • Build a pluggable build architecture

    Build a pluggable build architecture

    After dependencies are checked out, there will usually be a build step. This would be where compilation occurs (for clients where it's relevant), or post-processing like Babel, etc.

    The main advantages of building this into Arbiter are:

    • We already have information about the ordering of dependencies, and can apply this to the build (i.e., build all dependencies before the dependent)
    • We can interleave/parallelize building with the rest of dependency resolution, since there are points at which we know that certain dependencies are "locked in"
    enhancement 
    opened by jspahrsummers 1
  • Build a version requirement API

    Build a version requirement API

    For example, this API should support constraints such as "compatible with 1.2", and allow checking them against specific versions (e.g., 1.2.3, which should be compatible).

    We could use the Carthage API for inspiration.

    enhancement 
    opened by jspahrsummers 1
  • Automatically generate updated documentation on CI

    Automatically generate updated documentation on CI

    Travis should be able to automatically regenerate our Doxygen docs, which we then push… somehow. Might need a separate gh-pages branch for this purpose.

    enhancement help wanted 
    opened by jspahrsummers 0
  • Make smart pointer types const-safe

    Make smart pointer types const-safe

    Right now, we have a lot of occurrences of, e.g., std::shared_ptr<ArbiterRequirement> which could instead be std::shared_ptr<const ArbiterRequirement>.

    enhancement 
    opened by jspahrsummers 0
  • Project idea: Yarn backed by Arbiter

    Project idea: Yarn backed by Arbiter

    Yesterday, Facebook announced a new package manager for JavaScript called Yarn.

    Yarn is a collaboration between Facebook, Exponent, Google, and Tilde.

    It's early days but once the Node bindings (and underlying core lib) are more mature it could be worth trying this.

    question 
    opened by paulyoung 2
  • [WIP] New interfaces for better graph resolution

    [WIP] New interfaces for better graph resolution

    🚧 WORK IN PROGRESS 🚧

    To do:

    • [x] Refactor resolution algorithm to be iterative? (Might make backmarking, etc. easier)
    • [x] Refactor ArbiterResolvedDependencyGraph to be a simple value type, and not store requirements
    • [ ] Figure out what to do for branch/commit pins ("unversioned" requirements)
    • [ ] Restore incremental resolution functionality
    • [x] Restore statistics about dependency resolution

    Future work:

    • Backmarking/culprit variables
    • Offer a mode where we try dependencies with the most constrained first—this means that ordering isn't deterministic over time, which is why it should be optional, but it should be more efficient if it eliminates more of the search space
    opened by jspahrsummers 3
Owner
Arbiter
Dependency manager library that supports decentralization
Arbiter
🌱Light and powerful C++ web framework for highly scalable and resource-efficient web application. It's zero-dependency and easy-portable.

Oat++ News Hey, meet the new oatpp version 1.2.5! See the changelog for details. Check out the new oatpp ORM - read more here. Oat++ is a modern Web F

Oat++ 6k Jan 4, 2023
A simple tcp tunnel on c using sockets Right now it only supports linux systems

A simple tcp tunnel on c using sockets Right now it only supports linux systems build BY MAKE mkdir build make cd build ./tunnel.o <localport> <rem

notaweeb 8 Sep 20, 2021
a lightweight and performant multicast DNS (mDNS) reflector with modern design, supports zone based reflection and IPv6

mDNS Reflector mDNS Reflector (mdns-reflector) is a lightweight and performant multicast DNS (mDNS) reflector with a modern design. It reflects mDNS q

Yuxiang Zhu 90 Dec 10, 2022
A GlobalProtect VPN client (GUI) for Linux based on OpenConnect and built with Qt5, supports SAML auth mode.

A GlobalProtect VPN client (GUI) for Linux based on OpenConnect and built with Qt5, supports SAML auth mode.

Kevin Yue 603 Jan 2, 2023
It is a wireless temperature, pressure and humidity sensor, supports working in Zigbee networks.

It is a wireless temperature, pressure and humidity sensor, supports working in Zigbee networks. (Zigbee2mqtt open source project). Built on CC2530 chip (Zigbee), two modications: Ebyte E18-MS1PA2-PCB radio module with amplifier and Ebyte E18-MS1-PCB. Powered by the most common AAA batteries.

Andrew Lamchenko 40 Dec 30, 2022
Built a peer-to-peer group based file sharing system where users could share or download files from the groups they belonged to. Supports parallel downloading with multiple file chunks from multiple peers.

Mini-Torrent Built a peer-to-peer group based file sharing system where users could share or download files from the groups they belonged to. Supports

null 1 Nov 15, 2021
Graphical small-internet client for windows, linux, MacOS X and BSDs. Supports gemini, http, https, gopher, finger.

Graphical small-internet client for windows, linux, MacOS X and BSDs. Supports gemini, http, https, gopher, finger.

Felix Queißner 569 Dec 30, 2022
RPC based on C++ Workflow. Supports Baidu bRPC, Tencent tRPC, thrift protocols.

中文版入口 SRPC Introduction SRPC is an RPC system developed by Sogou. Its main features include: Base on Sogou C++ Workflow, with the following features:

C++ Workflow Project and Ecosystem 1.5k Jan 3, 2023
TLS 1.3 implementation in C (master supports RFC8446 as well as draft-26, -27, -28)

picotls Picotls is a TLS 1.3 (RFC 8446) protocol stack written in C, with the following features: support for three crypto engines "OpenSSL" backend u

H2O 442 Dec 28, 2022
We use Clash as the backend proxy, which supports Shadowsocks(R), V2Ray, and Trojan protocols.

We use Clash as the backend proxy, which supports Shadowsocks(R), V2Ray, and Trojan protocols.

Dr. Incognito 1.4k Dec 31, 2022
ESP8266 WiFi Connection manager with fallback web configuration portal

ESP8266 WiFi Connection manager with fallback web configuration portal

null 5.6k Jan 4, 2023
Linux Terminal Service Manager (LTSM) is a set of service programs that allows remote computers to connect to a Linux operating system computer using a remote terminal session (over VNC or RDP)

Linux Terminal Service Manager (LTSM) is a set of service programs that allows remote computers to connect to a Linux operating system computer using a remote terminal session (over VNC)

null 34 Dec 16, 2022
An SSH file manager that lets you edit files like they are local

An SSH file manager that lets you edit files like they are local

Allan Boll 505 Jan 10, 2023
Package manager for linux that installs windows apps through wine

Winepkg A package manager for linux that installs windows apps through wine. Usage winepkg -Si mspaint Install Dependencies: wine winetricks wget cur

null 2 May 27, 2022
Mongoose Embedded Web Server Library - a multi-protocol embedded networking library with TCP/UDP, HTTP, WebSocket, MQTT built-in protocols, async DNS resolver, and non-blocking API.

Mongoose - Embedded Web Server / Embedded Networking Library Mongoose is a networking library for C/C++. It implements event-driven non-blocking APIs

Cesanta Software 9k Jan 1, 2023
C Hypertext Library - A library for writing web applications in C

CHL C Hypertext Library - A library for writing web applications in C #include <chl/chl.h> int main() { chl_set_default_headers(); chl_print_header

null 271 Nov 14, 2022
The C++ Network Library Project -- cross-platform, standards compliant networking library.

C++ Network Library Modern C++ network programming libraries. Join us on Slack: http://slack.cpp-netlib.org/ Subscribe to the mailing list: https://gr

C++ Network Library 1.9k Dec 27, 2022
C++ peer to peer library, built on the top of boost

Breep What is Breep? Breep is a c++ bridged peer to peer library. What does that mean? It means that even though the network is constructed as a peer

Lucas Lazare 110 Nov 24, 2022
Cross-platform, efficient, customizable, and robust asynchronous HTTP/WebSocket server C++14 library with the right balance between performance and ease of use

What Is RESTinio? RESTinio is a header-only C++14 library that gives you an embedded HTTP/Websocket server. It is based on standalone version of ASIO

Stiffstream 924 Jan 6, 2023