An eventing framework for building high performance and high scalability systems in C.

Overview

NOTE: THIS PROJECT HAS BEEN DEPRECATED AND IS NO LONGER ACTIVELY MAINTAINED

As of 2019-03-08, this project will no longer be maintained and will be archived. Thank you to all the contributors and users of libphenom for their work over the years.

If you are looking for a comparable library (albeit in C++) that has many of the same features and more, you should check out the actively-maintained folly library.

The rest of the README is preserved as-is for historical purposes:

libPhenom

libPhenom is an eventing framework for building high performance and high scalability systems in C

System Requirements

libPhenom is known to compile and pass its test suite on:

  • Linux systems with epoll
  • OS X
  • Illumos and Solaris style systems that have port_create(3C).

libPhenom has been known to compile and pass its test suite on these systems, but they have not been tried in a little while, so may require a little bit of TLC:

  • BSDish systems that have the kqueue(2) facility, including FreeBSD 9.1 and OpenBSD 5.2

libPhenom depends on:

  • pkg-config to find dependencies.
  • Concurrency Kit for its excellent concurrency primitives and key data structures. If you are on OS/X, you can brew install concurrencykit. On other platforms, you will You will need to install CK yourself as it is not widely distributed at this time. We expect to be able to find CK using pkg-config.
  • OpenSSL is required; this should be provided by your OS. You will need to install openssl-devel or libssl-dev or a similarly named package.
  • autoconf, automake and libtool are required to build libPhenom

libPhenom works best if built with GCC version 4.3 or later, but should be able to build with any C99 compiler.

Build Status

Facilities

  • Memory management with counters - record how much of which kinds of memory your application is using.
  • Jobs - decompose your application into portions of work and let the phenom scheduler manage getting them done
  • streaming I/O with buffers
  • Handy data structures (hash tables, lists, queues)
  • Variant data type to enable serialization and deserialization of JSON
  • A printf implementation with registerable object formatting

Goals

  • Balance ease of use with performance
  • Aim to be neutral wrt. your choice of threaded or event-based dispatch and work well with both.
  • Where possible, avoid contention points in our implementation so as to avoid limiting scalability with the number of cores in the system.

How to use these docs

If you're reading these on http://facebook.github.io/libphenom, simply start typing and the search box will suggest topics. You may select topics from the Topics menu or browse the header files via the Headers menu.

Getting it

You can obtain the sources from https://github.com/facebook/libphenom:

$ git clone https://github.com/facebook/libphenom.git

or grab a snapshot of master

Build

$ ./autogen.sh
$ ./configure
$ make
$ make check
$ sudo make install

Quick Start for using the library

You'll want to set up the main loop using something like this:

// Always include phenom/defs.h first to correctly set up the compilation env
#include "phenom/defs.h"
#include "phenom/configuration.h"
#include "phenom/job.h"
#include "phenom/log.h"
#include "phenom/sysutil.h"

int main(int argc, char **argv)
{
  // Must be called prior to calling any other phenom functions
  ph_library_init();
  // Optional config file for tuning internals
  ph_config_load_config_file("/path/to/my/config.json");
  // Enable the non-blocking IO manager
  ph_nbio_init(0);

  // Do stuff here to register client/server stuff.
  // This enables a very simple request/response console
  // that allows you to run diagnostic commands:
  // `echo memory | nc -UC /tmp/phenom-debug-console`
  // (on BSD systems, use `nc -Uc`!)
  // The code behind this is in
  // https://github.com/facebook/libphenom/blob/master/corelib/debug_console.c
  ph_debug_console_start("/tmp/phenom-debug-console");

  // Run
  ph_sched_run();

  return 0;
}

And compile it using something like this: (you need optimization enabled)

$ gcc -O2 main.c `pkg-config libphenom --cflags --libs`

(this will link against libphenom).

A more detailed example can be found at https://github.com/facebook/libphenom/blob/master/examples/echo.c

Want more inspiration? Take a look at the code in the test suite.

Status

We're still hacking and evolving this library, so there may be some rough edges. We're very open to feedback; check out the Contributing section below.

Getting Help

We've started an IRC channel for realtime conversations; join us in #phenom @ freenode

Contributing

If you're thinking of hacking on libPhenom we'd love to hear from you! Feel free to use the Github issue tracker and pull requests to discuss and submit code changes.

We (Facebook) have to ask for a "Contributor License Agreement" from someone who sends in a patch or code that we want to include in the codebase. This is a legal requirement; a similar situation applies to Apache and other ASF projects.

If we ask you to fill out a CLA we'll direct you to our online CLA page where you can complete it easily. We use the same form as the Apache CLA so that friction is minimal.

License

libPhenom is made available under the terms of the Apache License 2.0. See the LICENSE file that accompanies this distribution for the full text of the license.

Comments
  • Compilation fixes

    Compilation fixes

    I had to make the following changes to get libphenom to build on ubuntu 12.04. These changes seem reasonable to me, but let me know if there's a better way to do this.

    opened by brucespang 10
  • Compilation error, make check, dns.t

    Compilation error, make check, dns.t

    Hi, there is a compilation error:

    OS X 10.8.5 Installed Autotools (autoconf, automake, libtool) Installed c-ares (with macport)

    $ pkg-config --libs libcares
    -L/opt/local/lib -lcares
    
    $ ./autogen.sh
    $ ./configure
    $ make
    
    $ make check
    make  check-TESTS
    PASS: tests/counter.t
    PASS: tests/memory.t
    PASS: tests/timer.t
    PASS: tests/printf.t
    PASS: tests/iobasic.t
    PASS: tests/stream.t
    PASS: tests/tpool.t
    PASS: tests/string.t
    PASS: tests/hashtable.t
    PASS: tests/sockaddr.t
    ./test-driver: line 95: 16030 Segmentation fault: 11  "$@" > $log_file 2>&1
    FAIL: tests/dns.t
    PASS: tests/variant.t
    PASS: tests/bench/iopipes.t
    ============================================================================
    Testsuite summary for phenom 0.1
    ============================================================================
    # TOTAL: 13
    # PASS:  12
    # SKIP:  0
    # XFAIL: 0
    # FAIL:  1
    # XPASS: 0
    # ERROR: 0
    ============================================================================
    See ./test-suite.log
    ============================================================================
    make[2]: *** [test-suite.log] Error 1
    make[1]: *** [check-TESTS] Error 2
    make: *** [check-am] Error 2
    

    The content of the test-suite.log is:

    ==================================
       phenom 0.1: ./test-suite.log
    ==================================
    
    # TOTAL: 13
    # PASS:  12
    # SKIP:  0
    # XFAIL: 0
    # FAIL:  1
    # XPASS: 0
    # ERROR: 0
    
    .. contents:: :depth: 2
    
    FAIL: tests/dns.t
    =================
    
    #     Failed test (tests/dns.c:check_addrinfo_result() at line 46)
    

    Please help~

    opened by agreatfool 10
  • Add data pointer to listener struct.

    Add data pointer to listener struct.

    This allow us to preserve more state about the listener object when we handle an accept.

    For example, I have a module that handles http requests a bit like sinatra. I'd like to have multiple instances of this module, so that a request to / on two different ports can be handled two different ways. Without being able to store data in the listener object, there's no nice way to figure out how we should handle a given request.

    opened by brucespang 9
  • hashtable crash

    hashtable crash

    when i set 67108864 or more elements, the hashtalbe crash.

    2013-11-27 4 45 30 pm

    (gdb) p *ht $5 = {nelems = 67108864, table_size = 134217728, elem_size = 16, mask = 134217727, kdef = 0x7ffff7ff6ea0, vdef = 0x7ffff7ff68e0, table = 0x7ffe68e92020 ""}

    my init: reph = ph_ht_init(&(mth->ht), 10000, &ph_ht_string_key_def, &ph_ht_uint32_val_def);

    bug 
    opened by maczpc 7
  • Compile Failure

    Compile Failure

    Hey guys,

    Libphenom is currently refusing to compile on OS X 10.10, both with Clang and (GNU) gcc.

    Here's a gist of the build log.

    Using:

    OpenSSL 1.0.1j Concurrencykit 0.4.4 Automake 1.14.1 Autoconf 2.69 GNU Libtool 2.4.2 Pkg-config 0.28

    opened by DomT4 6
  • Make _XOPEN_SOURCE definition agree with _POSIX_C_SOURCE for POSIX 2008 env

    Make _XOPEN_SOURCE definition agree with _POSIX_C_SOURCE for POSIX 2008 env

    If we want a POSIX 2008 environment as requested by _POSIX_C_SOURCE here, _XOPEN_SOURCE should be 700 so they are in agreement.

    See: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap02.html#tag_02_01_04

    Fixes build on FreeBSD 10 which redefines _POSIX_C_SOURCE depending on _XOPEN_SOURCE def.

    opened by kev009 6
  • Update to ck 0.3.3 and introduce git subtree for ck vendoring

    Update to ck 0.3.3 and introduce git subtree for ck vendoring

    First commit contains the changes necessary for ck 0.3.3. Would appreciate some peer review here!

    • In upstream, ck_ring changed quite a bit. ck_ring_init() doesn't touch the buffer, and it was removed from ck_ring_t. Instead, a new opaque ck_ring_buffer_t must be separately initialized and is passed in to each enqueue() and dequeue(). https://github.com/sbahra/ck/commit/3edb523da5f6c9dcea41879ceb6643dcc7bde305

    The rest is a switch to using git subtree to vendorize ck. For a good overview, http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/. IMO, this is the least smelly way to vendor something, carrying the full repo is fine, and this gives the full power of git for maintenance of the vendor branch.

    opened by kev009 6
  • New Function ph_string_concatenate

    New Function ph_string_concatenate

    Why ph_string_concatenate function is not included in the libPhenom??

    ph_string_t *ph_string_concatenate(ph_string_t *a,ph_string_t *b) { uint32_t len1=strlen(a); uint32_t len2=strlen(b); uint32_t i;

      for(i=len1;i<len1+len2;i++)
     {
              a->buf[i]=b->buf[i];
     }
    
     return(a);
    

    }

    enhancement 
    opened by SakalyaDeshpande 6
  • iobasic.t segfault: os x 10.8.5 w/ clang 3.3

    iobasic.t segfault: os x 10.8.5 w/ clang 3.3

    Not able to reproduce this on every run, but here's a backtrace:

    Program exited normally.
    (gdb) run
    Starting program: /Users/jbergstroem/Work/libphenom/tests/iobasic.t
    1..13
    ok 1 - PH_OK == ph_nbio_init(0)
    ok 2 - PH_OK == ph_job_init(&pipe_job)
    ok 3 - 0 == ph_pipe(pipe_fd, PH_PIPE_NONBLOCK)
    ok 4 - set up ping
    ok 5 - 100ms resolution: diff=100
    ok 6 - 1 == read(pipe_fd[0], &buf, sizeof(buf))
    ok 7 - 100ms resolution: diff=101
    
    Program received signal EXC_BAD_ACCESS, Could not access memory.
    Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000004
    [Switching to process 21613 thread 0x10b]
    0x00000001000165f5 in ck_epoch_synchronize (global=<value temporarily unavailable, due to optimizations>, record=0x0) at thirdparty/ck/src/ck_epoch.c:395
    395     record->epoch = delta;
    (gdb) bt
    #0  0x00000001000165f5 in ck_epoch_synchronize (global=<value temporarily unavailable, due to optimizations>, record=0x0) at thirdparty/ck/src/ck_epoch.c:395
    #1  0x000000010001660e in ck_epoch_barrier (global=0x100020600, record=0x0) at thirdparty/ck/src/ck_epoch.c:403
    #2  0x000000010000d149 in destroy_thread (ptr=0x101200008) at corelib/thread.c:108
    #3  0x00000001001104b0 in _pthread_tsd_cleanup ()
    #4  0x00000001001099d2 in _pthread_exit ()
    #5  0x000000010010977d in _pthread_start ()
    #6  0x00000001000f61a1 in thread_start ()
    Current language:  auto; currently minimal
    
    bug 
    opened by jbergstroem 6
  • Unbundle Concurrency Kit

    Unbundle Concurrency Kit

    This removes the thirdparty/ck directory and uses pkg-config to specify the concurrency-kit dependency. It forces ck to be <= 0.3.2, as that's the last compatible release with the current libphenom.

    For more reference, see #46 and the comments in #59.

    opened by brucespang 5
  • Upgrade ck to 0.4.1

    Upgrade ck to 0.4.1

    I have a libphenom application that uses concurrency-kit extensively. It needs some things from ck that are not available in 0.3.0. This upgrades to ck-0.4.1 to make those things available.

    There were some changes to libphenom that were necessary due to breaking api changes in ck (https://github.com/sbahra/ck/commit/3edb523da5f6c9dcea41879ceb6643dcc7bde305). I've updated job.c and job.h to use the new api.

    The libphenom tests pass, and I haven't seen any issues in my application level testing.

    Some questions:

    1. Does libphenom have a better way to update ck than copying in the code for the new tag?
    2. Is there a preferred way to benchmark these changes to make sure they're not regressions?
    opened by brucespang 5
  • compile error with the latest libphenom and concurrencyKit

    compile error with the latest libphenom and concurrencyKit

    I got the following compile error when doing ./autogen.sh; ./configure; make. I did a git clone on this repo and the github repo on concurrencyKit.

    Any ideas? Thanks.

    libtool: compile:  gcc -DHAVE_CONFIG_H -Ithirdparty/ck/include -Iinclude -I. -DPHENOM_IMPL=1 -Werror -m64 -fno-omit-frame-pointer -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D__x86_64__ -I/usr/local/include -std=gnu1x -pedantic -Wall -Wextra -Wno-declaration-after-statement -W -Wundef -Wendif-labels -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wno-nested-externs -Winline -Wdisabled-optimization -fstrict-aliasing -pipe -Wno-parentheses -O2 -g -gdwarf-2 -MT corelib/libphenom_la-counter.lo -MD -MP -MF corelib/.deps/libphenom_la-counter.Tpo -c corelib/counter.c  -fPIC -DPIC -o corelib/.libs/libphenom_la-counter.o
    In file included from include/phenom/counter.h:81:0,
                     from corelib/counter.c:17:
    include/phenom/defs.h:368:41: error: static assertion failed: "counter_iterator_definition_bad"
     #  define ph_static_assert(expr, msg)   _Static_assert(expr, #msg)
                                             ^
    corelib/counter.c:170:1: note: in expansion of macro 'ph_static_assert'
     ph_static_assert(sizeof(struct ph_counter_scope_iterator)
    
    
    opened by teraPacket 1
  • Tag a (stable) release?

    Tag a (stable) release?

    Hello,

    It’d be very helpful if you could tag a release so we know the code there is stable and can be used. I work on Homebrew and we provide a way to install libphenom but we rely on the git HEAD so can’t provide checksums and can’t ensure users will always get the same code depending on the time they install it, making it harder for us to debug issues.

    Thanks!

    opened by bfontaine 0
  • tests/timer failures on FreeBSD

    tests/timer failures on FreeBSD

    I haven't dug into this yet. I'm not getting these consistently, sometime the run will be successful. The affinity failed message is always present ("nbio" and "job" in the message were added by me).

    1387793688.893 err: /1 nbio failed to set thread 0x80240e200 affinity
    1387793688.893 err: /2 nbio failed to set thread 0x80280e200 affinity
    1387793688.893 err: /3 nbio failed to set thread 0x802c0e200 affinity
    1387793688.894 err: /4 nbio failed to set thread 0x80300e200 affinity
    1387793688.894 err: /5 nbio failed to set thread 0x80340e200 affinity
    1387793688.894 err: /6 nbio failed to set thread 0x80380e200 affinity
    1387793688.894 err: /7 nbio failed to set thread 0x803c0e200 affinity
    1387793688.894 err: /8 nbio failed to set thread 0x80400e200 affinity
    1387793688.894 err: /9 nbio failed to set thread 0x80440e200 affinity
    1387793688.894 err: /10 nbio failed to set thread 0x80480e200 affinity
    1387793688.894 err: /11 nbio failed to set thread 0x804c0e200 affinity
    1387793688.895 err: dns/13 job failed to set thread 0x80540e200 affinity
    1387793688.895 err: dns/12 job failed to set thread 0x80500e200 affinity
    1387793688.893 err: phenom:sched/0 nbio failed to set thread 0x80209a200 affinity
    #     Failed test (tests/timer.c:record_tick() at line 38)
    #     Failed test (tests/timer.c:record_tick() at line 38)
    # Looks like you failed 2 tests of 8.
    # ELAPSED: 749ms
    
    1..8
    ok 1 - PH_OK == ph_nbio_init(0)
    ok 2 - PH_OK == ph_job_init(&timer)
    ok 3 - PH_OK == ph_job_set_timer_at(&timer, last_tick)
    ok 4 - 100ms resolution: diff=106
    ok 5 - 100ms resolution: diff=106
    not ok 6 - 100ms resolution: diff=213
    not ok 7 - 100ms resolution: diff=214
    ok 8 - PH_OK == ph_sched_run()
    
    do-not-reap 
    opened by kev009 12
  • profile and optimize find_record

    profile and optimize find_record

    Throwing this one out there for folks that want to dive in.

    In the bufq API we have ph_bufq_consume_record which searches the bufq for a matching record delimiter.

    For many internet protocols this delimiter is CRLF. For many other applications, we're likely to be looking for LF.

    There are a couple of optimizations that could be investigated.

    • Can we accelerate the memmem call using the sse3_memchr function from here? http://repnop.org/carte/snippets/simd/. The performance.data file indicates that this performs consistently better than the darwin libc. To adopt this, we'd need to detect sse3 either at runtime or compile time
    • are there specializations of sse3_memchr that can be made for detecting CRLF?
    • are there specializations of string matching algorithms with a constant, known needle that we could use?
    • is there a more efficient way to match needles across the "straddle" point in cases where the delimiter straddles discontiguous memory regions?
    • for long records, we make repeated calls and search across the same memory regions repeatedly. We can surely cache the last searched offset and improve efficiency. We'd need to make sure that we invalidate this offset in the appropriate circumstances (mostly when we've consumed past that point)
    enhancement 
    opened by wez 4
Owner
Facebook Archive
These projects have been archived and are generally unsupported, but are still available to view and use
Facebook Archive
C++14 evented IO libraries for high performance networking and media based applications

LibSourcey C++ Networking Evolved LibSourcey is a collection of cross platform C++14 modules and classes that provide developers with an arsenal for r

Sourcey 1.3k Dec 21, 2022
Kigs framework is a C++ modular multipurpose cross platform framework.

Kigs framework is a C++ modular multi-purpose cross-platform framework. It was used as a basis for many professionnal projects. The main goal was to b

null 74 Nov 28, 2022
JUCE is an open-source cross-platform C++ application framework for desktop and mobile applications, including VST, VST3, AU, AUv3, RTAS and AAX audio plug-ins.

JUCE is an open-source cross-platform C++ application framework used for rapidly developing high quality desktop and mobile applications, including VS

JUCE 4.7k Jan 1, 2023
Idle is an asynchronous and hot-reloadable C++ dynamic component framework

Idle is an asynchronous, hot-reloadable, and highly reactive dynamic component framework similar to OSGI that is: ?? Modular: Your program logic is en

Denis Blank 173 Dec 7, 2022
Framework for Enterprise Application Development in c++, HTTP1/HTTP2/HTTP3 compliant, Supports multiple server backends

The ffead-cpp Framework ffead-cpp is a web-framework, application framework, utilities all bundled into one. It also provides an embedded HTTP/Web-Soc

Sumeet Chhetri 540 Dec 19, 2022
🔥 bhook(aka ByteHook) is a PLT hook framework for Android app.

?? bhook(aka ByteHook) is a PLT hook framework for Android app. Most of ByteDance's Android apps use bhook as the PLT hook solution online.

Bytedance Inc. 1.4k Jan 8, 2023
PYNQ Framework for ANTSDR

PYNQ Framework for ANTSDR This project was inspired by PYNQ and PlutoSDR. There are already many SDR platforms based on ZYNQ and AD9361, so does ANTSD

null 21 Oct 20, 2022
KoanLogic 400 Dec 25, 2022
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 11.6k Jan 5, 2023
An open-source C++ library developed and used at Facebook.

Folly: Facebook Open-source Library What is folly? Folly (acronymed loosely after Facebook Open Source Library) is a library of C++14 components desig

Facebook 24k Jan 1, 2023
Functional Programming Library for C++. Write concise and readable C++ code.

FunctionalPlus helps you write concise and readable C++ code. Table of contents Introduction Usage examples Type deduction and useful error messages T

Tobias Hermann 1.7k Dec 29, 2022
Easy to use, header only, macro generated, generic and type-safe Data Structures in C

C Macro Collections Easy to use, header only, macro generated, generic and type-safe Data Structures in C. Table of Contents Installation Contributing

Leonardo Vencovsky 345 Jan 5, 2023
A collection of single-file C libraries. (generic containers, random number generation, argument parsing and other functionalities)

cauldron A collection of single-file C libraries and tools with the goal to be portable and modifiable. Libraries library description arena-allocator.

Camel Coder 40 Dec 29, 2022
C++ Parallel Computing and Asynchronous Networking Engine

As Sogou`s C++ server engine, Sogou C++ Workflow supports almost all back-end C++ online services of Sogou, including all search services, cloud input method,online advertisements, etc., handling more than 10 billion requests every day. This is an enterprise-level programming engine in light and elegant design which can satisfy most C++ back-end development requirements.

Sogou-inc 9.7k Dec 26, 2022
An eventing framework for building high performance and high scalability systems in C.

NOTE: THIS PROJECT HAS BEEN DEPRECATED AND IS NO LONGER ACTIVELY MAINTAINED As of 2019-03-08, this project will no longer be maintained and will be ar

Meta Archive 1.7k Dec 14, 2022
OceanBase is an enterprise distributed relational database with high availability, high performance, horizontal scalability, and compatibility with SQL standards.

What is OceanBase database OceanBase Database is a native distributed relational database. It is developed entirely by Alibaba and Ant Group. OceanBas

OceanBase 5.1k Jan 4, 2023
mpiFileUtils - File utilities designed for scalability and performance.

mpiFileUtils provides both a library called libmfu and a suite of MPI-based tools to manage large datasets, which may vary from large directory trees to large files.

High-Performance Computing 133 Jan 4, 2023
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large-scale graphs with billions of vertices (nodes) and trillions of edges, with milliseconds of latency. It delivers enterprise-grade high performance to simplify the most complex data sets imaginable into meaningful and useful information.

vesoft inc. 8.4k Jan 9, 2023
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large scale graphs with dozens of billions of vertices (nodes) and trillions of edges, with milliseconds of latency.

vesoft inc. 834 Dec 24, 2022
Concurrency Kit 2.1k Jan 4, 2023