An eventing framework for building high performance and high scalability systems in C.

Overview

NOTE: THIS PROJECT HAS BEEN DEPRECATED AND IS NO LONGER ACTIVELY MAINTAINED

As of 2019-03-08, this project will no longer be maintained and will be archived. Thank you to all the contributors and users of libphenom for their work over the years.

If you are looking for a comparable library (albeit in C++) that has many of the same features and more, you should check out the actively-maintained folly library.

The rest of the README is preserved as-is for historical purposes:

libPhenom

libPhenom is an eventing framework for building high performance and high scalability systems in C

System Requirements

libPhenom is known to compile and pass its test suite on:

  • Linux systems with epoll
  • OS X
  • Illumos and Solaris style systems that have port_create(3C).

libPhenom has been known to compile and pass its test suite on these systems, but they have not been tried in a little while, so may require a little bit of TLC:

  • BSDish systems that have the kqueue(2) facility, including FreeBSD 9.1 and OpenBSD 5.2

libPhenom depends on:

  • pkg-config to find dependencies.
  • Concurrency Kit for its excellent concurrency primitives and key data structures. If you are on OS/X, you can brew install concurrencykit. On other platforms, you will You will need to install CK yourself as it is not widely distributed at this time. We expect to be able to find CK using pkg-config.
  • OpenSSL is required; this should be provided by your OS. You will need to install openssl-devel or libssl-dev or a similarly named package.
  • autoconf, automake and libtool are required to build libPhenom

libPhenom works best if built with GCC version 4.3 or later, but should be able to build with any C99 compiler.

Build Status

Facilities

  • Memory management with counters - record how much of which kinds of memory your application is using.
  • Jobs - decompose your application into portions of work and let the phenom scheduler manage getting them done
  • streaming I/O with buffers
  • Handy data structures (hash tables, lists, queues)
  • Variant data type to enable serialization and deserialization of JSON
  • A printf implementation with registerable object formatting

Goals

  • Balance ease of use with performance
  • Aim to be neutral wrt. your choice of threaded or event-based dispatch and work well with both.
  • Where possible, avoid contention points in our implementation so as to avoid limiting scalability with the number of cores in the system.

How to use these docs

If you're reading these on http://facebook.github.io/libphenom, simply start typing and the search box will suggest topics. You may select topics from the Topics menu or browse the header files via the Headers menu.

Getting it

You can obtain the sources from https://github.com/facebook/libphenom:

$ git clone https://github.com/facebook/libphenom.git

or grab a snapshot of master

Build

$ ./autogen.sh
$ ./configure
$ make
$ make check
$ sudo make install

Quick Start for using the library

You'll want to set up the main loop using something like this:

// Always include phenom/defs.h first to correctly set up the compilation env
#include "phenom/defs.h"
#include "phenom/configuration.h"
#include "phenom/job.h"
#include "phenom/log.h"
#include "phenom/sysutil.h"

int main(int argc, char **argv)
{
  // Must be called prior to calling any other phenom functions
  ph_library_init();
  // Optional config file for tuning internals
  ph_config_load_config_file("/path/to/my/config.json");
  // Enable the non-blocking IO manager
  ph_nbio_init(0);

  // Do stuff here to register client/server stuff.
  // This enables a very simple request/response console
  // that allows you to run diagnostic commands:
  // `echo memory | nc -UC /tmp/phenom-debug-console`
  // (on BSD systems, use `nc -Uc`!)
  // The code behind this is in
  // https://github.com/facebook/libphenom/blob/master/corelib/debug_console.c
  ph_debug_console_start("/tmp/phenom-debug-console");

  // Run
  ph_sched_run();

  return 0;
}

And compile it using something like this: (you need optimization enabled)

$ gcc -O2 main.c `pkg-config libphenom --cflags --libs`

(this will link against libphenom).

A more detailed example can be found at https://github.com/facebook/libphenom/blob/master/examples/echo.c

Want more inspiration? Take a look at the code in the test suite.

Status

We're still hacking and evolving this library, so there may be some rough edges. We're very open to feedback; check out the Contributing section below.

Getting Help

We've started an IRC channel for realtime conversations; join us in #phenom @ freenode

Contributing

If you're thinking of hacking on libPhenom we'd love to hear from you! Feel free to use the Github issue tracker and pull requests to discuss and submit code changes.

We (Facebook) have to ask for a "Contributor License Agreement" from someone who sends in a patch or code that we want to include in the codebase. This is a legal requirement; a similar situation applies to Apache and other ASF projects.

If we ask you to fill out a CLA we'll direct you to our online CLA page where you can complete it easily. We use the same form as the Apache CLA so that friction is minimal.

License

libPhenom is made available under the terms of the Apache License 2.0. See the LICENSE file that accompanies this distribution for the full text of the license.

Comments
  • Compilation fixes

    Compilation fixes

    I had to make the following changes to get libphenom to build on ubuntu 12.04. These changes seem reasonable to me, but let me know if there's a better way to do this.

    opened by brucespang 10
  • Compilation error, make check, dns.t

    Compilation error, make check, dns.t

    Hi, there is a compilation error:

    OS X 10.8.5 Installed Autotools (autoconf, automake, libtool) Installed c-ares (with macport)

    $ pkg-config --libs libcares
    -L/opt/local/lib -lcares
    
    $ ./autogen.sh
    $ ./configure
    $ make
    
    $ make check
    make  check-TESTS
    PASS: tests/counter.t
    PASS: tests/memory.t
    PASS: tests/timer.t
    PASS: tests/printf.t
    PASS: tests/iobasic.t
    PASS: tests/stream.t
    PASS: tests/tpool.t
    PASS: tests/string.t
    PASS: tests/hashtable.t
    PASS: tests/sockaddr.t
    ./test-driver: line 95: 16030 Segmentation fault: 11  "$@" > $log_file 2>&1
    FAIL: tests/dns.t
    PASS: tests/variant.t
    PASS: tests/bench/iopipes.t
    ============================================================================
    Testsuite summary for phenom 0.1
    ============================================================================
    # TOTAL: 13
    # PASS:  12
    # SKIP:  0
    # XFAIL: 0
    # FAIL:  1
    # XPASS: 0
    # ERROR: 0
    ============================================================================
    See ./test-suite.log
    ============================================================================
    make[2]: *** [test-suite.log] Error 1
    make[1]: *** [check-TESTS] Error 2
    make: *** [check-am] Error 2
    

    The content of the test-suite.log is:

    ==================================
       phenom 0.1: ./test-suite.log
    ==================================
    
    # TOTAL: 13
    # PASS:  12
    # SKIP:  0
    # XFAIL: 0
    # FAIL:  1
    # XPASS: 0
    # ERROR: 0
    
    .. contents:: :depth: 2
    
    FAIL: tests/dns.t
    =================
    
    #     Failed test (tests/dns.c:check_addrinfo_result() at line 46)
    

    Please help~

    opened by agreatfool 10
  • Add data pointer to listener struct.

    Add data pointer to listener struct.

    This allow us to preserve more state about the listener object when we handle an accept.

    For example, I have a module that handles http requests a bit like sinatra. I'd like to have multiple instances of this module, so that a request to / on two different ports can be handled two different ways. Without being able to store data in the listener object, there's no nice way to figure out how we should handle a given request.

    opened by brucespang 9
  • hashtable crash

    hashtable crash

    when i set 67108864 or more elements, the hashtalbe crash.

    2013-11-27 4 45 30 pm

    (gdb) p *ht $5 = {nelems = 67108864, table_size = 134217728, elem_size = 16, mask = 134217727, kdef = 0x7ffff7ff6ea0, vdef = 0x7ffff7ff68e0, table = 0x7ffe68e92020 ""}

    my init: reph = ph_ht_init(&(mth->ht), 10000, &ph_ht_string_key_def, &ph_ht_uint32_val_def);

    bug 
    opened by maczpc 7
  • Compile Failure

    Compile Failure

    Hey guys,

    Libphenom is currently refusing to compile on OS X 10.10, both with Clang and (GNU) gcc.

    Here's a gist of the build log.

    Using:

    OpenSSL 1.0.1j Concurrencykit 0.4.4 Automake 1.14.1 Autoconf 2.69 GNU Libtool 2.4.2 Pkg-config 0.28

    opened by DomT4 6
  • Make _XOPEN_SOURCE definition agree with _POSIX_C_SOURCE for POSIX 2008 env

    Make _XOPEN_SOURCE definition agree with _POSIX_C_SOURCE for POSIX 2008 env

    If we want a POSIX 2008 environment as requested by _POSIX_C_SOURCE here, _XOPEN_SOURCE should be 700 so they are in agreement.

    See: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap02.html#tag_02_01_04

    Fixes build on FreeBSD 10 which redefines _POSIX_C_SOURCE depending on _XOPEN_SOURCE def.

    opened by kev009 6
  • Update to ck 0.3.3 and introduce git subtree for ck vendoring

    Update to ck 0.3.3 and introduce git subtree for ck vendoring

    First commit contains the changes necessary for ck 0.3.3. Would appreciate some peer review here!

    • In upstream, ck_ring changed quite a bit. ck_ring_init() doesn't touch the buffer, and it was removed from ck_ring_t. Instead, a new opaque ck_ring_buffer_t must be separately initialized and is passed in to each enqueue() and dequeue(). https://github.com/sbahra/ck/commit/3edb523da5f6c9dcea41879ceb6643dcc7bde305

    The rest is a switch to using git subtree to vendorize ck. For a good overview, http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/. IMO, this is the least smelly way to vendor something, carrying the full repo is fine, and this gives the full power of git for maintenance of the vendor branch.

    opened by kev009 6
  • New Function ph_string_concatenate

    New Function ph_string_concatenate

    Why ph_string_concatenate function is not included in the libPhenom??

    ph_string_t *ph_string_concatenate(ph_string_t *a,ph_string_t *b) { uint32_t len1=strlen(a); uint32_t len2=strlen(b); uint32_t i;

      for(i=len1;i<len1+len2;i++)
     {
              a->buf[i]=b->buf[i];
     }
    
     return(a);
    

    }

    enhancement 
    opened by SakalyaDeshpande 6
  • iobasic.t segfault: os x 10.8.5 w/ clang 3.3

    iobasic.t segfault: os x 10.8.5 w/ clang 3.3

    Not able to reproduce this on every run, but here's a backtrace:

    Program exited normally.
    (gdb) run
    Starting program: /Users/jbergstroem/Work/libphenom/tests/iobasic.t
    1..13
    ok 1 - PH_OK == ph_nbio_init(0)
    ok 2 - PH_OK == ph_job_init(&pipe_job)
    ok 3 - 0 == ph_pipe(pipe_fd, PH_PIPE_NONBLOCK)
    ok 4 - set up ping
    ok 5 - 100ms resolution: diff=100
    ok 6 - 1 == read(pipe_fd[0], &buf, sizeof(buf))
    ok 7 - 100ms resolution: diff=101
    
    Program received signal EXC_BAD_ACCESS, Could not access memory.
    Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000004
    [Switching to process 21613 thread 0x10b]
    0x00000001000165f5 in ck_epoch_synchronize (global=<value temporarily unavailable, due to optimizations>, record=0x0) at thirdparty/ck/src/ck_epoch.c:395
    395     record->epoch = delta;
    (gdb) bt
    #0  0x00000001000165f5 in ck_epoch_synchronize (global=<value temporarily unavailable, due to optimizations>, record=0x0) at thirdparty/ck/src/ck_epoch.c:395
    #1  0x000000010001660e in ck_epoch_barrier (global=0x100020600, record=0x0) at thirdparty/ck/src/ck_epoch.c:403
    #2  0x000000010000d149 in destroy_thread (ptr=0x101200008) at corelib/thread.c:108
    #3  0x00000001001104b0 in _pthread_tsd_cleanup ()
    #4  0x00000001001099d2 in _pthread_exit ()
    #5  0x000000010010977d in _pthread_start ()
    #6  0x00000001000f61a1 in thread_start ()
    Current language:  auto; currently minimal
    
    bug 
    opened by jbergstroem 6
  • Unbundle Concurrency Kit

    Unbundle Concurrency Kit

    This removes the thirdparty/ck directory and uses pkg-config to specify the concurrency-kit dependency. It forces ck to be <= 0.3.2, as that's the last compatible release with the current libphenom.

    For more reference, see #46 and the comments in #59.

    opened by brucespang 5
  • Upgrade ck to 0.4.1

    Upgrade ck to 0.4.1

    I have a libphenom application that uses concurrency-kit extensively. It needs some things from ck that are not available in 0.3.0. This upgrades to ck-0.4.1 to make those things available.

    There were some changes to libphenom that were necessary due to breaking api changes in ck (https://github.com/sbahra/ck/commit/3edb523da5f6c9dcea41879ceb6643dcc7bde305). I've updated job.c and job.h to use the new api.

    The libphenom tests pass, and I haven't seen any issues in my application level testing.

    Some questions:

    1. Does libphenom have a better way to update ck than copying in the code for the new tag?
    2. Is there a preferred way to benchmark these changes to make sure they're not regressions?
    opened by brucespang 5
  • compile error with the latest libphenom and concurrencyKit

    compile error with the latest libphenom and concurrencyKit

    I got the following compile error when doing ./autogen.sh; ./configure; make. I did a git clone on this repo and the github repo on concurrencyKit.

    Any ideas? Thanks.

    libtool: compile:  gcc -DHAVE_CONFIG_H -Ithirdparty/ck/include -Iinclude -I. -DPHENOM_IMPL=1 -Werror -m64 -fno-omit-frame-pointer -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D__x86_64__ -I/usr/local/include -std=gnu1x -pedantic -Wall -Wextra -Wno-declaration-after-statement -W -Wundef -Wendif-labels -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wno-nested-externs -Winline -Wdisabled-optimization -fstrict-aliasing -pipe -Wno-parentheses -O2 -g -gdwarf-2 -MT corelib/libphenom_la-counter.lo -MD -MP -MF corelib/.deps/libphenom_la-counter.Tpo -c corelib/counter.c  -fPIC -DPIC -o corelib/.libs/libphenom_la-counter.o
    In file included from include/phenom/counter.h:81:0,
                     from corelib/counter.c:17:
    include/phenom/defs.h:368:41: error: static assertion failed: "counter_iterator_definition_bad"
     #  define ph_static_assert(expr, msg)   _Static_assert(expr, #msg)
                                             ^
    corelib/counter.c:170:1: note: in expansion of macro 'ph_static_assert'
     ph_static_assert(sizeof(struct ph_counter_scope_iterator)
    
    
    opened by teraPacket 1
  • Tag a (stable) release?

    Tag a (stable) release?

    Hello,

    It’d be very helpful if you could tag a release so we know the code there is stable and can be used. I work on Homebrew and we provide a way to install libphenom but we rely on the git HEAD so can’t provide checksums and can’t ensure users will always get the same code depending on the time they install it, making it harder for us to debug issues.

    Thanks!

    opened by bfontaine 0
  • tests/timer failures on FreeBSD

    tests/timer failures on FreeBSD

    I haven't dug into this yet. I'm not getting these consistently, sometime the run will be successful. The affinity failed message is always present ("nbio" and "job" in the message were added by me).

    1387793688.893 err: /1 nbio failed to set thread 0x80240e200 affinity
    1387793688.893 err: /2 nbio failed to set thread 0x80280e200 affinity
    1387793688.893 err: /3 nbio failed to set thread 0x802c0e200 affinity
    1387793688.894 err: /4 nbio failed to set thread 0x80300e200 affinity
    1387793688.894 err: /5 nbio failed to set thread 0x80340e200 affinity
    1387793688.894 err: /6 nbio failed to set thread 0x80380e200 affinity
    1387793688.894 err: /7 nbio failed to set thread 0x803c0e200 affinity
    1387793688.894 err: /8 nbio failed to set thread 0x80400e200 affinity
    1387793688.894 err: /9 nbio failed to set thread 0x80440e200 affinity
    1387793688.894 err: /10 nbio failed to set thread 0x80480e200 affinity
    1387793688.894 err: /11 nbio failed to set thread 0x804c0e200 affinity
    1387793688.895 err: dns/13 job failed to set thread 0x80540e200 affinity
    1387793688.895 err: dns/12 job failed to set thread 0x80500e200 affinity
    1387793688.893 err: phenom:sched/0 nbio failed to set thread 0x80209a200 affinity
    #     Failed test (tests/timer.c:record_tick() at line 38)
    #     Failed test (tests/timer.c:record_tick() at line 38)
    # Looks like you failed 2 tests of 8.
    # ELAPSED: 749ms
    
    1..8
    ok 1 - PH_OK == ph_nbio_init(0)
    ok 2 - PH_OK == ph_job_init(&timer)
    ok 3 - PH_OK == ph_job_set_timer_at(&timer, last_tick)
    ok 4 - 100ms resolution: diff=106
    ok 5 - 100ms resolution: diff=106
    not ok 6 - 100ms resolution: diff=213
    not ok 7 - 100ms resolution: diff=214
    ok 8 - PH_OK == ph_sched_run()
    
    do-not-reap 
    opened by kev009 12
  • profile and optimize find_record

    profile and optimize find_record

    Throwing this one out there for folks that want to dive in.

    In the bufq API we have ph_bufq_consume_record which searches the bufq for a matching record delimiter.

    For many internet protocols this delimiter is CRLF. For many other applications, we're likely to be looking for LF.

    There are a couple of optimizations that could be investigated.

    • Can we accelerate the memmem call using the sse3_memchr function from here? http://repnop.org/carte/snippets/simd/. The performance.data file indicates that this performs consistently better than the darwin libc. To adopt this, we'd need to detect sse3 either at runtime or compile time
    • are there specializations of sse3_memchr that can be made for detecting CRLF?
    • are there specializations of string matching algorithms with a constant, known needle that we could use?
    • is there a more efficient way to match needles across the "straddle" point in cases where the delimiter straddles discontiguous memory regions?
    • for long records, we make repeated calls and search across the same memory regions repeatedly. We can surely cache the last searched offset and improve efficiency. We'd need to make sure that we invalidate this offset in the appropriate circumstances (mostly when we've consumed past that point)
    enhancement 
    opened by wez 4
Owner
Meta Archive
These projects have been archived and are generally unsupported, but are still available to view and use
Meta Archive
The c++ micro framework for building web applications based on workflow

wfrest The c++ micro framework for building web applications based on workflow ⌛️ Build Step 1 : install workflow git clne [email protected]:sogou/workfl

Shiyu Yi 539 Jan 3, 2023
A framework for building native Windows apps with React.

React Native for Windows Build native Windows apps with React. See the official React Native website for an introduction to React Native. React Native

Microsoft 15.2k Jan 2, 2023
A framework for building Mobile cross-platform UI

Weex A framework for building Mobile cross-platform UI. Distribution Support Android 4.1 (API 16), iOS 9.0+ and WebKit 534.30+. platform status Androi

Alibaba 18k Jan 9, 2023
NavMeshComponents - High Level API Components for Runtime NavMesh Building

Status of the project Development This project is now developed as part of the AI Navigation package. Please add that package to your project in order

Unity Technologies 2.7k Jan 8, 2023
This is the Arduino® compatible port of the AIfES machine learning framework, developed and maintained by Fraunhofer Institute for Microelectronic Circuits and Systems.

AIfES for Arduino® AIfES (Artificial Intelligence for Embedded Systems) is a platform-independent and standalone AI software framework optimized for e

null 166 Jan 4, 2023
PHP Encoder, protect PHP scripts in PHP 8 and PHP 7, High Performance, Compitable with X86_64, MIPS, ARM platform and Ubuntu/Centos/OpenWRT system.

What's FRICC2? FRICC2 is a PHP Script encryption tool. When you are developing a commercial software using PHP, the script can be distributed as encry

Hoowa Sun 43 Dec 12, 2022
Visualization Library is a C++ middleware for high-performance 2D and 3D graphics applications based on OpenGL 1.x-4.x supporting Windows, Linux and Mac OS X.

Visualization Library 2.2 Gallery About Visualization Library is a C++ middleware for high-performance 2D and 3D graphics applications based on the in

Michele 313 Nov 8, 2022
A C++11 large integer library with effective high performance, simplistic in nature and also clean in the eyes.

BigIntegerCPP BigIntegerCPP is a C++11 port of large integer library used in CryptoLib4Pascal. It allows mostly parsing of numbers as strings in diffe

Telepati 26 Dec 22, 2022
Convenient, high-performance RGB color and position control for console output

Oof (omnipotent output friend) It's common for C++ programs to write output to the console. But consoles are far more capable than what they are usual

Sebastian Werhausen 776 Dec 27, 2022
Real Time, High performance BOT detection and protection

REAL-TIME BOT PROTECTION CHALLENGE IronFox https://innovera.ir IronFox is a real-time and high performance bot protection, that using Nginx as a reve

Khalegh Salehi 3 Jun 5, 2022
Emergency alert and tracer for realtime high-performance computing app (work in progress, currently supported env is only Linux x86-64).

HPC Emerg Emergency alert and tracer for realtime high-performance computing app (work in progress, currently supported env is only Linux x86-64). Exa

Ammar Faizi 7 Jan 19, 2022
Building and Executing Position Independent Shellcode from Object Files in Memory

PIC-Privileges Building and Executing Position Independent Shellcode from Object Files in Memory. This is a pingback to the blogpost I wrote at https:

Paranoid Ninja 108 Dec 26, 2022
Design files and resources for building a wireless N64 controller

wireless-n64-controller This project and its documentation is a Work-In-Progress. I'm still working on writing everything down and working out kinks i

Spencer Fraint 35 Dec 23, 2022
Repository for building and operating REVOLVER: An automatic protein purification system for gravity columns. Developed at the University of Toronto.

REVOLVER: An automated protein purification system This repository contains all the hardware and firmware files to build and operate REVOLVER: an auto

Laboratory for Metabolic Systems Engineering 3 Jun 14, 2022
MyOwnBricks - A library for building your own sensors and devices compatible with the modern LEGO PoweredUp system.

English version (See at the end for the French version) MyOwnBricks MyOwnBricks is a library for building your own sensors and devices compatible with

null 5 Sep 26, 2022
High performance C++11 signals

High performance C++11 signals See Performance of a C++11 Signal System for the original source code, as well as performance measurements compared to

Lars Pensjö 145 Nov 28, 2022
High Performance Streams Based on Coroutine TS ⚡

Conduit ⚡ Lazy High Performance Streams using Coroutine TS Conduit is a utility library for building and transforming, ranges and lazy (infinite) iter

LoopPerfect 144 Dec 3, 2022
high performance C++20 implementation of std::variant

A minimal compile-time overhead, C++20 implementation of std::variant. Fully standard conforming with a couple of documented differences.

null 37 Nov 22, 2022
Diff Match Patch is a high-performance library in multiple languages that manipulates plain text.

The Diff Match and Patch libraries offer robust algorithms to perform the operations required for synchronizing plain text. Diff: Compare two blocks o

Google 5.9k Dec 31, 2022