Ingescape - Model-based framework for broker-free distributed software environments

Overview

Ingescape - Model-based framework for broker-free distributed software environments

Overview

Scope and Goals

Ownership and License

Dependencies with other open source projects

Building and installing Ingescape

Using Ingescape - a simple example (or two)

API Summary

Hints to contributors

Overview

Ingescape is a framework for the orchestration of broker-free distributed heterogeneous software (any language, any OS) running on different threads or processes on a computer, different computers on a local network, and different networks, even through the Internet. All these orchestrated distributed software are called Ingescape agents. An Ingescape platform is a set of agents meshed together and able to communicate in a fully decentralized way.

Ingescape brings a neat and easy model-based approach to describe and enforce the communications between agents. The model supports:

  • Data flow : Agents expose named and typed inputs and outputs. Each agent describes a mapping between its inputs and outputs of other agents resulting in a global data flow architecture. Ingescape data flow uses high-speed, low-latency PUB/SUB communications from outputs of  agents to inputs of other agents, with the optional capability to dispatch data between workers for workload distribution.

three agents mapped together into a platform

  • Request/reply services: Agents expose named services with named and typed parameters that are equivalent to REST-like web services on one hand and RPC/RMI solutions on the other hand. Any agent can both expose its own services and use services from other agents, without strict client/server segmentations.

services sequence

Because Ingescape is developed in C, it can be used out of the box in C++ and Objective-C and bindings can be created for practically any programming language on any operating system. At the moment,  official bindings exist for C#, Python, QML Javascript and NodeJS Javascript. Android Java, Oracle JRE Java and web-app Javascript are also supported with some peculiarities. If you need support for another language, please create an issue or - even better - submit your binding as a merge request!

Scope and Goals

Ingescape provides the concepts to build, implement and monitor modern heterogeneous distributed systems with interesting new capabilities. To do so, Ingescape merges the concepts of request/reply communications with the various popular producers/brokers/consumers network patterns, into a consistent, minimal set. The resulting expressivity enables Ingescape to handle 99.9% of the network-based communication patterns, going much further than any third-party technology taken individually. And if you need to go into the 0.1%, ZeroMQ is there as a backup for highly customized architectures.

At the same time, Ingescape remains open and makes it remarkably easy to interoperate with many other related technologies including Apache Kafka, RabbitMQ, MQTT, OpenDDS, ROS, etc.

Ingescape platforms are broker-free and fully decentralized. This means that the Ingescape agents in a distributed platform never rely on any kind of central entity. They are meshed together in a P2P manner and remain as loosely-coupled as possible. On an Ingescape platform, agents can be clients and/or servers, producers and/or consumers, all this at the same time. In this context, Kafka-like or MQTT-like brokers are irrelevant because producers and consumers communicate directly. Most of the time, agents discover one another automatically using UDP broadcast (zbeacons). Ingescape can also use  actual brokers (zgossip), i.e. go-betweens in charge only of putting agents in relation and not misnamed servers whose failure would make the system collapse.

Ingescape platforms are fully dynamic. Agents can add, edit or remove inputs, outputs and services at any time. Mappings between inputs and outputs can be added or removed at any time as well. Agents can come and go on a platform as often as they like. Multiple instances of agents are properly supported. In all cases, Ingescape manages the overall consistency automatically and each agent can subscribe to events to get informed and adapt to any of these changes if they need to. Data routing is handled automatically on-the-fly by the Ingescape library in each agent, based on the active mappings at a given time.

Ingescape is thread-safe with a low code footprint. In a given software process or application, the Ingescape API enables to create and control one or several agents from any number of threads. This means that existing code is very lightly affected by the introduction of Ingescape: the Ingescape code  integrates surgically with the existing one, independently from the architecture in place. For heavy data computation, Ingescape  integrates seamlessly in the most advanced parallel and multi-threaded architectures, whether to handle received data or to send to other agents.

Ingescape is multi-transport... And you have nothing to do to benefit from it! Based on the slick ZeroMQ multi-transport capabilities and Ingescape meshed network, each agent chooses automatically the most efficient transport to support data flows with each other agent. If the two agents are in the same process, shared memory is used. If the agents are on the same computer IPC (for UNIX boxes) or the Loopback (Windows) is used. And for the other cases, TCP is the preferred choice. With this ability, Ingescape agents minimize the latency, increase the bandwidth and optimize the network depending on its actual topology.

Ingescape is secure. The Ingescape library supports the ZeroMQ security layer using public/private certificates (SASL, Curve25519, 256 bits keys). This way, agents can validate their identity to others, platforms are joined by allowed agents only (i.e. agents will communicate only if they know each other's identity), and all communications are encrypted.

Other relevant characteristics:

  • Ingescape inputs, outputs and services support the following types: impulsions, bool/int/double numbers,  strings, raw binary data.
  • Raw binary data can be of any type and it is simple to use Ingescape in combination with Protobuf.
  • Agents definitions and mappings can be written to and read from an open JSON format support by the Ingescape library.
  • Ingescape comes with logging functions supporting the console, files and data streams to access the logs remotely in real-time.
  • Based on Zyre capabilities, Ingescape supports elections between competing agents, e.g. master/slave determination.

Ownership and License

The contributors are listed in AUTHORS. This project uses the MPL v2 license, see LICENSE. Ingescape uses the C4.1 (Collective Code Construction Contract) process for contributions. To report an issue, use the Ingescape issue tracker at github.com.

Dependencies with other open source projects

Ingescape relies on the following libraries:

  • libzmq as the core network library for the communication between agents using underlying ZeroMQ communication patterns,
  • czmq as a wrapper to the library and for its additional, extremely useful services,
  • zyre as the cornerstone for meshed networks with discovery mechanisms,
  • libsodium for everything crypto.

Ingescape also embeds:

  • A lightly modified version of the Yajl JSON parser and generator. See the corresponding source files for related ownership and licence.
  • Uthash for inline macros bringing lists and hash tables.  See the corresponding source files for related ownership and licence.

Building and installing Ingescape

Ingescape is designed primarily to be compiled and installed using cmake, which offers powerful means to generate projects for other types of IDEs or solutions. In addition, Ingescape offers pre-configured projects for the following targets:

  • Xcode
  • Qt Creator with both a PRI and a PRO projects
  • Visual studio

By commodity, Ingescape includes its dependencies as a set of git submodules. Please use the proper git command to pull them as well. The cmake compilation can either compile and install Ingescape and all its dependencies, or compile and install ingescape only, if you want to manage the dependencies externally or if they are already on your system.

Ingescape and dependencies

Dependencies are fetched from forks managed by the ingescape team on github and used for long-term industrial maintenance. This is the best way to have a full stack for debugging and contributing to ingescape, using your preferred environment.

git clone --recurse-submodules git://github.com/zeromq/ingescape.git
cd ingescape
mkdir build
cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DOSX_UNIVERSAL=ON \
-DCMAKE_OSX_DEPLOYMENT_TARGET=10.11 -DWITH_DEPS=ON
make -j8 -C build
sudo make -C build install
sudo ldconfig

Ingescape only

All the dependencies need to be fetched and installed from their official repositories using the latest master or stable tags. Because Ingescape uses draft APIs from CZMQ and Zyre, compilation might rarely break depending on their respective evolutions and the delay to make the necessary updates.

git clone --depth 1 -b stable https://github.com/jedisct1/libsodium.git
cd libsodium
./autogen.sh && ./configure && make check
make -j8
sudo make install
cd ..

git clone git://github.com/zeromq/libzmq.git
cd libzmq
mkdir build
cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DWITH_LIBSODIUM=ON -DENABLE_DRAFTS=ON -DWITH_TLS=OFF -DCMAKE_OSX_DEPLOYMENT_TARGET=10.11
make -j8 -C build
sudo make -C build install
sudo ldconfig
cd ..

git clone git://github.com/zeromq/czmq.git
cd czmq
mkdir build
cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DENABLE_DRAFTS=ON -DCMAKE_OSX_DEPLOYMENT_TARGET=10.11
make -j8 -C build
sudo make -C build install
sudo ldconfig
cd ..

git clone git://github.com/zeromq/zyre.git
cd zyre
mkdir build
cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DENABLE_DRAFTS=ON -DCMAKE_OSX_DEPLOYMENT_TARGET=10.11
make -j8 -C build
sudo make -C build install
sudo ldconfig
cd ..

git clone git://github.com/zeromq/ingescape.git
cd ingescape
mkdir build
cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_OSX_DEPLOYMENT_TARGET=10.11
make -j8 -C build
sudo make -C build install
sudo ldconfig
cd ..

Testing

Two test programs are built and installed with Ingescape. They are called igsTester and igsPartner and are installed in /usr/local/bin on *nix boxes. They enable various series of tests - static and dynamic - to check the library. They are optional if you are a user and not a contributor.

Static tests on most of the API

# adjust device and port depending on your computer
igsTester --device "en0" --port 5670 --static

These tests call almost all functions in the API and verify their results with assertions. If the program runs properly (despite a lot of expected error messages in the console) and terminates by returning 0, it means that everything went OK.

Dynamic tests

In a console, run:

igsPartner --device "en0" --port 5670 --verbose --auto

In another console, run:

igsTester --device "en0" --port 5670 --verbose --auto

Both agents discover each other and run tests automatically. If the two programs run properly (despite a lot of expected error messages in the console) and both terminate by returning 0, it means that everything went OK.

Interactive tests

The two agents also provide advanced console commands to test security, brokers, additional agents in the process, etc.

The command lines to enable interactive testing are the following:

igsPartner --device "en0" --port 5670 --verbose --interactiveloop
igsTester --device "en0" --port 5670 --verbose --interactiveloop

#in each console, then type /help to display the available console commands
/help
Available commands in the terminal:
	/publish : runs the iop publication tests
	/services : runs the service tests
	/channels : runs the channels tests
	/editor agent_uuid : runs the editor (i.e. private bus API) tests on a specific agent
	/activate : activates secondAgent
	/deactivate : deactivates secondAgent
	/service_local : firstAgent calls service secondCall on secondAgent
	/gossip : restart in gossip mode
	/security : restart and enable security in gossip mode (edit code to use self-discovery instead of gossip)
	/quit : quits the agent
	/help : displays this message

Helper scripts

Two scripts are provided at the root of the repo:

  • bootstrap_sysroot.sh is run without any parameter. It creates a sysroot directory at the same level as the repo dir and compiles and installs the library and its dependencies inside it. This script also creates the xcode projects for the dependencies, which are necessary for the main xcode project located in builds/xcode and specific to each repo location. If you are not using macos, just ignore the created xcode dirs.
  • install_on_system.sh is a simple commodity script to install Ingescape and its dependencies on your system. This should work on any *nix and windows boxes using cmake.

Using Ingescape - a simple example (or two)

To illustrate the use of mappings between inputs/outputs and the use of services, we build bogus examples for two agents.

Modeling the agents

ShapeRecognizer receives image data on its input and provides a corresponding string with a recognized shape on its output. ShapeRecognizer exposes a getShapeStatistics service expecting a shapeName (string) and a fromDate (integer) as parameters. The getShapeStatistics answers with a receiveShapeStatistics call to the caller, providing a shapeName (string) and a percentage parameters.

ImagesProvider receives a string on its shape input and sends bogus image data on its output every second. Every five seconds, it calls the getShapeStatistics service on ShapeRecognizer and exposes a receiveShapeStatistics service to get the answers.

Both agents are coded to map their input to each other's output. It is illustrated this way :

example mapping

Services are designed to communicate this way:

example services sequence

The code and compilation

Here is the code for the ShapeRecognizer agent to be copied into a main.c file:

c); igs_info(" from date : %d", firstArgument->next->i); //composing bogus statistics and sending them igs_service_arg_t *list = NULL; igs_service_args_add_string(&list, firstArgument->c); igs_service_args_add_int(&list, 65); igs_service_call(senderAgentUUID, "receiveShapeStatistics", &list, token); } int main(int argc, const char * argv[]) { int nb = 0; char **devices = igs_net_devices_list(&nb); printf("list of network devices for your computer:\n"); for (int i = 0; i < nb; i++) printf(" %s\n", devices[i]); printf("Use one of them in igs_start_with_device.\n"); igs_free_net_devices_list(devices, nb); igs_log_set_console(true); igs_log_set_file(true, NULL); igs_log_set_stream(true); igs_agent_set_name("ShapeRecognizer"); igs_definition_set_version("1.0"); igs_input_create("image", IGS_DATA_T, 0, 0); igs_observe_input("image", imageCallback, NULL); igs_mapping_add("image", "ImagesProvider", "image"); //map our image input to the ImagesProvider agent igs_output_create("recognizedShape", IGS_STRING_T, 0, 0); igs_service_init("getShapeStatistics", shapeStatisticsFunction, NULL); igs_service_arg_add("getShapeStatistics","shapeName", IGS_STRING_T); igs_service_arg_add("getShapeStatistics","fromDate", IGS_INTEGER_T); igs_start_with_device("en0", 5670); //customize the network device based on your computer getchar(); //returns when entering any character in the console igs_stop(); return 0; } ">
//
//  main.c
//  ShapeRecognizer version 1.0
//  Created by Stéphane Valès on 2021/07/23
//
//  no description
//

#ifdef _WIN32
#define WIN32_LEAN_AND_MEAN
#define NOMINMAX
#include 
   
    
#include 
    
     
#endif

#include 
     
      

void imageCallback(igs_iop_type_t iopType, const char* name, igs_iop_value_type_t valueType,
                   void* value, size_t valueSize, void* myData){
    igs_info("%s received (%zu bytes)", name, valueSize);
    //doing some sophisticated AI stuff here...
    igs_output_set_string("recognizedShape", "Rectangle");
}

void shapeStatisticsFunction(const char *senderAgentName, const char *senderAgentUUID,
                             const char *callName, igs_service_arg_t *firstArgument, size_t nbArgs,
                             const char *token, void* myData){
    igs_info("%s(%s) called %s with parameters:", senderAgentName, senderAgentUUID, callName);
    igs_info("    shapeName : %s", firstArgument->c);
    igs_info("    from date : %d", firstArgument->next->i);

    //composing bogus statistics and sending them
    igs_service_arg_t *list = NULL;
    igs_service_args_add_string(&list, firstArgument->c);
    igs_service_args_add_int(&list, 65);
    igs_service_call(senderAgentUUID, "receiveShapeStatistics", &list, token);
}

int main(int argc, const char * argv[]) {
    int nb = 0;
    char **devices = igs_net_devices_list(&nb);
    printf("list of network devices for your computer:\n");
    for (int i = 0; i < nb; i++)
        printf(" %s\n", devices[i]);
    printf("Use one of them in igs_start_with_device.\n");
    igs_free_net_devices_list(devices, nb);

    igs_log_set_console(true);
    igs_log_set_file(true, NULL);
    igs_log_set_stream(true);
    igs_agent_set_name("ShapeRecognizer");

    igs_definition_set_version("1.0");

    igs_input_create("image", IGS_DATA_T, 0, 0);
    igs_observe_input("image", imageCallback, NULL);
    igs_mapping_add("image", "ImagesProvider", "image"); //map our image input to the ImagesProvider agent

    igs_output_create("recognizedShape", IGS_STRING_T, 0, 0);

    igs_service_init("getShapeStatistics", shapeStatisticsFunction, NULL);
    igs_service_arg_add("getShapeStatistics","shapeName", IGS_STRING_T);
    igs_service_arg_add("getShapeStatistics","fromDate", IGS_INTEGER_T);

    igs_start_with_device("en0", 5670); //customize the network device based on your computer

    getchar(); //returns when entering any character in the console

    igs_stop();
    return 0;
}

     
    
   

And here is how to compile it in a minimal way using gcc:

gcc -Wall -g -I/usr/local/include/ -std=gnu99 -o main.o -c main.c
gcc -o ShapeRecognizer main.o -L/usr/local/lib -lingescape
./ShapeRecognizer

The compilation requires  linking against the Ingescape library only.

Here is the code for the ImagesProvider agent to be copied into another main.c file:

c); igs_info(" percentage : %d", firstArgument->next->i); } int publishImage(zloop_t *loop, int timer_id, void *arg){ void *image = malloc(128); //create bogus data igs_output_set_data("image", image, 128); free(image); igs_info("publish image data"); return 0; } int askStatistics(zloop_t *loop, int timer_id, void *arg){ igs_service_arg_t *list = NULL; igs_service_args_add_string(&list, "Rectangle"); igs_service_args_add_int(&list, 1627033669); igs_service_call("ShapeRecognizer", "getShapeStatistics", &list, "bogus_query_id"); igs_info("ask ShapeRecognizer for statistics using getShapeStatistics service"); return 0; } int main(int argc, const char * argv[]) { int nb = 0; char **devices = igs_net_devices_list(&nb); printf("list of network devices for your computer:\n"); for (int i = 0; i < nb; i++) printf(" %s\n", devices[i]); printf("Use one of them in igs_start_with_device.\n"); igs_free_net_devices_list(devices, nb); igs_log_set_console(true); igs_log_set_file(true, NULL); igs_log_set_stream(true); igs_agent_set_name("ImagesProvider"); igs_definition_set_version("1.0"); igs_input_create("shape", IGS_STRING_T, 0, 0); igs_observe_input("shape", shapeCallback, NULL); igs_mapping_add("shape", "ShapeRecognizer", "recognizedShape"); igs_output_create("image", IGS_DATA_T, 0, 0); igs_service_init("receiveShapeStatistics", shapeStatisticsFunction, NULL); igs_service_arg_add("receiveShapeStatistics","shapeName", IGS_STRING_T); igs_service_arg_add("receiveShapeStatistics","percentage", IGS_INTEGER_T); igs_start_with_device("en0", 5670); //customize the network device based on your computer zloop_t *loop = zloop_new(); zloop_timer(loop, 1000, 0, publishImage, NULL); zloop_timer(loop, 5000, 0, askStatistics, NULL); zloop_start(loop); //will terminate on interruption or Ctrl+C igs_stop(); return 0; } ">
//
//  main.c
//  ImagesProvider version 1.0
//  Created by Stéphane Valès on 2021/07/23
//
//  no description
//

#ifdef _WIN32
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
#endif
#define NOMINMAX
#include 
   
    
#include 
    
     
#endif

#include 
     
      

void shapeCallback(igs_iop_type_t iopType, const char* name, igs_iop_value_type_t valueType,
                   void* value, size_t valueSize, void* myData){
    igs_info("%s recognized as %s", name, (char*)value);
}

void shapeStatisticsFunction(const char *senderAgentName, const char *senderAgentUUID,
                             const char *callName, igs_service_arg_t *firstArgument, size_t nbArgs,
                             const char *token, void* myData){
    igs_info("%s(%s) called %s with parameters:", senderAgentName, senderAgentUUID, callName);
    igs_info("    shapeName : %s", firstArgument->c);
    igs_info("    percentage : %d", firstArgument->next->i);
}

int publishImage(zloop_t *loop, int timer_id, void *arg){
    void *image = malloc(128); //create bogus data
    igs_output_set_data("image", image, 128);
    free(image);
    igs_info("publish image data");
    return 0;
}

int askStatistics(zloop_t *loop, int timer_id, void *arg){
    igs_service_arg_t *list = NULL;
    igs_service_args_add_string(&list, "Rectangle");
    igs_service_args_add_int(&list, 1627033669);
    igs_service_call("ShapeRecognizer", "getShapeStatistics", &list, "bogus_query_id");
    igs_info("ask ShapeRecognizer for statistics using getShapeStatistics service");
    return 0;
}

int main(int argc, const char * argv[]) {
    int nb = 0;
    char **devices = igs_net_devices_list(&nb);
    printf("list of network devices for your computer:\n");
    for (int i = 0; i < nb; i++)
        printf(" %s\n", devices[i]);
    printf("Use one of them in igs_start_with_device.\n");
    igs_free_net_devices_list(devices, nb);

    igs_log_set_console(true);
    igs_log_set_file(true, NULL);
    igs_log_set_stream(true);
    igs_agent_set_name("ImagesProvider");

    igs_definition_set_version("1.0");

    igs_input_create("shape", IGS_STRING_T, 0, 0);
    igs_observe_input("shape", shapeCallback, NULL);
    igs_mapping_add("shape", "ShapeRecognizer", "recognizedShape");

    igs_output_create("image", IGS_DATA_T, 0, 0);

    igs_service_init("receiveShapeStatistics", shapeStatisticsFunction, NULL);
    igs_service_arg_add("receiveShapeStatistics","shapeName", IGS_STRING_T);
    igs_service_arg_add("receiveShapeStatistics","percentage", IGS_INTEGER_T);

    igs_start_with_device("en0", 5670); //customize the network device based on your computer

    zloop_t *loop = zloop_new();
    zloop_timer(loop, 1000, 0, publishImage, NULL);
    zloop_timer(loop, 5000, 0, askStatistics, NULL);
    zloop_start(loop); //will terminate on interruption or Ctrl+C

    igs_stop();
    return 0;
}

     
    
   

And here is how to compile it in a minimal way using gcc:

gcc -Wall -g -I/usr/local/include/ -std=gnu99 -o main.o -c main.c
gcc -o ImagesProvider main.o -L/usr/local/lib -lingescape -lczmq
./ImagesProvider

Because with use a CZMQ zloop in our example, the compilation requires linking against the Ingescape and CZMQ libraries.

You may be surprised by the almost complete absence of network code in these examples. This is the main objective of Ingescape: focusing on the exchanging data rather than on managing network connections. For the two agents to discover each other and establish communication, there is only one line of code that matters:

igs_start_with_device("en0", 5670); //customize the network device based on your computer

This function starts an agent on the network, using a network device on your computer and a network port that shall be free from any TCP binding. The list of network devices for your computer is displayed by this code:

int nb = 0;
char **devices = igs_net_devices_list(&nb);
printf("list of network devices for your computer:\n");
for (int i = 0; i < nb; i++)
	printf(" %s\n", devices[i]);
printf("Use one of them in igs_start_with_device.\n");
igs_free_net_devices_list(devices, nb);

Depending on your computer and operating system, network device names may be very different. On Windows, you may expect things like "Ethernet 2", "Wi-Fi" or "Loopback". On UNIX systems, expect things like "eth0" or "en0". If the list is empty, this means that no active network device exists on your computer and that you should enable WiFi or plug an ethernet cable. Between two agents on the same computer, use the same device and port. Between two agents on two different computers, use the same port and devices connected to the same local network. When this is done, the two agents discover and connect to each other automatically.

When the two agents are finally running on the same port, on the same network device or same local network, images published by ImagesProvider on its image output are received by ShapeRecognizer on its image input. When this happens, ShapeRecognizer publishes a recognized shape on its recognizedShape output, that is finally received by ImagesProvider on its shape input. In the same way, getShapeStatistics and receiveShapeStatistics services answer one to the other. All communications (output publication and service call) are initiated by ImagesProvider based on the two zloop timers.

Communications can be checked in the console or in the log files that are create in ~/Documents/Ingescape/logs.

Same example using external models as resources

The Ingescape library supports models for the definitions of the agents and their mapping inside a given platform. There are two ways to declare a model:

  • Using the API to hardcode it
  • Loading a JSON resource file representing the model

Here is the JSON definition for ShapeRecognizer:

{
    "definition": {
        "name": "ShapeRecognizer",
        "version": "1.0",
        "inputs": [
            {
                "name": "image",
                "type": "DATA"
            }
        ],
        "outputs": [
            {
                "name": "recognizedShape",
                "type": "STRING",
                "value": ""
            }
        ],
        "parameters": [

        ],
        "services": [
            {
                "name": "getShapeStatistics",
                "arguments": [
                    {
                        "name": "shapeName",
                        "type": "STRING"
                    },
                    {
                        "name": "fromDate",
                        "type": "INTEGER"
                    }
                ]
            }
        ]
    }
}

And here is the JSON mapping for ShapeRecognizer:

{
    "mappings": [
        {
            "fromInput": "image",
            "toAgent": "ImagesProvider",
            "toOutput": "image"
        }
    ],
    "splits": [

    ]
}

Instead of using the API to hardcode the definition and the mapping, the two JSON files can be loaded as external resources. The modified code in the main function is as follows:

int main(int argc, const char * argv[]) {
	int nb = 0;
	char **devices = igs_net_devices_list(&nb);
	printf("list of network devices for your computer:\n");
	for (int i = 0; i < nb; i++)
	    printf(" %s\n", devices[i]);
	printf("Use one of them in igs_start_with_device.\n");
	igs_free_net_devices_list(devices, nb);

	igs_log_set_console(true);
	igs_log_set_file(true, NULL);
	igs_log_set_stream(true);
	igs_definition_load_file("ShapeRecognizer.json");
	igs_mapping_load_file("ShapeRecognizer_mapping.json");

	igs_observe_input("image", imageCallback, NULL);

	igs_service_init("getShapeStatistics", shapeStatisticsFunction, NULL);

	igs_start_with_device("en0", 5670); //customize the network device based on your computer

	getchar(); //returns when entering any character in the console

	igs_stop();
	return 0;
}

All the functions used to create inputs, outputs, services and their arguments become useless when loading a model. The only remaining functions are the ones enabling to observe inputs and link services with the actual code. The models also cover the creation of mapping entries, avoiding specific code for this.

API Summary

The Ingescape library comes with two headers. The ingescape.h header is dedicated to the global functions and the API to use when you only have one agent in your process or application. The second header, igsagent.h, provides all the specific functions dedicated to several agents inside the same process. Basically, igsagent.h provides a subset of the ingescape.h functions with a different prefix and an agent pseudo-object as first parameter, plus new/destroy/activate/deactivate specific capabilities.

The global Ingescape API is centralized in a single header file. The header is heavily commented, with examples when it is relevant. Here is the structure of ingescape.h:

  • Agent initialization, control and events
  • Editing & inspecting definitions, adding and removing inputs/outputs/parameters
  • Reading and writing inputs/outputs/parameters
  • Mappings edition & inspection
  • Services edition & inspection
  • Timers
  • Communicating via channels
  • Brokers (when self-discovery is not possible)
  • Security
  • Elections and leadership between agents
  • Administration, logging, configuration and utilities
    • Logging functions
    • Version, command line, logging parameters, definition and mapping path, IPC transport configuration
    • Advanced network configuration and monitoring
  • JSON parsing and generation

Hints to contributors

Ingescape has built an architecture and data exchange model above some amazing ZeroMQ technologies. Please keep that in mind to decide where to contribute between CZMQ, Zyre and Ingescape.

The Ingescape API does not fully comply with the CLASS style guide yet but progressively get closer. We already share its objectives of simplicity and minimalism. Do read your code after you write it and ask, "Can I make this simpler?"

Comments
  • Fix default agent created even if not needed. resolve #25

    Fix default agent created even if not needed. resolve #25

    Try to fix issue: https://github.com/zeromq/ingescape/issues/25

    I added test on igsTester to highlight the issue.

    The fix consist on adding platform_name in igs context, initialize to zyre_node name. Its Value is agent name in case of using default agent from igs, or zyre_node uuid in case of multi igs agent.

    If you agree with the issue, maybe you want to discuss about the function admin_log from ingescape_private.h, because now I change the interface. I can keep the old function, who will call the new one.

    opened by Fafou 2
  • Launch IGS execution engine without the default agent's name fail

    Launch IGS execution engine without the default agent's name fail

    Hello,

    I'm trying to use Ingescape with a multi agent program.

    I'm working on C++ so I start to create a C++ binding to IGS agent, but I think that it's not a problem. Conceptually the software architecture it's really simple:

    1. Initiallise all my IGS agents
    2. Launch IGS execution engine for my platform/program
    3. Launch all my agents's processing

    I made a small example to show the "problem": https://gist.github.com/Fafou/cd1f87140588b6ed5cd6a8d6d51a3f48 (I build this example with: g++ -std=c++17 -o testAgent testAgents.cpp -lingescape -lczmq -lpthread)

    Today it didn't work because I didn't "manage" the internal IGS agent in addition to my agents. Without setting a name to the internal agent, IGS do not start. (a small comment in this example, after "igs_start_with_device", explains the cause of the problem)

    For me, but I could be wrong, IGS execution engine do not need to create a default agent if user use it's own agents.

    If user use default functions like: igs_input_create, in that case, yes he didn't want to manage it's own agent, so IGS can create agent for him, it's the "easy way" to use IGS. But if user want to manage its own agents by using igsagent_input_create, in that case we just need a valid IGS context, without default agent.

    do you agree with my analysis? or am I the one who is using this library incorrectly ?

    opened by Fafou 1
  • Problem: ingescape library compiled on a different version of macOS fails to link

    Problem: ingescape library compiled on a different version of macOS fails to link

    There are issues when using a verison of ingescape library that was not built on a macOS version not using the same SDK as ours. CMake seems to link explicitly against a given version of macOS SDK.

    Solution: Force --sysroot compilation parameter on macOS to the non-versioned SDK instead of the versioned one to be compatible with other versions.

    opened by Mathsoum 0
  • Problem: Some errors remains on the CI jobs

    Problem: Some errors remains on the CI jobs

    Solution: A combination of missing submodule updated and correct CLI parameters for macOS job.

    Also remove an old Debian job that is irrelevant today.

    opened by Mathsoum 0
  • Merge PUBLIC and INGESCAPE_EXPORT macros to have a single DLL export macro

    Merge PUBLIC and INGESCAPE_EXPORT macros to have a single DLL export macro

    Both these macros help with __declspec declaration for API exports in Windows DLLs.

    Previous PUBLIC macro was used in the code and never retired when fitting the project for ZeroMQ. Then we had both PUBLIC and INGESCAPE_EXPORT macros for similar behaviors. Both used in the code.

    This PR retires the PUBLIC macro and replaces its previous uses with the INGESCAPE_EXPORT macro.

    opened by Mathsoum 0
  • Problem: Static libraries built by our pipelines are linked against dynamic libraries

    Problem: Static libraries built by our pipelines are linked against dynamic libraries

    This means that there is no solution to have a fully statically linked stack.

    Solution: Update the CMake chain so static libraries can be linked against static dependencies, providing a fully statically linked stack.
    This is handled by a CMake variable called STATIC_ZMQ_LINK introduced for this purpose.
    STATIC_ZMQ_LINK is ON by default by can be deactivated if needed.

    Note: Changes had to be made in the submodules to read the STATIC_ZMQ_LINK variable.

    opened by Mathsoum 0
  • Problem: Bindings cannot link against static version of the library because of position dependent code

    Problem: Bindings cannot link against static version of the library because of position dependent code

    Solution: Make static library build with position independent code using the -fPIC flag.

    CMake provides an easy way to do this with set(CMAKE_POSITION_INDEPENDENT_CODE ON)

    opened by Mathsoum 0
  • Problem: Static lib names on Windows do not follow the same patterns as other ZeroMQ projects

    Problem: Static lib names on Windows do not follow the same patterns as other ZeroMQ projects

    Solution: Renaming

    The pattern is:

    • <lib_name>.lib/<lib_name>.dll => dynamic library
    • lib<lib_name>.lib => static library

    For ingescape:

    • ingescape.lib/ingescape.dll => dynamic library
    • libingescape.lib => static library
    opened by Mathsoum 0
  • Problem: Pipelines do not produce a static library for Windows

    Problem: Pipelines do not produce a static library for Windows

    Solution: Tell the pipeline to build static libraries too (was explicitly disabled).

    In order for the pipeline to complete and validate the changes, I had to fix several side-points

    • fix CI android jobs
    • use IP instead of DN for net shared drives so DNS resolving is not an issue

    The other jobs on our pipelines follow these points already.

    opened by Mathsoum 0
  • Problem: Packaged versions of ingescape for Debian are all stripped of the debug symbols

    Problem: Packaged versions of ingescape for Debian are all stripped of the debug symbols

    Solution: Generate multiple packages.

    • One stripped, called ingescape-dev
    • One in Release mode with symbols, called ingescape-no-strip-dev
    • One in Debug mode with all the symbols and without optimization, called ingescape-debug-dev

    The PR also includes a fix to trigger jobs on our private pipelines

    opened by Mathsoum 0
  • Problem: SIGSEGV occurs when trying to compare mappings with splits

    Problem: SIGSEGV occurs when trying to compare mappings with splits

    Solution: There was a copy/paste mistake is the mapping_is_equal method. Using the right variable name solved the issue.

    This PR increments the PATCH version number from 3.0.1 to 3.0.2

    opened by Mathsoum 0
Owner
The ZeroMQ project
The ZeroMQ project
null 5.6k Sep 21, 2022
Frog is an integration of memory-based natural language processing (NLP) modules developed for Dutch. All NLP modules are based on Timbl, the Tilburg memory-based learning software package.

Frog - A Tagger-Lemmatizer-Morphological-Analyzer-Dependency-Parser for Dutch Copyright 2006-2020 Ko van der Sloot, Maarten van Gompel, Antal van den

Language Machines 70 Aug 24, 2022
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

Distributed (Deep) Machine Learning Community 23.2k Sep 24, 2022
A Modular Framework for LiDAR-based Lifelong Mapping

LT-mapper News July 2021 A preprint manuscript is available (download the preprint). LT-SLAM module is released.

Giseop Kim 258 Sep 21, 2022
This is a group project for the unit Technical Software Design.

electoral-project This is a group project for the unit Technical Software Design. Group number: 9 Members of this group: Grace Tang, Lorien Cutler, Jo

Grace Tang 3 May 24, 2021
Caffe: a fast open framework for deep learning.

Caffe Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berke

Berkeley Vision and Learning Center 32.9k Sep 22, 2022
R2LIVE is a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses the measurement from the LiDAR, inertial sensor, visual camera to achieve robust, accurate state estimation.

R2LIVE is a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses the measurement from the LiDAR, inertial sensor, visual camera to achieve robust, accurate state estimation.

HKU-Mars-Lab 576 Sep 5, 2022
[RSS 2021] An End-to-End Differentiable Framework for Contact-Aware Robot Design

DiffHand This repository contains the implementation for the paper An End-to-End Differentiable Framework for Contact-Aware Robot Design (RSS 2021). I

Jie Xu 48 Jul 29, 2022
Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

File systems and Storage Lab (FSL) 182 Aug 26, 2022
C-based/Cached/Core Computer Vision Library, A Modern Computer Vision Library

Build Status Travis CI VM: Linux x64: Raspberry Pi 3: Jetson TX2: Backstory I set to build ccv with a minimalism inspiration. That was back in 2010, o

Liu Liu 6.9k Sep 20, 2022
An SDL2-based implementation of OpenAL in a single C file.

MojoAL MojoAL is a full OpenAL 1.1 implementation, written in C, in a single source file. It uses Simple Directmedia Layer (SDL) 2.0 to handle much of

Ryan C. Gordon 88 Sep 13, 2022
This robot lcoalisation package for lidar-map based localisation using multi-sensor state estimation.

A ROS-based NDT localizer with multi-sensor state estimation This repo is a ROS based multi-sensor robot localisation. An NDT localizer is loosely-cou

null 42 Sep 14, 2022
PDCurses - a curses library for environments that don't fit the termcap/terminfo model.

Welcome to PDCurses! PDCurses is an implementation of X/Open curses for multiple platforms. The latest version can be found at: https://pdcurses.org/

William McBrine 800 Sep 19, 2022
Allows for multiple SwitchBot buttons and curtains to be controlled via MQTT sent to ESP32. ESP32 will send BLE commands to switchbots and return MQTT responses to the broker. Also supports Meter/Temp Sensor

SwitchBot-MQTT-BLE-ESP32 Switchbot local control using ESP32. no switchbot hub used/required. works with any smarthub that supports MQTT https://githu

null 299 Sep 22, 2022
Eclipse Mosquitto - An open source MQTT broker

Mosquitto is an open source implementation of a server for version 5.0, 3.1.1, and 3.1 of the MQTT protocol. It also includes a C and C++ client library, and the mosquitto_pub and mosquitto_sub utilities for publishing and subscribing.

Eclipse Foundation 6.6k Sep 20, 2022
New generation message broker service

Push1st Push1st is open source multiple protocol PUB/SUB message broker server (Pusher, MQTT, RAW WebSocket). Key features Suitable for distributed on

Naveksoft 16 Jan 18, 2022
Distributed, Encrypted, Fractured File System - A custom distributed file system written in C with FUSE

A custom FUSE-based filesystem that distributes encrypted shards of data across machines on a local network, allowing those files to be accessible from any machine.

Charles Averill 15 Sep 4, 2022
Kunlun distributed DBMS is a NewSQL OLTP relational distributed database management system

Kunlun distributed DBMS is a NewSQL OLTP relational distributed database management system. Application developers can use Kunlun to build IT systems that handles terabytes of data, without any effort on their part to implement data sharding, distributed transaction processing, distributed query processing, crash safety, high availability, strong consistency, horizontal scalability. All these powerful features are provided by Kunlun.

zettadb 108 Sep 6, 2022
ESP32-Skid-Steer - Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on the model.

ESP32-Skid-Steer Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on

null 5 Feb 4, 2022