Issue

I've had several requests to make the syntax for linking compute kernels less clunky and more C++ like. That means keeping the map::link syntax but also supporting something with a bit more polish.

Potential Solution

int
main( int argc, char **argv )
{  
   //generic random kernel instantiations
   kernel f;
   kernel g;
   //explicit declaration of map for example, potentially keep hidden one for compatibility
   map m;
   //in order version
   m >> f >> "a"_o >> "b"_i >> g;
   //out order version
   m >> f >> "a"_o >> raft::ooo >> "b"_i >> g;
   m.exe();
   return( EXIT_SUCCESS );
}

Currently RaftLib is using the Makefile system, which works well for Linux and OS X. However, I also need to have RaftLib available in Windows, and available for Visual Studio projects.

At the moment, how can I compile RaftLib in such a way that it can be readily used by Visual-Studio-based projects? If I compile using cygwin Makefile system, then can the library still be usable by Visual Studio?

In the long run, would it be a good idea to migrate RaftLib to Cmake, so that it can automatically generate build scripts for any systems?

@f-tischler , Can you double check to make sure I'm not stomping on any of the updates you planned? @mr-j0nes, wouldn't mind an extra set of eyes just in case....

Primary changes for this pull request are:

• move core affinity code out to separate module
• move demangle code to separate module
• update cmake files to support namespace injection
• bugfix in mapbase.hpp which could leave uninitialized portinfo objects
• move "portNotFound" code from header and place in cpp file so that demangle gets into an object vs. being re-included in every header/raft file (would be a pain with the namespace injection later on)
• Fix pkg-config install mechanism to make building raftlib apps far easier
• update versions of shm, affinity, demangle libs (to also include shm bug fix)

I've been playing around with raftlib, but I've hit a brick wall when trying to output strings: Here is my test kernel:

class Sum : public raft::kernel
{
public:
    Sum()
        : raft::kernel()
    {
        // Define ports
        input.addPort<T>("input_a", "input_b");
        output.addPort<std::string>("sum");
    }

    raft::kstatus run() override
    {
        T a, b;
        input["input_a"].pop(a);
        input["input_b"].pop(b);

        auto c(output["sum"].template allocate_s<std::string>());
        std::stringstream ss;
        ss << a << " + " << b << " = " << (a + b) << " [" << GetCurrentThreadId() << "]\n";
        *c = ss.str();

        return raft::proceed;
    }
};

This crashes inside map::exe() after completing the kernels and deallocating memory. I've tried allocate and send and that doesn't run at all.

So my question is: Are std::string typed ports supported or am I doing something stupid?

RaftLib is an actor-oriented framework for stream processing, and one of the most important applications are video stream processing, which uses data structures such as cv::Mat from OpenCV or similar data structures, that use non-POD pointers such as smart pointers within the structure.

However, as illustrated in Issue #4, currently non-POD pointers are not supported and the presence of such in objects in queues will cause segfault.

This makes raftlib unusable in any substantial applications that involves complex data structures, and is a shopstopper for RaftLib. Therefore we need to add support of non-POD pointers to RaftLib as top priority

Just started trying to compile Master on Windows using CMake. Is Boost a requirement? It's not listed, but I am getting errors finding demangle.hpp as well as affinity.hpp (not sure if that's from boost or not...)

I should mention I tried compiling from v2020.07.01 tag as well with the same result.

Any help?

Hey Jonathan, hope you're doing well..

Say we have A >> B >> C A sends one message to B. B then splits this message into several hundreds of other messages that needs to send to C.

The problem I face is the following:

If I return raft::proceed after B sends a message to C, then the framework does not start B again (because B already consumed the input message from A)

If B sends all messages to C in a loop without returning raft::proceed it might happen that the queue gets saturated with so many messages.

So what is the best way to proceed in this scenario ?

Thanks in advance..

Hi there,

Great job with the lib. Let's say I have A >> B >> C. Is it possible to allocate data on A and just release it on C ? I don't seem to understand how to do it ..

Thanks.. PS: in a different post I provided a "poc.cpp" where I show (more or less) what I want to do ..

hi, I clone all the project from the github and compile the project, but I still can't make the examples, I want to know if the examples don't match the api now?

Describe the bug

Raftlib code compilation gives "file does not exist error" for "raftinc/hh.hpp." On inspection, the file is included in defs.hpp but doesn't exist anywhere in the source code. Not sure if its renamed or removed completely

Steps To Reproduce

• run any example (the sum example in the getting started section of raftlib), and it doesn't compile using pkg-config by throwing the error

Expected behavior

Raftlib example code to compile

In progress, will set to draft momentarily.

In this pull request

• changes for the fifo, ringbuffers, blocked structures
• aim is to make all structures with the ringbuffer compatible once again with shared memory so we can go back to multiprocess in addition to multi-thread.
• changes should also make it amenable to make the fifo itself a module, and therefore be able to insert whatever FIFOs we want to (e.g. #zeromq, https://github.com/zeromq/libzmq) and boostfifo, even MPI with wrappers.

@f-tischler, could you look over chances with an eye towards win, I'm gonna double check momentarily to make doubly sure I didn't miss any Win specific guards that were supposed to be in place.

• [x] ## Description

1. In order to adjust benchmark working size without bringing huge files, we let filereader kernel optionally take an argument, repetitions, which is the number of times a file is read repeatedly to feed the dataflow graph with arbitrary data to process.
2. The QThread include path was added with leading -I and the compiling options generated by cmake would be collapsed (missing space between two -I) making include directories invalid.

Type of change

• [x] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [x] This change requires a documentation update

How Has This Been Tested?

Changes are individually tested with the word search example.

• [ ] Runs locally on Windows
• [x] Runs locally on Linux
• [ ] Runs locally on OS X

Details

QThread library is compiled and installed at /bambo/qthread_install.

RaftLib build is configured with the cmake command:

$ QTHREAD_PATH=/bambo/qthread_install/ cmake -DCMAKE_INSTALL_PREFIX=/bambo/raft_install ..

Then make install.

To compile the search example:

$ export PKG_CONFIG_PATH=/mnt/ampere3/bambo/bmkvl/raft_install/pkgconfig/:$PKG_CONFIG_PATH
$ g++ -o search -DQTHREAD_FOUND $(pkg-config --cflags raftlib) ../examples/cppalgo/search/bmh.cpp $(pkg-config --libs raftlib)

To run the search example:

$ ./search ../examples/cppalgo/search/alice.txt
# or with 2 repetitions
$ ./search ../examples/cppalgo/search/alice.txt 2

Checklist:

• [ ] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] Any dependent changes have been merged and published in downstream modules
• [x] I have checked my code and corrected any misspellings

Describe the bug

The following line of code:

 m += ( raft::kset( g0, g1, g2 ) >> raft::kset( a, b, c ) ) >= ( j >> print );

should parse according to the grammar, indeed there is a test case in the dev branch to ensure that this is the case. Syntax, however, fails to parse in the master branch.

To fix, pull test cases to master branch, and debug from there.

Steps To Reproduce

Run test case ksetContContJoin.cpp

Expected behavior

• M producers in first kset should be joined to M consumers in second kset

Bug Description

If we build a map with no data source, meaning every kernel in the map has an input port, then a segmentation fault would be triggered upon execution.

Example To Reproduce the Bug

#include <iostream>
#include <raft>

class MasterKernel : public raft::kernel {

private:
    int dim;
    long chunk_size;
    bool waiting = false;
    uint8_t task_id;

public:
    MasterKernel(int _dim, long _chunk_size) :
        raft::kernel(), dim(_dim), chunk_size(_chunk_size) {
        std::cout << "master constructor\n";
        // Create ports
        input.addPort<uint8_t>("input");
        output.addPort<uint8_t>("output");
        task_id = 0;
    }

    virtual raft::kstatus run() {
        size_t numRead = 0;

        //std::cout << "Master waiting=" << waiting << std::endl;
        //if (waiting) {
        //    input["input"].recycle();
        //    waiting = false;
        //    if (2 == task_id) {
        //        return raft::stop;
        //    }
        //}

        // Get the number of points to operate on
        std::cout << "Master reads " << numRead << std::endl;

        // Push our output data
        output["output"].push<uint8_t>(task_id++);
        waiting = true;
        std::cout << "Master pushed\n";

        return raft::proceed;
    }
};


class LocalSearchKernel : public raft::kernel {
    int id;

public:
    LocalSearchKernel(int _id) : raft::kernel(), id(_id) {
        std::cout << "search constructor\n";
        // Create ports
        input.addPort<uint8_t>("input");
        output.addPort<uint8_t>("output");
    }
    virtual raft::kstatus run() {
        uint8_t &taskid( input["input"].peek<uint8_t>() );
        std::cout << __func__ << "(pop taskid=" << taskid << ")\n";
        output["output"].push<uint8_t>(taskid);
        input["input"].recycle();

        return raft::proceed;
    }
};

int main() {
  MasterKernel master(1, 10);
  LocalSearchKernel search(2);

  raft::map m;

  m += master >> search;
  m += search >> master;

  std::cout << "Executing map\n";
    // Execute the map
  m.exe< partition_dummy, pool_schedule, stdalloc, no_parallel >();

}

Is your feature request related to a problem? Please describe. Feature request related to closed issue #138, all submodules are not auto-downloaded when using the zip download. When compiling, no .git file is present and an ambiguous error message results.

Describe the solution you'd like Either download the code within the cmake or some server side script that would enable github to package the correct files for download.

Describe alternatives you've considered

1. add a hook in the cmake helper that does the git submodule initialization to check for a .git file first and if none exists then use the cmake download functionality to pull it down.
2. we could simply add a warning in the cmake script to download the appropriate files.

Additional context None

I'm working on Windows and for some reason I can't get stream operators to work. I have created a series of nodes (in my case I need them to be pointers. I tested this first by modifying one of the examples and that worked fine. However, in my code base I get an error. I'm porting a data flow application to use raftlib, converting connections to mappings. Here is my code:

for (Argument* out : function->inputs()) {
  for (Port* dest : out->connections()) {
    Statement* destStatement = dest->owner();
      raft_statement* destNode = statementMap[destStatement]; // raft_statement is a kernel
      m += (*inputNode)[out->name()] >> (*destNode)[dest->name()]; // inputNode is a kernel
    }
  }
}

The error I get is:

error C2678: binary '>>': no operator found which takes a left-hand operand of type 'raft::kernel' (or there is no acceptable conversion)
while trying to match the argument list '(raft::kernel, raft::kernel)'

Any ideas what I can be doing wrong?