I just started go-smf implementation. Go is missing flatbuffers parser so now I was thinking about using smf_gen to do that, do you have any thoughts?

this is a wip pr for review. the primary goal of this patch is to support cmake builds with git submodules to enable smf to work well in c++/cmake project workflows. it also includes some clean-up of existing cmake files.

this patch set can test builds and unit tests on {sid,stretch,buster,artful,zesty,fc25,fc26,fc27} x {makefiles, ninja} x {gcc, clang}. it's been tested with everything except clang since Seastar is having some issues there. you can run all of these configurations with:

git submodule update --init --recursive
ci/local.sh

it'd be great to get some early feedback, and testing with existing workflows.

dependencies

the deps that were handled by ansible roles are now included as sub-modules and built with cmake, provided that support isn't already available in the a target distribution. this includes zstd, xxhash, flat buffers, google test, HDR histogram, and fast range.

several of the deps handled by ansible did not seem to be used (e.g. roaring bitmaps, afl, pmem, etc...), so those can be added back later, or in case I missed something or they are used in separate workflows.

I didn't run into any issues with versions of boost, so there is no special handling there.

questions/remaining

1. some ansible deps like cpplint aren't setup yet. it isn't strictly needed for the build, so that can added that back in, or maybe just use ansible to do the release builds when cpplint can be applied?

2. the biggest thing left to do is get the cmake branch merged upstream in Seastar. Currently this PR points to some patches on top of a cmake branch maintained by a scylla developer. I suspect it won't be too long before that get merged.

3. packaging. I didn't do anything with cpack. is that something that is currently in use? I'm not very familiar with cpack, but it seemed to still produce src packages. adding explicit rpm/deb packaging isn't too hard, but maybe out of scope at the moment.

that's all I can think of right now.

update 1 (done)

• all of the build configurations can be run in parallel on semaphore ci, I presume. I do this with another project and its been incredibly helpful at catching early issues (https://github.com/noahdesu/zlog/blob/master/.travis.yml#L23). I don't have any experience setting up semaphore ci.

update 2 (done)

• the sanitizer stuff is missing

update 3 (done)

make sure that release/package/etc.. script are working on the new build

update 4 (done)

.common_build.sh can be deleted i think. just gets the nprocs

There are 2 main places in the hotpath that are lookup heavy.

1. Method dispatch of the RPC framework.
2. in the WAL path we do lookup of topic,partition tuples for every write.

We need to make sure that those 2 paths are optimized.

currently we use unordered_map.

However other map datastructures are designed for lookups! - We do not do a lot of insertions - so it's ok if they are slow. However, the lookups have to be exceedingly fast.

https://github.com/skarupke/flat_hash_map/blob/master/flat_hash_map.hpp

I'm not convinced that if we stored the RPC methods in a std::vector + for loop it wouldn't be faster for example.

This is blocked by the microbenchmark framework - i need to add the stuff for google-benchmark next.

Compiling the demo apps outside the smf tree, using smf as a git sub-module. This seems to work well, but the include paths are a little weird.

For example, the app wants to include histogram stuff:

#include "histogram/histogram_seastar_utils.h"

but for this path to work we'd need to add the entire smf/src directory to the list of includes of the including project. alternatively, one can do:

smf/src/include/smf/{histogram, rpc, etc...}

and in the including project simply do include_directories(smf/src/include) then

#include <smf/histogram_seastar_utils.h>

etc...

there are other schemes, too...

Hello, I have been surveying using Seastar to build a network stack for an application. However, I find out that this usually requires completely rewriting the application to adopt the seastar. When I look at the SMF, I am wondering if this RPC library can be used as a communication library within a conventional application?

this PR is for identifying any major issues before polishing it further. it is effectively the cooking approach without the cooking infrastructure. it works pretty well. not all dependencies have been ported over, but most. i happened to have everything on fc29 NOTE: this isn't yet integrated into the build.sh script etc....

Usage

cmake .
make

Current status

[nwatkins@smash smf-ng2]$ nice -n 19 ctest
Test project /home/nwatkins/src/smf-ng2
      Start  1: smf_checksum_benchmark_test
 1/11 Test  #1: smf_checksum_benchmark_test ......................   Passed    7.66 sec
      Start  2: smf_fbsalloc_benchmark_test
 2/11 Test  #2: smf_fbsalloc_benchmark_test ......................   Passed   45.59 sec
      Start  3: smf_histograms_integration_test
 3/11 Test  #3: smf_histograms_integration_test ..................   Passed    0.18 sec
      Start  4: smf_rpc_integration_test
 4/11 Test  #4: smf_rpc_integration_test .........................***Failed    4.60 sec
      Start  5: smf_rpc_recv_timeout_integration_test
 5/11 Test  #5: smf_rpc_recv_timeout_integration_test ............   Passed    0.24 sec
      Start  6: smf_rpc_multiple_remote_addrs_integration_test
 6/11 Test  #6: smf_rpc_multiple_remote_addrs_integration_test ...   Passed    0.19 sec
      Start  7: smf_rpc_backpressure_integration_test
 7/11 Test  #7: smf_rpc_backpressure_integration_test ............***Failed   12.88 sec
      Start  8: smf_rpc_send_client_timeout_integration_test
 8/11 Test  #8: smf_rpc_send_client_timeout_integration_test .....***Failed   32.02 sec
      Start  9: smf_rpc_bad_msg_t_integration_test
 9/11 Test  #9: smf_rpc_bad_msg_t_integration_test ...............   Passed    0.08 sec
      Start 10: smf_randomstr_unit_test
10/11 Test #10: smf_randomstr_unit_test ..........................   Passed    0.08 sec
      Start 11: smf_simple_hist_unit_test
11/11 Test #11: smf_simple_hist_unit_test ........................   Passed    0.07 sec

73% tests passed, 3 tests failed out of 11

#contributors/help w/ cmake pls!

Seastar added a complicated cmake set up here:

https://github.com/scylladb/seastar/commit/c79e9dec76186f76d1a2e48187e3073449a60bfe

I found cmake-cooking.sh hard to understand.

I think it does:

ExternalProjectAdd and given how cmake build systems are I'm sure it's doing the right thing under the covers.

It uses (i think standard ML) to generate a bash script. This bash script contains a giant string. The string is a cmake macro This cmake macro builds & passes through arguments for ExternalProjectAdd It build them and tells cmake to add it. It builds them w/ default options (we want to change some options to make them as static a build as possible)

I'm not sure if cmake-cooking.sh is a good fit for this project to use.

Ideally we should strive to have as a static of a compilation as we can it gives the compiler much more room for optimizing function calls etc.

We already have a git submodules for some dependencies and some apt-get or dnf-install dependencies. The mix is less than ideal. We should do one consistently. I vote for explicitly declaring all our deps. Testing against a known set and supporting that.

I like jessies idea of a recipe but as the main way of building this project.

We can say we tested against a known set of dependencies. Harder to find ABI crashes and try to debug other issues in practice. Except of course for openssl / gnutls (makes making security updates easier)

Jessie already did a lot of hardwork here: https://github.com/scylladb/seastar/blob/master/recipe/dev.cmake

We can try to make that our main dependency file. But of course it requires cmake-cooking.sh macro. A potential route is we render it and just save it to a cooking.cmake and use it as is for the first pass.

The ideal scenario would be for all our deps to be an ExternalProjectAdd type of defs.

Alternative designs are welcomed.

Currently, it is really hard to exclude targets and to set versions, etc on the cmake build

https://cmake.org/cmake/help/v3.0/module/ExternalProject.html

Effectively, it prints a bunch of warnings from different projects, it runs tests and the downside is that only mature projects have guards against them via a simple -DCMAKE_<PROJ_SPECIFIC_OPT>=OFF

which means a bunch of custom logic for them. (non pkgfind'd things)

Test project /home/agallego/workspace/smf/build_debug
    Start 1: Histogram
1/7 Test #1: Histogram ...............................   Passed    0.05 sec
    Start 2: HistogramLogging
2/7 Test #2: HistogramLogging ........................   Passed    0.03 sec
    Start 3: HistogramAtomic
3/7 Test #3: HistogramAtomic .........................   Passed    0.00 sec
    Start 4: flattests
4/7 Test #4: flattests ...............................   Passed    0.04 sec
    Start 5: smf_histograms_integration_test
5/7 Test #5: smf_histograms_integration_test .........   Passed    0.20 sec
    Start 6: smf_rpc_integration_test
6/7 Test #6: smf_rpc_integration_test ................   Passed    0.25 sec
    Start 7: smf_rpc_recv_timeout_integration_test
7/7 Test #7: smf_rpc_recv_timeout_integration_test ...   Passed    0.20 sec

100% tests passed, 0 tests failed out of 7

builds OK, please test before merging :)

also, it does seem that the blessed way of using seastar as a dependency in an application is for the application to adopt the cmake-cooking way of things, and use seastar as an "ingredient". that integration looks pretty nice as you can control the build (e.g. turn of tests etc...). so maybe that is a good next step, but will just need to be investigated. perhaps if this works we merge and create a ticket for investigating the cooking show version.

Maybe we could build and deploy an image in CI with entire built smf but mainly

1. demo_apps - client and server for testing with Go (and later node.js) implementation I am currently working on
2. smf_gen - can be useful for someone
3. load_gen - not sure what toolchain do you use to graph latencies but it would be useful to have it too :+1:

This all would be mainly useful for CI testing and this image could be built as the test.

I can submit a pull request just tell me your requirements, OS etc. (Ubuntu 16.04?)

I recently noticed that on release builds the rpc_generated.h file is produced using 64-bit types here:

https://github.com/smfrpc/smf/blob/master/src/include/smf/rpc_generated.h#L630

And with a debug build the same is produced with a 32-bit type. Oddly, it looks like 64-bit is hardcoded in the flatbuffers idl generator.

I'm wondering if anyone has seen this before I go looking deeper.

This makes it easy to build smf using bazel with auto-generated fbs.

I probably could make it include as a system header in bazel but this seems like a valid change.

Bumps nokogiri from 1.12.5 to 1.13.9.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps tzinfo from 1.2.7 to 1.2.10.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Describe the bug The cmake process is trying to download other packages. They all fail when I'm behind a proxy. It is not clear what tool is used to download, but it fails. The environment is set with a correct http/https proxies. wget works, so does curl.

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Asciinema

see https://asciinema.org/ for recording video in terminal

** Environment Info **

• operating system
• git hash
• compiler

Additional context Add any other context about the problem here.

Describe the bug Running the build script errors out with the following message:

ninja: build stopped: subcommand failed.
CMake Error at CMakeLists.txt:51 (message):
  Build step for deps-build failed: 1


-- Configuring incomplete, errors occurred!
See also "/root/smf/build/release/CMakeFiles/CMakeOutput.log".

I get this not just on Ubuntu 19.04, but also Ubuntu 20.04, and Fedora 31.

To Reproduce Steps to reproduce the behavior:

1. docker run -it ubuntu:disco
2. apt update -y && apt upgrade -y
3. apt install git
4. git clone https://github.com/smfrpc/smf.git
5. cd smf
6. ./install-deps.sh
7. ./tools/build.sh -r
8. Build fails per text above.

Expected behavior I expected a successful release build.

Environment Info

• Ubuntu 19.04, 20.04, and Fedora 31 running in Docker containers
• On current master branch.
• GCC 8 and 9