Add a fetch endpoint curl "127.0.0.1:9990/fetch?start=2016-10-31T06:33:44.866Z&end=2018-10-31T06:33:44.866Z&key=testkey"

Add an update endpoint curl "127.0.0.1:9990/update" --data-binary @metrics Metrics file

testkey 1 1484620850
testkey 3 1484620883

Summary: 7138b7d exposed the fact that our dependencies are all very old.

We still need to come up with a beter solution than manually updating when things break, but for now, this gets things working again by bumping the version numbers.

Updating the dependencies also broke two things that required fixing:

• fbthrift now generates extra files that need compiling
• folly::Singleton doesn't like being used before folly::init()

Test Plan:

• make test passes
• beringei_main runs without crashing

Summary:

• Require gtest headers to be downloaded before building unit tests
• Turn thrift generation into a proper build target

Test Plan:

• Clean clean build with -j10 now succeeds instead of failing
• Repeated make now does not recompile everything
• Touching any .thrift file or build_thrift.sh still causes a rebuild
• make clean now deletes thrift-generated files
• Build now works with ninja: mkdir build && cd build && cmake -GNinja .. && ninja

Summary: Currently we expect the user/developer to build/generate thrift files before running 'cmake ..'. This also means that users/developers need to remember to build thrift files on every change. This diff automates the process.

execute_process() is run on 'cmake ..' add_custom_command() is run on 'make'

The one disadvantage of this is that 'make' will now always rebuild every target that depends on these thrift files.

Test Plan:

1. Test 'cmake ..' with no build errors OUTPUT: -- FOLLY: /usr/local/facebook/include -- WANGLE: /usr/local/facebook/include -- PROXYGEN: /usr/local/facebook/include Building Required Thrift Files Building file: beringei_data.thrift Building file: beringei_grafana.thrift Building file: beringei.thrift -- Configuring done -- Generating done

2. Test 'cmake.. ' with build error

OUTPUT: -- FOLLY: /usr/local/facebook/include -- WANGLE: /usr/local/facebook/include -- PROXYGEN: /usr/local/facebook/include Building Required Thrift Files Building file: beringei_data.thrift [ERROR:beringei_data.thrift:13] (last token was 'sruct') syntax error [FAILURE:beringei_data.thrift:13] Parser error during include pass. CMake Error at CMakeLists.txt:56 (message): Could not build thrfft file.

3. Test 'make' with no build errors -> Success

4. Test 'make' with build errors

OUTPUT: [ 15%] Built target beringei_test_util [ 16%] Building CXX object beringei/if/CMakeFiles/beringei_thrift.dir/gen-cpp2/BeringeiService_processmap_compact.cpp.o [ 17%] Building CXX object beringei/if/CMakeFiles/beringei_thrift.dir/gen-cpp2/beringei_constants.cpp.o [ 18%] Building CXX object beringei/if/CMakeFiles/beringei_thrift.dir/gen-cpp2/beringei_data_types.cpp.o [ 19%] Building CXX object beringei/if/CMakeFiles/beringei_thrift.dir/gen-cpp2/beringei_data_constants.cpp.o [ 20%] Building CXX object beringei/if/CMakeFiles/beringei_thrift.dir/gen-cpp2/beringei_types.cpp.o [ 21%] Building CXX object beringei/if/CMakeFiles/beringei_thrift.dir/gen-cpp2/beringei_grafana_constants.cpp.o [ 22%] Building CXX object beringei/if/CMakeFiles/beringei_thrift.dir/gen-cpp2/beringei_grafana_types.cpp.o [ 23%] Linking CXX static library libberingei_thrift.a Building file: beringei_data.thrift Building file: beringei_grafana.thrift Building file: beringei.thrift [ERROR:beringei.thrift:15] (last token was 'ervice') syntax error [FAILURE:beringei.thrift:15] Parser error during include pass. beringei/if/CMakeFiles/beringei_thrift.dir/build.make:328: recipe for target 'beringei/if/libberingei_thrift.a' failed make[2]: *** [beringei/if/libberingei_thrift.a] Error 1 CMakeFiles/Makefile2:173: recipe for target 'beringei/if/CMakeFiles/beringei_thrift.dir/all' failed make[1]: *** [beringei/if/CMakeFiles/beringei_thrift.dir/all] Error 2 Makefile:138: recipe for target 'all' failed make: *** [all] Error 2

Summary: Update dependencies to be a month newer.

This also forces .thrift files to be recompiled when the thrift libraries are updated, avoiding mismatches between old generated thrift files and new thrift common libraries after pulling in new dependencies with setup_ubuntu.sh.

Test Plan: CircleCI

Summary:

Continuous Build Integration for Beringei using CIrcleCI and Docker. CircleCI only supports Ubuntu 14. Hence we setup a docker Ubuntu 16.10 container and run the build/test inside the docker container.

I should have stacked my diffs/pull requests. This Pull request is built on top of https://github.com/facebookincubator/beringei/pull/4 (automate build for thrift files).

Test Plan: CircleCI.

Found in SJC: MCS average was around 8.1 even though MCS never drops below 9. Likely reason is that I was not considering missing data. This diff considers the missing data when calculating average SNR, MCS and txPower.

This PR add Vagrantfile for using vagrant. It's more friendly for development than docker, you just need vagrant up and it will mount the folder into /vagrant in the guest vm. I've test it on some OS, maybe someone with a Mac can help test if works for Mac OS.

• when run vagrant up for the first time, it will run setup_ubuntu.sh
• tested host
  • Fedora 25
  • Ubuntu 16.04 LTS
  • Windows 10

And I don't know if it's possible to have a hack folder like k8s did, to put all the environment setup, test script (in the readme), format check scripts together, though there is one setup_ubuntu.sh but there could be more in the future.

beringei
  build
  beringei
  hack 
   setup_ubuntu.sh
   gen_config.sh
   start_beringei.sh
   insert_random.sh
   read.sh
   check_format.sh
````

Adapted from queryHelper.js

-accept a topology json struct which includes nodes, links, sites, etc. -fetch all key ids for the given topology name from the db -allow a formatter to take the key names in the DB and transform them into their short-names -return a list of the key names

This is a very simple PR. It just adds a .dockerignore file. This helps with docker caching, since we add the whole directory while building the docker image, ignoring those files does not trigger new steps. As a result cached steps will not be used only when the code itself changes.

Looks like we missed porting the cleaner thread to beringei (which periodically goes through and compacts the keylist and deletes old block files). This should make the service more stable/efficient over time.

Tested by running the service for > 24 hours while constantly pushing data into it:

  ./beringei/service/beringei_main \
    -beringei_configuration_path /tmp/beringei.json \
    -create_directories \
    -sleep_between_bucket_finalization_secs 60 \
    -allowed_timestamp_behind 300 \
    -bucket_size 600 \
    -buckets $((86400/600)) \
    -logtostderr \
    -v=2

In another terminal:

  while [[ 1 ]]; do
    ./beringei/tools/beringei_put \
        -beringei_configuration_path /tmp/beringei.json \
        testkey ${RANDOM} \
        -logtostderr -v 3
    sleep 30
done

This change allows Beringei with a known good version of the upstream files. setup_ubuntu.sh works and the dockerfile builds cleanly. I've taken the setup TG is using and wrapped it in #ifdef's to peg the versions.

Summary: Pull in the latest changes from various dependencies.

• Update FB dependencies to a newer date
• Use a commit hash for wangle, since its version tagging seems to have failed this week
• Update to the latest zstd version

Test Plan:

• It builds
• Tests pass