yadcc - Yet Another Distributed C++ Compiler

Related tags

Compilers yadcc
Overview

Yadcc 分布式 C++ 编译器

Yadcc是一套腾讯广告自研的工业级C++分布式编译系统。目前在我们1700+核的集群中每天编译300,0000+个目标文件,产出约3~5TB,已经持续稳定运营 8 个月。

2021 年 6 月,正式对外开源。

取决于代码逻辑及本地机器配置,yadcc可以利用几百乃至1000+核同时编译(内部而言我们使用512并发编译),大大加快构建速度。

具体简介及技术细节可以参考我们的技术文档

系统要求

  • Linux 3.10 及以上内核,暂不支持其他操作系统;
  • x86-64 处理器;
  • 编译yadcc需要GCC 8 及以上版本的编译器,基于yadcc进行分布式编译时可以支持其他更低版本编译器。

基本原理

ccachedistccicecc等工具类似;

  • 我们的客户端伪装成编译器(通常是通过ln -sf yadcc g++创建的符号链接)
  • 通过将我们的客户端伪装的编译器加入PATH头部,这样构建系统就会实际执行yadcc来编译
  • yadcc会按照命令行对源代码进行预处理,得到一个自包含的的预处理结果
  • 以预处理结果、编译器签名、命令行参数等为哈希,查询缓存,如果命中,直接返回结果
  • 如果不命中,就请求调度器获取一个编译节点,分发过去做编译
  • 等待直到从编译集群中得到编译结果,并更新缓存

由于预处理时间通常远小于编译时间,因此这样可以降低单个文件的本地开销。同时,由于等待编译结果时本地无需进行操作,因此可以增大本地的编译并发度(如8核机器通常可以make -j100),以此实现更高的吞吐。

需要注意的是,分布式编译通常只能提高吞吐,但是不能降低单个文件的编译耗时(假设不命中缓存)。因此,对于无法并发编译的工程,除非命中缓存,否则分布式编译通常不能加快编译,反而可能有负面效果。

设计特点

我们的系统由调度器、缓存服务器、守护进程及客户端组成:

  • 对上层的构建系统(Make、CMake,Blade、Bazel 等)透明,方便适配各种构建系统。
  • 调度器全局共享,所有请求均由调度节点统一分配。这样,低负载时可允许客户端尽可能提交更多的任务,集群满载时可阻塞新请求避免过载
  • 中心的调度节点也避免了需要客户机感知编译集群的列表的需要,降低运维成本。
  • 编译机向调度器定期心跳,这样我们不需要预先在调度器处配置编译机列表,降低运维成本。
  • 分布式缓存避免不必要的重复编译。同时本地守护进程处会维护缓存的布隆过滤器,避免无意义的缓存查询引发不必要的网络延迟。
  • 使用本地守护进程和外界通信,这避免了每个客户端均反复进行TCP启动等操作,降低开销。另外这也允许我们在守护进程处维护一定的状态,提供更多的优化可能。
  • 客户端会和本地守护进程通信,综合控制本地任务并发度避免本地过载。
  • 我们通过编译器哈希区分版本,这允许我们的集群中存在多个不同版本的编译器

同时,我们做了多层重试,确保不会因为网络抖动、编译机异常离线等工业场景常见的问题导致的不必要的失败。

开始使用

Yadcc自带了必要的第三方库,因此通常不需要额外安装依赖。

需要注意的是,yadcc通过git-submodule引用flare,因此编译之前需要执行git submodule update拉取flare。另外由于flare代码仓库需要git-lfs支持,因此您还需要安装git-lfs。具体可以参考flare的相关说明

git clone https://github.com/Tencent/yadcc --recurse-submodules

git clone https://github.com/Tencent/yadcc
cd yadcc
git submodule init
git submodule update .

编译yadcc

可以使用如下命令编译yadcc

./blade build yadcc/...

搭建环境及使用

搭建环境及使用方式可以参考详细文档

效果

我们搭建了一个 1000 多核的测试机群,在一些大型 C++ 项目上实测了效果。

LLVM 项目:

在我们的测试环境中共计 6124 个编译目标,结果如下:

  • 本地8并发编译:47分51秒
  • 分布式256并发:3分11秒

对于我们内部的一组更大的实际产品项目代码上:

  • 16C 开发机本地 8 并发:2时18分17秒
  • ccache+distcc, -j144:44分23秒
  • 76C 高性能开发机,-j80:25分18秒
  • yadcc:9分25秒

详情参见性能测试对比

总体而言,yadcc 有相当明显的性能优势。

相关项目

对于大型 C++ 项目,构建速度一直是个比较的问题,因此目前也已经有了一些相关的项目:

比较而言,Bazel 相关的两个协议是非跨构建系统的,只能用于 Bazel 或者支持这些协议的构建系统;distcc 缺乏统一的调度;其他几个只是编译缓存;icecream 是和 yadcc 最接近的,也是我们一开始尝试使用的,不过在并行数百个任务时,性能和稳定性表现不佳。

因此我们才开发了 yadcc,除了提供更好的性能和可靠性外,还额外增加了一些其他功能。

Yadcc 是我们目前已知的内置缓存机制的通用分布式编译系统。

Issues
  • build failed, undefined reference to `idn2_xxx'

    build failed, undefined reference to `idn2_xxx'

    I have build yadcc suceessfully on a server but failed on another one. The error is at tho bottom.

    on the suceess server, curl's configure is without idn2: checking for libidn2 options with pkg-config... no configure: IDN_LIBS: "-lidn2" configure: IDN_LDFLAGS: "" configure: IDN_CPPFLAGS: "" configure: IDN_DIR: "" checking if idn2_lookup_ul can be linked... no checking idn2.h usability... no checking idn2.h presence... no checking for idn2.h... no configure: WARNING: Cannot find libraries for IDN support: IDN disabled

    on the failed server, there is a libidn2.so: checking for libidn2 options with pkg-config... found configure: pkg-config: IDN_LIBS: "-lidn2" configure: pkg-config: IDN_LDFLAGS: "" configure: pkg-config: IDN_CPPFLAGS: "" configure: pkg-config: IDN_DIR: "" checking if idn2_lookup_ul can be linked... yes checking idn2.h usability... yes checking idn2.h presence... yes checking for idn2.h... yes

    But if the env has libidn2.so, why yadcc do not link it? I don't know blade very well, it seems to be an internal tool in tencent. I have add -lidn2 in linkflags in BLADE_ROOT, but it do not work.

    =========================================================

    Blade: Entering directory /home/zhut/davidwang/yadcc' Blade(info): Loading config file "/home/zhut/davidwang/yadcc/thirdparty/blade/blade.conf" Blade(info): Loading config file "/home/zhut/davidwang/yadcc/BLADE_ROOT" Blade(info): Loading BUILD files... Blade(info): Loading done. Blade(info): Analyzing dependency graph... Blade(info): Analyzing done. Blade(info): Generating backend build code... Blade(info): Generating done. Blade(info): Building... Blade(info): Adjust build jobs number(-j N) to be 8 [1/3] LINK BINARY build64_release/yadcc/scheduler/yadcc-scheduler FAILED: build64_release/yadcc/scheduler/yadcc-scheduler g++ -o build64_release/yadcc/scheduler/yadcc-scheduler -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/yadcc/api/libscheduler_proto.a build64_release/yadcc/api/libenv_desc_proto.a build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a -Wl,--no-whole-archive @build64_release/yadcc/scheduler/yadcc-scheduler.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in functionconn_free': url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference toidn2_check_version' /usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference toidn2_strerror' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x49ac): undefined reference to idn2_free' collect2: error: ld returned 1 exit status [2/3] LINK BINARY build64_release/yadcc/cache/yadcc-cache FAILED: build64_release/yadcc/cache/yadcc-cache g++ -o build64_release/yadcc/cache/yadcc-cache -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/yadcc/api/libcache_proto.a build64_release/yadcc/cache/libdisk_cache_engine.a build64_release/yadcc/cache/libnull_cache_engine.a build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a -Wl,--no-whole-archive @build64_release/yadcc/cache/yadcc-cache.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in functionconn_free': url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference toidn2_check_version' /usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference toidn2_strerror' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x49ac): undefined reference to idn2_free' collect2: error: ld returned 1 exit status [3/3] LINK BINARY build64_release/yadcc/daemon/yadcc-daemon FAILED: build64_release/yadcc/daemon/yadcc-daemon g++ -o build64_release/yadcc/daemon/yadcc-daemon -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/builtin/libtcmalloc_profiler_http_handler.a build64_release/yadcc/api/libcache_proto.a build64_release/yadcc/api/libdaemon_proto.a build64_release/yadcc/api/libscheduler_proto.a build64_release/yadcc/daemon/local/libmessages_proto.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a build64_release/yadcc/daemon/libcache_meta_proto.a build64_release/yadcc/api/libenv_desc_proto.a -Wl,--no-whole-archive @build64_release/yadcc/daemon/yadcc-daemon.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in functionconn_free': url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference toidn2_check_version' /usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference toidn2_strerror' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x49ac): undefined reference to `idn2_free' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Blade(error): Build failure. Blade(info): Cost time 4.43s Blade(error): Failure

    opened by aguludunu 5
  • task cannot be started

    task cannot be started

    E0803 21:09:16.471520 8840 task_dispatcher.cc:314] [10.243.2.241:48228] Unrecognized compilation environment [7446218a5a1cd826cd663c99919545490da3c56af4ffd1d53d1554cd0768b0ee] is requested by [10.243.2.241].

    opened by mathslimin 5
  • Memory Usage for Large File Operations

    Memory Usage for Large File Operations

    {
        flare::NoncontiguousBuffer standard_output;
        ......
        auto result = flare::FlattenSlow(output->standard_output);
        auto deps_list = flare::Split(result, '\n');
    }
    

    a 400 MB file is cached in standard_output. After I use it, I find that the memory is still occupied and cannot be released.

    opened by XlousZ 2
  • Some configuration consultations on the flare server and client

    Some configuration consultations on the flare server and client

    My machine: 16u32g

    Is there a simple relationship? flare_rpc_client_max_connections_per_server,flare_concurrency_hint,flare_rpc_server_stream_concurrency,flare_rpc_client_stream_concurrency, etc

    Is it possible to provide a configuration guide document based on the configuration of personal computers?

    opened by XlousZ 2
  • daemon can not find possible compiler in ubuntu 20.04

    daemon can not find possible compiler in ubuntu 20.04

    after install gcc by apt install in ubuntu 20.04 /usr/bin/gcc is a symlink file, the realpath of gcc is /usr/bin/x86_64-linux-gnu-gcc-9 the daemon can not find any compiler, because the bellow code skip symlink

      if (S_ISLNK(buf.st_mode)) {          // Symbolic link, ignore it.
        continue;
      }
    

    https://github.com/Tencent/yadcc/blob/3f776f9a40582c9435c06cf5e2cf83b3313f0f81/yadcc/daemon/cloud/compiler_registry.cc#L72

    opened by cyqw 2
  • some building problems on ubuntu

    some building problems on ubuntu

    1、error occurred under gcc8 when building python2.7. compiled successfully under gcc7

    2、python: can't decompress data; zlib not available when building python2.7. solution

    3、ubuntu 18.04.5(g++ 8.5)、ubuntu20.04(g++ 9.3) The same compilation error occurred when i tryed to build yadcc on my PC which installed with ubuntu 18.04.5(g++ 8.5) or ubuntu20.04(g++ 9.3), but pthread.a(/usr/lib/x86_64-linux-gnu/libpthread.a) and pthread.so(/usr/lib/x86_64-linux-gnu/libpthread.so) exist. Strangely enough, it can be compiled successfully in the company environment which installed with ubuntu 18.04.

    [3/75] LINK SHARED build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.so
    FAILED: build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.so
    g++ -shared -o build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.so  -Lthirdparty/jdk/lib -Xlinker --no-undefined build64_release/flare/rpc/protocol/http/builtin/rpc_form_handler.objs/rpc_form_handler.cc.o build64_release/flare/rpc/protocol/http/builtin/libstatic_resources.so build64_release/flare/rpc/protocol/http/libhttp.so build64_release/flare/rpc/protocol/http/libbinlog_proto.so build64_release/flare/rpc/protocol/http/libbuffer_io.so build64_release/flare/rpc/protocol/http/libhttp_handler.so build64_release/flare/rpc/protocol/http/libmessage.so build64_release/flare/base/libcompression.so build64_release/flare/base/compression/libgzip.so build64_release/flare/base/compression/liblz4.so build64_release/thirdparty/lz4/lib64/liblz4.so build64_release/flare/base/compression/libsnappy.so build64_release/thirdparty/snappy/lib64/libsnappy.so build64_release/flare/base/compression/libzstd.so build64_release/thirdparty/zstd/lib64/libzstd.so build64_release/flare/base/buffer/libcompression_output_stream.so build64_release/flare/base/compression/libutil.so build64_release/flare/base/compression/libcompression.so build64_release/flare/net/http/libpacket_desc.so build64_release/flare/net/http/libhttp_request.so build64_release/flare/net/http/libhttp_response.so build64_release/flare/net/http/libhttp_message.so build64_release/flare/net/http/libhttp_headers.so build64_release/flare/net/http/libtypes.so build64_release/flare/rpc/internal/libsession_context.so build64_release/flare/rpc/binlog/liblog_reader.so build64_release/flare/rpc/binlog/libdry_runner.so build64_release/flare/base/libstatus.so build64_release/flare/fiber/liblogging.so build64_release/flare/fiber/libtimer.so build64_release/flare/fiber/libwork_queue.so build64_release/flare/rpc/binlog/liblog_writer.so build64_release/flare/rpc/binlog/libdumper.so build64_release/flare/rpc/binlog/libpacket_desc.so build64_release/flare/base/buffer/libzero_copy_stream.so build64_release/flare/rpc/tracing/libtracing_ops.so build64_release/flare/rpc/tracing/libtracing_ops_provider.so build64_release/thirdparty/opentracing-cpp/lib/libopentracing.so build64_release/flare/rpc/protocol/libstream_protocol.so build64_release/flare/base/libbuffer.so build64_release/flare/base/buffer/libbuiltin_buffer_block.so build64_release/flare/init/libon_init.so build64_release/flare/rpc/protocol/libcontroller.so build64_release/flare/rpc/protocol/libmessage.so build64_release/flare/base/experimental/libuuid.so build64_release/flare/base/net/libendpoint.so build64_release/flare/fiber/libfiber_basic.so build64_release/flare/base/internal/libindex_alloc.so build64_release/flare/fiber/detail/libscheduling_parameters.so build64_release/flare/fiber/detail/libfiber_impl.so build64_release/flare/fiber/detail/libcontext.so build64_release/flare/fiber/detail/librun_queue.so build64_release/flare/fiber/detail/libstack_allocator.so build64_release/flare/base/libmonitoring.so build64_release/flare/base/internal/libdpc.so build64_release/flare/base/monitoring/libinit.so build64_release/flare/base/monitoring/libdispatcher.so build64_release/flare/base/monitoring/libmonitoring_system.so build64_release/thirdparty/yaml-cpp/lib/libyaml-cpp.a build64_release/flare/base/libobject_pool.so build64_release/flare/base/object_pool/libdisabled.so build64_release/flare/base/object_pool/libglobal.so build64_release/flare/base/object_pool/libmemory_node_shared.so build64_release/flare/base/libexposed_var.so build64_release/flare/base/libtsc.so build64_release/flare/base/write_mostly/libmetrics.so build64_release/thirdparty/jsoncpp/lib/libjsoncpp.so build64_release/flare/base/internal/libtime_keeper.so build64_release/flare/base/internal/libbackground_task_host.so build64_release/flare/base/internal/libthread_pool.so build64_release/flare/base/internal/libcpu.so build64_release/flare/base/libstring.so build64_release/flare/base/object_pool/libthread_local.so build64_release/flare/base/libdemangle.so build64_release/flare/base/liberased_ptr.so build64_release/flare/base/thread/libattribute.so build64_release/flare/base/thread/liblatch.so build64_release/flare/base/thread/libout_of_duty_callback.so build64_release/flare/base/liblogging.so build64_release/flare/base/libchrono.so build64_release/flare/base/internal/libmemory_barrier.so build64_release/flare/base/internal/liblogging.so build64_release/thirdparty/fmt/lib64/libfmt.so build64_release/flare/base/thread/libspinlock.so build64_release/thirdparty/protobuf-3.4.1/libprotobuf.so build64_release/thirdparty/protobuf-3.4.1/libprotobuf_lite.so build64_release/thirdparty/glog/lib/libglog.so build64_release/thirdparty/gflags/lib/libgflags.so build64_release/thirdparty/zlib/lib/libz.so build64_release/thirdparty/ctemplate/lib/libctemplate.so -ldl -lrt -lz -lpthread -lm
    /usr/bin/ld: build64_release/flare/rpc/protocol/http/builtin/rpc_form_handler.objs/rpc_form_handler.cc.o: in function `flare::object_pool::detail::memory_node_shared::GlobalPoolDescriptor* flare::object_pool::detail::memory_node_shared::GetGlobalPoolDescriptor<flare::PolymorphicBuffer>()':
    /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:700: undefined reference to `pthread_once'
    collect2: error: ld returned 1 exit status
    
    opened by XlousZ 1
  • 对于跨操作系统的分布式编译应该怎么做

    对于跨操作系统的分布式编译应该怎么做

    机器1:client(user,且不不接受任务):ubuntu1804,gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 机器2:servant(专用编译机):ubuntu2004,gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0。 servant上支持的gcc-7的哈希值与user需要的不一致,对于这种情况有什么建议吗?ubuntu2004能编译出来和ubuntu1804哈希值一样的gcc吗?你们对于这样的场景,你们是怎么做的?

    opened by lijiaonothing 2
  • Fail to build yadcc

    Fail to build yadcc

    yadcc/build64_release/thirdparty/opentracing-cpp/opentracing-cpp-1.5.1/3rd_party/include/opentracing/catch2/catch.hpp:6631:45: error: size of array ‘altStackMem’ is not an integral constant-expression 6631 | char FatalConditionHandler::altStackMem[SIGSTKSZ] = {};

    glibc changed SIGSTKSZ to be a syscall instead of being constant. #catchorg/Catch2#2178

    opened by Domain 0
  • Can i Preprocess with PCH and compile the job remotely?

    Can i Preprocess with PCH and compile the job remotely?

    As official doc describes, PCH can be used in preprocess, and insert "#pragma GCC pch_preprocess "filename"" into the output. Furthermore, the path can be absolute or relative to the working dir. That's to say, it is possible to use PCH to achieve distribute build, which could improve the compilation efficiency. And i have skipped your code and found during the preparation, yadcc skips the PCH and just use -E, -fno-working-directory and -fdirectives-only to produce preprocess output. Now, i have two questions and wish your reply.

    1、As i knew, -E option just preprocess the included headers directly without PCH, and should takes more time not only during the preprocess but also during the compilation. So, have u attemp to use PCH to achieve distributely build in yadcc, and if so, why not use it?

    2、As gcc doc describes, -fworking-directory is default setting. However, yadcc turns it off and user -fno-working-directory, and some debug information like file path will not be inserted into the preprocess output. As i think, the purpose is to avoid absoulte path in the output which may cause missing cache in spite of the same source file. I have tried to handle the preprocess output of msvc compiler, and found without absolute path, cache will be hit if the source file is same in different computer. It helps much to save compilation time, but i can use the results to start executable files but can not use the results obj files or exp files or pdb files to debug if turning off -fworking-directory . Thus, i wonder that have yadcc found similar problems after turning path off.

    Wish your reply, thank you very much.

    opened by chengjiaozyl 5
  • Importing changes from branch `next`

    Importing changes from branch `next`

    Code in master has long been superseded internally, using code in next brings us the following benefits:

    • Natural support for more languages, from design perspective (code itself, protocol, source file layout);
    • Support for --coverage (i.e., gcov);
    • More backends for cache.
    • ...

    Yet, IIRC, it does regress in performance when compiling C++. This is mostly due to switch of protocol between client and daemon, from a dirty-and-quick one to a JSON based one. JsonCpp costs us quite some cpu cycles here.

    That regression shouldn't hurt much, it's ~2%, and should be unnoticeable most of time.

    opened by 0x804d8000 0
Owner
Tencent
Tencent
Compiler Design Project: Simulation of front-end phase of C Compiler involving switch-case construct.

CSPC41 Compiler Design Project Assignment Compiler Design Project: Simulation of front-end phase of C Compiler involving switch-case construct. Using

Adeep Hande 1 Dec 15, 2021
distributed builds for C, C++ and Objective C

distcc -- a free distributed C/C++ compiler system by Martin Pool Current Documents: https://distcc.github.io/ Formally http://distcc.org/ "pump" func

distcc 1.7k Jun 30, 2022
A Small C Compiler

8cc C Compiler Note: 8cc is no longer an active project. The successor is chibicc. 8cc is a compiler for the C programming language. It's intended to

Rui Ueyama 5.5k Jun 26, 2022
A C compiler for LLVM. Supports C++11/14/1z C11

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies

LLVM 14.5k Jun 27, 2022
GNU Prolog is a native Prolog compiler

GNU Prolog is a native Prolog compiler with constraint solving over finite domains (FD)

Daniel Diaz 54 Jun 10, 2022
Take your first step in writing a compiler.

first-step Take your first step in writing a compiler. Building from Source Before building first-step, please make sure you have installed the follow

PKU Compiler Course 26 May 23, 2022
NCC is an ANSI/ISO-compliant optimizing C compiler.

The compiler is retargetable by design, but, at present, it only produces binaries for Linux/x86_64. As the compiler ABI differs somewhat from the System V ABI used by Linux, its code cannot be linked against Linux system libraries. It does, however, provide its own (incomplete) standard ANSI/Posix C library.

Charles E. Youse 0 Apr 1, 2022
nanoc is a tiny subset of C and a tiny compiler that targets 32-bit x86 machines.

nanoc is a tiny subset of C and a tiny compiler that targets 32-bit x86 machines. Tiny? The only types are: int (32-bit signed integer) char (8-

Ajay Tatachar 16 Feb 13, 2022
Smaller C is a simple and small single-pass C compiler

Smaller C is a simple and small single-pass C compiler, currently supporting most of the C language common between C89/ANSI C and C99 (minus some C89 and plus some C99 features).

Alexey Frunze 1.1k Jun 27, 2022
Microvm is a virtual machine and compiler

The aim of this project is to create a stack based language and virtual machine for microcontrollers. A mix of approaches is used. Separate memory is used for program and variable space (Harvard architecture). An interpreter, virtual machine and compiler are available. A demostration of the interpreter in action is presented below.

null 11 Jun 21, 2022
Aheui JIT compiler for PC and web

아희짓 개요 아희짓은 아희 언어를 위한 JIT (Just in Time) 컴파일러입니다. 어셈블러와 유틸 라이브러리외에 외부 라이브러리에 전혀 의존하지 않고 JIT을 바닥부터 구현합니다. 지원 환경 64비트 windows, mac, linux (x86 아키텍쳐) 웹어셈

Sunho Kim 27 Jan 2, 2022
C implementation of the Tiny BASIC compiler found in an article by Dr. Austin Henley

Teeny Tiny Basic A C implementation of the Tiny BASIC compiler found in this article and this github repo by Dr. Austin Henley. I did pretty well in A

Gavin Morris 5 Jun 12, 2022
bcc is an interactive compiler of a language called b.

bcc is an interactive compiler of a language called b.

kparc 16 Apr 5, 2022
This is a compiler written from scratch in C

C Compiler This is a compiler written from scratch in C, with fully supporting C18 as a goal. It can currently compile itself, and most simple program

null 28 Jun 10, 2022
mrcceppc is a reimplementation project for the Metrowerks mwcceppc compiler.

Compiler | mrcceppc mrcceppc is a reimplementation project for the Metrowerks mwcceppc compiler. Compiling Run generate_{version}.bat for which versio

null 8 Jun 20, 2022
A C header that allow users to compile brainfuck programs within a C compiler.

brainfuck.h A C header that allow users to compile brainfuck programs within a C compiler. You can insert the header into the top of your brainfuck so

null 2 Apr 18, 2022
Gilbraltar is a version of the OCaml compiler to be able to build a MirageOS for RaspberryPi 4.

Gilbraltar is a version of the OCaml compiler to be able to build a MirageOS for RaspberryPi 4. It's a work in progress repository to provide a dune's toolchain (as ocaml-freestanding) specialized for Raspberry Pi 4.

Calascibetta Romain 44 May 7, 2022
NaiveCC: a compiler frontend for a subset of C

NaiveCC: a compiler frontend for a subset of C This is a toy compiler frontend for a subset of the C programming language based on the LR(1) parsing t

Yuxiang Wei 1 Nov 15, 2021
A LLVM and Clang compiler toolchain built for kernel development

Cosmic-Clang Toolchain This is a LLVM and Clang compiler toolchain built for kernel development. Builds are always made from the latest LLVM sources r

Ǥђ๏ຮ₮⌁Ⲙครtє࿐ 0 Apr 12, 2022