yadcc - Yet Another Distributed C++ Compiler

Related tags

Compilers yadcc
Overview

Yadcc 分布式 C++ 编译器

Yadcc是一套腾讯广告自研的工业级C++分布式编译系统。目前在我们1700+核的集群中每天编译300,0000+个目标文件,产出约3~5TB,已经持续稳定运营 8 个月。

2021 年 6 月,正式对外开源。

取决于代码逻辑及本地机器配置,yadcc可以利用几百乃至1000+核同时编译(内部而言我们使用512并发编译),大大加快构建速度。

具体简介及技术细节可以参考我们的技术文档

系统要求

  • Linux 3.10 及以上内核,暂不支持其他操作系统;
  • x86-64 处理器;
  • 编译yadcc需要GCC 8 及以上版本的编译器,基于yadcc进行分布式编译时可以支持其他更低版本编译器。

基本原理

ccachedistccicecc等工具类似;

  • 我们的客户端伪装成编译器(通常是通过ln -sf yadcc g++创建的符号链接)
  • 通过将我们的客户端伪装的编译器加入PATH头部,这样构建系统就会实际执行yadcc来编译
  • yadcc会按照命令行对源代码进行预处理,得到一个自包含的的预处理结果
  • 以预处理结果、编译器签名、命令行参数等为哈希,查询缓存,如果命中,直接返回结果
  • 如果不命中,就请求调度器获取一个编译节点,分发过去做编译
  • 等待直到从编译集群中得到编译结果,并更新缓存

由于预处理时间通常远小于编译时间,因此这样可以降低单个文件的本地开销。同时,由于等待编译结果时本地无需进行操作,因此可以增大本地的编译并发度(如8核机器通常可以make -j100),以此实现更高的吞吐。

需要注意的是,分布式编译通常只能提高吞吐,但是不能降低单个文件的编译耗时(假设不命中缓存)。因此,对于无法并发编译的工程,除非命中缓存,否则分布式编译通常不能加快编译,反而可能有负面效果。

设计特点

我们的系统由调度器、缓存服务器、守护进程及客户端组成:

  • 对上层的构建系统(Make、CMake,Blade、Bazel 等)透明,方便适配各种构建系统。
  • 调度器全局共享,所有请求均由调度节点统一分配。这样,低负载时可允许客户端尽可能提交更多的任务,集群满载时可阻塞新请求避免过载
  • 中心的调度节点也避免了需要客户机感知编译集群的列表的需要,降低运维成本。
  • 编译机向调度器定期心跳,这样我们不需要预先在调度器处配置编译机列表,降低运维成本。
  • 分布式缓存避免不必要的重复编译。同时本地守护进程处会维护缓存的布隆过滤器,避免无意义的缓存查询引发不必要的网络延迟。
  • 使用本地守护进程和外界通信,这避免了每个客户端均反复进行TCP启动等操作,降低开销。另外这也允许我们在守护进程处维护一定的状态,提供更多的优化可能。
  • 客户端会和本地守护进程通信,综合控制本地任务并发度避免本地过载。
  • 我们通过编译器哈希区分版本,这允许我们的集群中存在多个不同版本的编译器

同时,我们做了多层重试,确保不会因为网络抖动、编译机异常离线等工业场景常见的问题导致的不必要的失败。

开始使用

Yadcc自带了必要的第三方库,因此通常不需要额外安装依赖。

需要注意的是,yadcc通过git-submodule引用flare,因此编译之前需要执行git submodule update拉取flare。另外由于flare代码仓库需要git-lfs支持,因此您还需要安装git-lfs。具体可以参考flare的相关说明

git clone https://github.com/Tencent/yadcc --recurse-submodules

git clone https://github.com/Tencent/yadcc
cd yadcc
git submodule init
git submodule update .

编译yadcc

可以使用如下命令编译yadcc

./blade build yadcc/...

搭建环境及使用

搭建环境及使用方式可以参考详细文档

效果

我们搭建了一个 1000 多核的测试机群,在一些大型 C++ 项目上实测了效果。

LLVM 项目:

在我们的测试环境中共计 6124 个编译目标,结果如下:

  • 本地8并发编译:47分51秒
  • 分布式256并发:3分11秒

对于我们内部的一组更大的实际产品项目代码上:

  • 16C 开发机本地 8 并发:2时18分17秒
  • ccache+distcc, -j144:44分23秒
  • 76C 高性能开发机,-j80:25分18秒
  • yadcc:9分25秒

详情参见性能测试对比

总体而言,yadcc 有相当明显的性能优势。

相关项目

对于大型 C++ 项目,构建速度一直是个比较的问题,因此目前也已经有了一些相关的项目:

比较而言,Bazel 相关的两个协议是非跨构建系统的,只能用于 Bazel 或者支持这些协议的构建系统;distcc 缺乏统一的调度;其他几个只是编译缓存;icecream 是和 yadcc 最接近的,也是我们一开始尝试使用的,不过在并行数百个任务时,性能和稳定性表现不佳。

因此我们才开发了 yadcc,除了提供更好的性能和可靠性外,还额外增加了一些其他功能。

Yadcc 是我们目前已知的内置缓存机制的通用分布式编译系统。

Comments
  • 文件预处理是由客户端自己完成吗

    文件预处理是由客户端自己完成吗

    在12c的机器上测试 make -j60 ,限制本地轻任务并发为12,并在 RewriteFile 前面加日志作为预处理时间。发现正常情况下预处理在1秒左右的文件,预处理变成了12秒。

    尝试从 RewriteFile 往下找,最后在似乎是由客户端调用的syscall。本地守护进程,设置了YADCC_LOG_LEVEL=0,也没打印预处理的日志。所以预处理是由客户端自己完成的?如果不是的话,能否告知在哪里加日志,可以获取预处理任务等待的时间?

    opened by zcfh 9
  • [yadcc arm] 资源包打包了源码目录的 libfakeroot.so, 期望打包构建出来的库

    [yadcc arm] 资源包打包了源码目录的 libfakeroot.so, 期望打包构建出来的库

    https://github.com/Tencent/yadcc/blob/3c0f34a36a664c0bd446b3fa5ed944bb0e656b0f/yadcc/client/libfakeroot/BUILD#L28

    构建了一个 arm 版本 yadcc, 其中发现资源库打包比较怪,打包的不是期望的构建出来的新库, 由于对于 blade 不熟悉, 没看出 src 中怎么引用其他库构建结果 只能是构建第一遍后,覆盖掉,然后再构建

    请问这里是否可以优化呢?

    opened by chencang1980 7
  • 安装报错

    安装报错

    [45/1637] MAKE //thirdparty/nghttp2:nghttp2_build
    ......
    libtool: install: /usr/bin/install -c .libs/libnghttp2.so.14.20.0 /home/name/workspace/yadcc/build64_release/thirdparty/nghttp2/lib/libnghttp2.so.14.20.0
    /usr/bin/install: cannot stat '.libs/libnghttp2.so.14.20.0': No such file or directory
    
    ll build64_release/thirdparty/nghttp2/nghttp2-1.41.0/lib/.libs
    
    -rw-rw-r-- 1 wuminghui03 wuminghui03 819848 Oct 13 17:46 libnghttp2.a
    lrwxrwxrwx 1 wuminghui03 wuminghui03     16 Oct 13 17:46 libnghttp2.la -> ../libnghttp2.la
    -rw-rw-r-- 1 wuminghui03 wuminghui03   1023 Oct 13 17:46 libnghttp2.lai
    lrwxrwxrwx 1 wuminghui03 wuminghui03     21 Oct 13 17:46 libnghttp2.so -> libnghttp2.so.14.20.0
    lrwxrwxrwx 1 wuminghui03 wuminghui03     21 Oct 13 17:46 libnghttp2.so.14 -> libnghttp2.so.14.20.0
    -rw-rw-r-- 1 wuminghui03 wuminghui03  27264 Oct 13 17:46 nghttp2_buf.o
    

    安装报错 .libs/libnghttp2.so.14.20.0 不存在,查看目录 build64_release/thirdparty/nghttp2/nghttp2-1.41.0/lib/.libs , 有指向 libnghttp2.so.14.20.0 但是这个库是不存在的,这个库应该是新构建的才对,但是并未出现这个动态库链接报错。 已经按照 REAMD 更新了submodule, 并且在 build/external/flare 指向过 git lfs install/pull, 是有步骤不对少拉了文件吗 机器 CentOS Linux release 7.4.1708 (Core)。

    opened by zcfh 7
  • build failed, undefined reference to `idn2_xxx'

    build failed, undefined reference to `idn2_xxx'

    I have build yadcc suceessfully on a server but failed on another one. The error is at tho bottom.

    on the suceess server, curl's configure is without idn2: checking for libidn2 options with pkg-config... no configure: IDN_LIBS: "-lidn2" configure: IDN_LDFLAGS: "" configure: IDN_CPPFLAGS: "" configure: IDN_DIR: "" checking if idn2_lookup_ul can be linked... no checking idn2.h usability... no checking idn2.h presence... no checking for idn2.h... no configure: WARNING: Cannot find libraries for IDN support: IDN disabled

    on the failed server, there is a libidn2.so: checking for libidn2 options with pkg-config... found configure: pkg-config: IDN_LIBS: "-lidn2" configure: pkg-config: IDN_LDFLAGS: "" configure: pkg-config: IDN_CPPFLAGS: "" configure: pkg-config: IDN_DIR: "" checking if idn2_lookup_ul can be linked... yes checking idn2.h usability... yes checking idn2.h presence... yes checking for idn2.h... yes

    But if the env has libidn2.so, why yadcc do not link it? I don't know blade very well, it seems to be an internal tool in tencent. I have add -lidn2 in linkflags in BLADE_ROOT, but it do not work.

    =========================================================

    Blade: Entering directory /home/zhut/davidwang/yadcc' Blade(info): Loading config file "/home/zhut/davidwang/yadcc/thirdparty/blade/blade.conf" Blade(info): Loading config file "/home/zhut/davidwang/yadcc/BLADE_ROOT" Blade(info): Loading BUILD files... Blade(info): Loading done. Blade(info): Analyzing dependency graph... Blade(info): Analyzing done. Blade(info): Generating backend build code... Blade(info): Generating done. Blade(info): Building... Blade(info): Adjust build jobs number(-j N) to be 8 [1/3] LINK BINARY build64_release/yadcc/scheduler/yadcc-scheduler FAILED: build64_release/yadcc/scheduler/yadcc-scheduler g++ -o build64_release/yadcc/scheduler/yadcc-scheduler -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/yadcc/api/libscheduler_proto.a build64_release/yadcc/api/libenv_desc_proto.a build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a -Wl,--no-whole-archive @build64_release/yadcc/scheduler/yadcc-scheduler.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in functionconn_free': url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference toidn2_check_version' /usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference toidn2_strerror' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x49ac): undefined reference to idn2_free' collect2: error: ld returned 1 exit status [2/3] LINK BINARY build64_release/yadcc/cache/yadcc-cache FAILED: build64_release/yadcc/cache/yadcc-cache g++ -o build64_release/yadcc/cache/yadcc-cache -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/yadcc/api/libcache_proto.a build64_release/yadcc/cache/libdisk_cache_engine.a build64_release/yadcc/cache/libnull_cache_engine.a build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a -Wl,--no-whole-archive @build64_release/yadcc/cache/yadcc-cache.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in functionconn_free': url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference toidn2_check_version' /usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference toidn2_strerror' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x49ac): undefined reference to idn2_free' collect2: error: ld returned 1 exit status [3/3] LINK BINARY build64_release/yadcc/daemon/yadcc-daemon FAILED: build64_release/yadcc/daemon/yadcc-daemon g++ -o build64_release/yadcc/daemon/yadcc-daemon -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/builtin/libtcmalloc_profiler_http_handler.a build64_release/yadcc/api/libcache_proto.a build64_release/yadcc/api/libdaemon_proto.a build64_release/yadcc/api/libscheduler_proto.a build64_release/yadcc/daemon/local/libmessages_proto.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a build64_release/yadcc/daemon/libcache_meta_proto.a build64_release/yadcc/api/libenv_desc_proto.a -Wl,--no-whole-archive @build64_release/yadcc/daemon/yadcc-daemon.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in functionconn_free': url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference toidn2_check_version' /usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference toidn2_strerror' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference toidn2_free' /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference toidn2_free' /usr/bin/ld: url.c:(.text+0x49ac): undefined reference to `idn2_free' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Blade(error): Build failure. Blade(info): Cost time 4.43s Blade(error): Failure

    opened by aguludunu 5
  • task cannot be started

    task cannot be started

    E0803 21:09:16.471520 8840 task_dispatcher.cc:314] [10.243.2.241:48228] Unrecognized compilation environment [7446218a5a1cd826cd663c99919545490da3c56af4ffd1d53d1554cd0768b0ee] is requested by [10.243.2.241].

    opened by mathslimin 5
  • distributed_cache_reader failed

    distributed_cache_reader failed

    W1022 14:18:14.378872 14776 distributed_cache_reader.cc:152] Failed to load compilation cache bloom filter from cache server: [1003] , last_bf_update: 355156, last_bf_full_update: 355158

    last_bf_update < last_bf_full_update, but cache server requires last_bf_update >= last_bf_full_update

    No logic error is found when working through distributed_cache_reader.cpp code.

    opened by XlousZ 3
  • tcmalloc segment

    tcmalloc segment

    When I do secondary development on yadcc daemon, the newly developed module causes segment errors. I feel confused as to what causes the problem.

    the following cases:

    • 1 @ 0x7fe968d7dc70 (unknown) @ 0x55f7bc7b1ea3 tcmalloc::ThreadCache::ReleaseToCentralCache() @ 0x55f7bc7b21f5 tcmalloc::ThreadCache::ListTooLong()
    • 2 @ 0x7fb467824c70 (unknown) @ 0x560a2ffc98d3 tcmalloc::CentralFreeList::FetchFromOneSpans() @ 0x560a2ffc9b9c tcmalloc::CentralFreeList::FetchFromOneSpansSafe() @ 0x560a2ffc9c63 tcmalloc::CentralFreeList::RemoveRange() @ 0x560a2ffd1d02 tcmalloc::ThreadCache::FetchFromCentralCache()

    The newly developed module: Flare::Fiber used

    opened by XlousZ 3
  • Memory Usage for Large File Operations

    Memory Usage for Large File Operations

    {
        flare::NoncontiguousBuffer standard_output;
        ......
        auto result = flare::FlattenSlow(output->standard_output);
        auto deps_list = flare::Split(result, '\n');
    }
    

    a 400 MB file is cached in standard_output. After I use it, I find that the memory is still occupied and cannot be released.

    opened by XlousZ 2
  • Some configuration consultations on the flare server and client

    Some configuration consultations on the flare server and client

    My machine: 16u32g

    Is there a simple relationship? flare_rpc_client_max_connections_per_server,flare_concurrency_hint,flare_rpc_server_stream_concurrency,flare_rpc_client_stream_concurrency, etc

    Is it possible to provide a configuration guide document based on the configuration of personal computers?

    opened by XlousZ 2
  • daemon can not find possible compiler in ubuntu 20.04

    daemon can not find possible compiler in ubuntu 20.04

    after install gcc by apt install in ubuntu 20.04 /usr/bin/gcc is a symlink file, the realpath of gcc is /usr/bin/x86_64-linux-gnu-gcc-9 the daemon can not find any compiler, because the bellow code skip symlink

      if (S_ISLNK(buf.st_mode)) {          // Symbolic link, ignore it.
        continue;
      }
    

    https://github.com/Tencent/yadcc/blob/3f776f9a40582c9435c06cf5e2cf83b3313f0f81/yadcc/daemon/cloud/compiler_registry.cc#L72

    opened by cyqw 2
  • Fail to build yadcc

    Fail to build yadcc

    yadcc/build64_release/thirdparty/opentracing-cpp/opentracing-cpp-1.5.1/3rd_party/include/opentracing/catch2/catch.hpp:6631:45: error: size of array ‘altStackMem’ is not an integral constant-expression 6631 | char FatalConditionHandler::altStackMem[SIGSTKSZ] = {};

    glibc changed SIGSTKSZ to be a syscall instead of being constant. #catchorg/Catch2#2178

    opened by Domain 1
  • 关于调度机制的问题

    关于调度机制的问题

    我架了4个编译节点, A:50核,B:50核, C:20核, D:20核。

    当我make -j80的时候, A:50核占满,B:30核,C,和D,偶尔会有占用一两核。 C和D,在这种情况下,大部分时候没有编译任务。

    想问下,有没有方案,让调度更平均一点,当make -j80的时候,按比例分配到节点 比如: A:28, B:28, C:12 D:12

    opened by gamark 1
  • Cannot contact delegate daemon. Daemon died?

    Cannot contact delegate daemon. Daemon died?

    我按技术文档配置,调度机和客户机都各自启动了scheduler和daemon,调度机检测到新的servant。

    I1110 14:19:54.431716 1780984 init.cc:114] Flare started.
    I1110 14:19:54.432881 1780984 runtime.cc:425] Using fiber scheduling profile [neutral].
    I1110 14:19:54.432904 1780984 runtime.cc:220] Starting 8 worker threads per group, for a total of 1 groups. The system is treated as UMA.
    I1110 14:19:54.436939 1780984 init.cc:122] Flare runtime initialized.
    I1110 14:19:54.702870 1780997 task_dispatcher.cc:205] [192.168.33.5:46518] Discovered new servant at [192.168.33.5:8335]
    

    但是运行YADCC_LOG_LEVEL=0 CXX='/home/ubuntu/.yadcc/symlinks/g++' LD='/home/ubuntu/.yadcc/symlinks/g++' CC='/home/ubuntu/.yadcc/symlinks/gcc' make -j8 提示

    [2022-11-10 14:20:33.068854] [TRACE] [yadcc/client/yadcc.cc:261] Started
    [2022-11-10 14:20:33.068886] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/home/ubuntu/.yadcc/symlinks].
    [2022-11-10 14:20:33.068912] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/usr/local/sbin].
    [2022-11-10 14:20:33.068919] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/usr/local/bin].
    [2022-11-10 14:20:33.068926] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/usr/sbin].
    [2022-11-10 14:20:33.068932] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/usr/bin].
    [2022-11-10 14:20:33.068942] [TRACE] [yadcc/client/utility.cc:99] Found [gcc] at [/usr/bin].
    [2022-11-10 14:20:33.068947] [TRACE] [yadcc/client/yadcc.cc:133] Using compiler: /usr/bin/gcc
    [2022-11-10 14:20:33.068980] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    [2022-11-10 14:20:34.064286] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    [2022-11-10 14:20:34.065163] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    [2022-11-10 14:20:34.068817] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    [2022-11-10 14:20:34.069060] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    [2022-11-10 14:20:35.064416] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    [2022-11-10 14:20:35.065259] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    [2022-11-10 14:20:35.068913] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    [2022-11-10 14:20:35.069152] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
    

    请问可能是什么问题?

    opened by Crispig 9
  • 对于跨操作系统的分布式编译应该怎么做

    对于跨操作系统的分布式编译应该怎么做

    机器1:client(user,且不不接受任务):ubuntu1804,gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 机器2:servant(专用编译机):ubuntu2004,gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0。 servant上支持的gcc-7的哈希值与user需要的不一致,对于这种情况有什么建议吗?ubuntu2004能编译出来和ubuntu1804哈希值一样的gcc吗?你们对于这样的场景,你们是怎么做的?

    opened by lijiaonothing 2
  • Can i Preprocess with PCH and compile the job remotely?

    Can i Preprocess with PCH and compile the job remotely?

    As official doc describes, PCH can be used in preprocess, and insert "#pragma GCC pch_preprocess "filename"" into the output. Furthermore, the path can be absolute or relative to the working dir. That's to say, it is possible to use PCH to achieve distribute build, which could improve the compilation efficiency. And i have skipped your code and found during the preparation, yadcc skips the PCH and just use -E, -fno-working-directory and -fdirectives-only to produce preprocess output. Now, i have two questions and wish your reply.

    1、As i knew, -E option just preprocess the included headers directly without PCH, and should takes more time not only during the preprocess but also during the compilation. So, have u attemp to use PCH to achieve distributely build in yadcc, and if so, why not use it?

    2、As gcc doc describes, -fworking-directory is default setting. However, yadcc turns it off and user -fno-working-directory, and some debug information like file path will not be inserted into the preprocess output. As i think, the purpose is to avoid absoulte path in the output which may cause missing cache in spite of the same source file. I have tried to handle the preprocess output of msvc compiler, and found without absolute path, cache will be hit if the source file is same in different computer. It helps much to save compilation time, but i can use the results to start executable files but can not use the results obj files or exp files or pdb files to debug if turning off -fworking-directory . Thus, i wonder that have yadcc found similar problems after turning path off.

    Wish your reply, thank you very much.

    opened by chengjiaozyl 5
Owner
Tencent
Tencent
Compiler Design Project: Simulation of front-end phase of C Compiler involving switch-case construct.

CSPC41 Compiler Design Project Assignment Compiler Design Project: Simulation of front-end phase of C Compiler involving switch-case construct. Using

Adeep Hande 1 Dec 15, 2021
distributed builds for C, C++ and Objective C

distcc -- a free distributed C/C++ compiler system by Martin Pool Current Documents: https://distcc.github.io/ Formally http://distcc.org/ "pump" func

distcc 1.8k Dec 27, 2022
A Small C Compiler

8cc C Compiler Note: 8cc is no longer an active project. The successor is chibicc. 8cc is a compiler for the C programming language. It's intended to

Rui Ueyama 5.8k Jan 8, 2023
A C compiler for LLVM. Supports C++11/14/1z C11

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies

LLVM 17.5k Jan 8, 2023
GNU Prolog is a native Prolog compiler

GNU Prolog is a native Prolog compiler with constraint solving over finite domains (FD)

Daniel Diaz 64 Dec 13, 2022
Take your first step in writing a compiler.

first-step Take your first step in writing a compiler. Building from Source Before building first-step, please make sure you have installed the follow

PKU Compiler Course 28 Aug 20, 2022
NCC is an ANSI/ISO-compliant optimizing C compiler.

The compiler is retargetable by design, but, at present, it only produces binaries for Linux/x86_64. As the compiler ABI differs somewhat from the System V ABI used by Linux, its code cannot be linked against Linux system libraries. It does, however, provide its own (incomplete) standard ANSI/Posix C library.

Charles E. Youse 0 Apr 1, 2022
nanoc is a tiny subset of C and a tiny compiler that targets 32-bit x86 machines.

nanoc is a tiny subset of C and a tiny compiler that targets 32-bit x86 machines. Tiny? The only types are: int (32-bit signed integer) char (8-

Ajay Tatachar 19 Nov 28, 2022
Smaller C is a simple and small single-pass C compiler

Smaller C is a simple and small single-pass C compiler, currently supporting most of the C language common between C89/ANSI C and C99 (minus some C89 and plus some C99 features).

Alexey Frunze 1.2k Jan 7, 2023
Microvm is a virtual machine and compiler

The aim of this project is to create a stack based language and virtual machine for microcontrollers. A mix of approaches is used. Separate memory is used for program and variable space (Harvard architecture). An interpreter, virtual machine and compiler are available. A demostration of the interpreter in action is presented below.

null 10 Aug 14, 2022
Aheui JIT compiler for PC and web

아희짓 개요 아희짓은 아희 언어를 위한 JIT (Just in Time) 컴파일러입니다. 어셈블러와 유틸 라이브러리외에 외부 라이브러리에 전혀 의존하지 않고 JIT을 바닥부터 구현합니다. 지원 환경 64비트 windows, mac, linux (x86 아키텍쳐) 웹어셈

Sunho Kim 28 Sep 23, 2022
C implementation of the Tiny BASIC compiler found in an article by Dr. Austin Henley

Teeny Tiny Basic A C implementation of the Tiny BASIC compiler found in this article and this github repo by Dr. Austin Henley. I did pretty well in A

Gavin Morris 7 Oct 4, 2022
bcc is an interactive compiler of a language called b.

bcc is an interactive compiler of a language called b.

kparc 18 Nov 7, 2022
This is a compiler written from scratch in C

C Compiler This is a compiler written from scratch in C, with fully supporting C18 as a goal. It can currently compile itself, and most simple program

null 29 Jan 6, 2023
mrcceppc is a reimplementation project for the Metrowerks mwcceppc compiler.

Compiler | mrcceppc mrcceppc is a reimplementation project for the Metrowerks mwcceppc compiler. Compiling Run generate_{version}.bat for which versio

null 9 Nov 21, 2022
A C header that allow users to compile brainfuck programs within a C compiler.

brainfuck.h A C header that allow users to compile brainfuck programs within a C compiler. You can insert the header into the top of your brainfuck so

null 1 Dec 30, 2022
Gilbraltar is a version of the OCaml compiler to be able to build a MirageOS for RaspberryPi 4.

Gilbraltar is a version of the OCaml compiler to be able to build a MirageOS for RaspberryPi 4. It's a work in progress repository to provide a dune's toolchain (as ocaml-freestanding) specialized for Raspberry Pi 4.

Calascibetta Romain 49 Oct 30, 2022
NaiveCC: a compiler frontend for a subset of C

NaiveCC: a compiler frontend for a subset of C This is a toy compiler frontend for a subset of the C programming language based on the LR(1) parsing t

Yuxiang Wei 1 Nov 15, 2021
A LLVM and Clang compiler toolchain built for kernel development

Cosmic-Clang Toolchain This is a LLVM and Clang compiler toolchain built for kernel development. Builds are always made from the latest LLVM sources r

Ǥђ๏ຮ₮⌁Ⲙครtє࿐ 0 Apr 12, 2022