CMake checks cache helper modules – for fast CI CMake builds!

Overview

cmake-checks-cache

Cross platform CMake projects do platform introspection by the means of "Check" macros. Have a look at CMake's How To Write Platform Checks wiki page for a detailed explanation.

There are quite a few of them:

All these checks will create small CMake project which will try to compile a C or C++ program, and based on the success of the compilation will set a CMake cache variable.

Usually in a Continuous Integration (CI) Build the compilation starts from scratch with a an empty build folder. Also the CI-Build is also set up using a compilation cache for the C / C++ code e.g. ccache, clcache etc.

Depending how the compilation cache is set up they might also speed up the CMake Check macros.

But what if we can cache CMake Check results? This is where cmake-checks-cache comes into action!

Creating the cmake_checks_cache.txt file

One needs to run CMake with -DCMAKE_MODULE_PATH pointing out to the CMakeChecksCache checkout! That was it!

In the build folder you will get a file named cmake_checks_cache.txt which contains all the trapped Check macro calls and all the CMAKE_* cache variables.

The part with the CMAKE_* cache variables is needed for skipping the initial CMake platform compiler / linker / assembler introspection. Please note that you need to copy also the CMakeFiles/ /CMake*.cmake files.

On Windows make sure that you use absolute paths with slashes for -DCMAKE_MODULE_PATH argument. Otherwise you will get some nasty CMake errors.

LLVM + clang example

The usual CMake command line looks like this (in a Visual C++ 2017 command prompt):

$ cmake -G "Ninja" -DCMAKE_BUILD_TYPE=Release ../llvm-6.0.0.src

For caching the CMake checks you should run CMake like this:

$ cmake -G "Ninja" -DCMAKE_BUILD_TYPE=Release ../llvm-6.0.0.src -DCMAKE_MODULE_PATH=c:/Projects/CMakeChecksCache

The files that you need to keep would be:

.
├── cmake_checks_cache.txt
└── CMakeFiles
    └── 3.11.0
        ├── CMakeASMCompiler.cmake
        ├── CMakeCCompiler.cmake
        ├── CMakeCXXCompiler.cmake
        ├── CMakeRCCompiler.cmake
        └── CMakeSystem.cmake

For using the CMake checks in a an empty folder (with the above files copied) you should run CMake like this:

$ cmake -G "Ninja" -DCMAKE_BUILD_TYPE=Release -C cmake_checks_cache.txt ../llvm-6.0.0.src

CMake documentation describes -C command line parameter like this:

-C

Pre-load a script to populate the cache.

When cmake is first run in an empty build tree, it creates a CMakeCache.txt file and populates it with customizable settings for the project. This option may be used to specify a file from which to load cache entries before the first pass through the project’s cmake listfiles. The loaded entries take priority over the project’s default values. The given file should be a CMake script containing SET commands that use the CACHE option, not a cache-format file.

Benchmark

I have ran the benchmarks on Windows 10 on a Lenovo W510 Laptop in a Visual C++ 2017 x64 command prompt.

LLVM + Clang 6.0.0 Ninja MSBuild
CMake first run (empty folder) 74.085 s 158.683 s
CMake second run (already configured) 26.196 s 45.564 s
CMake generating cmake_checks_cache.txt 71.098 s 149.46 s
CMake using cmake_check_cache.txt 26.792 s 45.01 s

For the Ninja benchmarks I have used the above given CMake calls. For MSBuild I simply omitted -G "Ninja" from the command lines, letting CMake use the default introspection for the Visual C++ 2017 x64 command prompt.

The reason that MSBuild is slower is that for every Check there is a small Visual Studio (MSBuild) solution created and then msbuild is ran for it. msbuild is written in C# and it has to use the .NET platform.

MSBuild generation is slower because it generates multiple variants x86, x64, Debug, and Release. See the CMake's Visual Studio 15 2017, respectively CMAKE_CONFIGURATION_TYPES, documentation for additional tweaks. I just wanted to keep things simple and use the defaults.

The generation of the cmake_checks_cache.txt is faster than the first CMake run because the operating system has done some resource caching of itself.

Blog entry

For more rationale and benchmarks have a look at my Speeding up CMake blog article.

Since I wrote the blog article I have comitted a few bugfixes to CMake, hence the need of CMake version 3.11.

Comments
  • Usage in a non-empty directory

    Usage in a non-empty directory

    Hi! I am trying to use cmake-checks-cache to improve the cmake time of subsequent LLVM configurations (im building LLVM in WSL and cmake is painfully slow there). That means that after the first run, I am running cmake in a build directory that already contains a CMakeCache and also a cmake_checks_cache.txt. The documentation only mentions an empty folder though:

    For using the CMake checks in a an empty folder (with the above files copied) you should run CMake like this:

    Does the same apply for an already populated folder for subsequent cmake runs? I.e. would I also have to use $ cmake -G "Ninja" -DCMAKE_BUILD_TYPE=Release -C cmake_checks_cache.txt ../my-llvm-source despite the fact that the build directory contains CMake configurations from a previous run?

    Thanks for sharing cmake-checks-cache!

    opened by fberlakovich 2
  • [Suggest] add  WIN32_LEAN_AND_MEAN , NOMINMAX , NO... to Avoid  effects of Windows.h

    [Suggest] add WIN32_LEAN_AND_MEAN , NOMINMAX , NO... to Avoid effects of Windows.h

    In the Windows.h header, if WIN32_LEAN_AND_MEAN is not defined, the preprocessor will includes other headers. So if you want to not include theses headers, you must define WIN32_LEAN_AND_MEAN before #include , else it won't have any effects

    https://docs.microsoft.com/en-us/windows/win32/winprog/using-the-windows-headers

    opened by Esp9527 1
  • `-DCMAKE_MODULE_PATH` no longer works

    `-DCMAKE_MODULE_PATH` no longer works

    With a recent version of LLVM on Windows, adding -DCMAKE_MODULE_PATH to the command-line has no effect. However, placing the following in the llvm/CMakeLists.txt file does generate the cmake_checks_cache.txt file as expected:

    list(INSERT CMAKE_MODULE_PATH 0 "${CMAKE_CURRENT_SOURCE_DIR}/../cmake-checks-cache/CMakeChecksCache")
    

    (Obviously this is dependent on where the cmake-checks-cache repo is cloned to.)

    CMake version: 3.15.1

    opened by cameron314 1
  • Customizable cache file and minor changes

    Customizable cache file and minor changes

    Use the CMAKE_CHECKS_CACHE_FILE variable to set the location of the cache checks file. Escape the quotes if they are present in a variable (some CFLAGS might contain them for example). Added wrappers for check_cxx_symbol_exists and test_big_endian

    opened by pedronavf 1
  • that will CheckTypeSize fail if more than 2 sub module to check CheckTypeSize

    that will CheckTypeSize fail if more than 2 sub module to check CheckTypeSize

    `include_guard(GLOBAL)

    cmake_minimum_required(VERSION 3.11)

    include(DumpCMakeVariables)

    include(${CMAKE_ROOT}/Modules/CheckTypeSize.cmake)`

    maybe need move "include(${CMAKE_ROOT}/Modules/CheckTypeSize.cmake)" to the front of "include_guard(GLOBAL)"

    opened by Esp9527 0
  • What does the workflow for CI look like when you need to regenerate the cache due to changes?

    What does the workflow for CI look like when you need to regenerate the cache due to changes?

    I'd love to use this, but I'm not sure how it's meant to work. If I generate the cache and then reuse it for all future builds, what happens when something is added/removed/changed to cmake that means regenerating the cache? is there some way of detecting that and generating the cache again? or just always generate the cache per-pipeline-per-os? could 32/64/arm builds result in different caches?

    e.g. CheckTypeSize what if the size of the type changed since it was last cached?

    when the majority of time is spent in FetchContent, does this help at all?

    opened by Zephilinox 2
Owner
Cristian Adam
Cristian Adam
CMake module to speed up builds.

cotire Cotire (compile time reducer) is a CMake module that speeds up the build process of CMake based build systems by fully automating techniques as

Sascha Kratky 1.3k Dec 26, 2022
Tundra is a code build system that tries to be accurate and fast for incremental builds

Tundra, a build system Tundra is a high-performance code build system designed to give the best possible incremental build times even for very large s

Andreas Fredriksson 400 Dec 23, 2022
CMake modules to help use sanitizers

sanitizers-cmake CMake module to enable sanitizers for binary targets. Include into your project To use FindSanitizers.cmake, simply add this reposito

Matt Arsenault 295 Dec 23, 2022
CMake Community Modules

CMake Community Modules The CMake Community Modules are CMake modules created and maintained by the community. This repository's main product is the C

Edgar 2 Dec 14, 2022
Extra CMake Modules for YARP and friends

YCM Extra CMake Modules for YARP and friends Documentation Online documentation is available

Robotology 43 Dec 27, 2022
Extra modules and scripts for CMake.

Extra CMake Modules Introduction The Extra CMake Modules package, or ECM, adds to the modules provided by CMake, including ones used by find_package()

KDE GitHub Mirror 104 Nov 16, 2022
CMake modules for some scientific libraries

A collection of CMake modules, which can mostly be used independently. The utilities for writing robust Find* modules might be useful until CMake take

Jed Brown 83 Dec 4, 2022
Common CMake modules

CMake Modules This repository contains common CMake modules and a collection of find scripts to locate non-CMake dependencies. The recommended way to

Eyescale Software GmbH 144 Dec 27, 2022
Additional CMake functionality. Most of the modules are from Ryan Pavlik

Additional CMake functionality. Most of the modules are from Ryan Pavlik

Lars Bilke 452 Dec 28, 2022
Ryan's CMake Modules Collection

This is a collection of CMake modules that I've produced during the course of a variety of software development.

Ryan A. Pavlik 891 Jan 9, 2023
A set of cmake modules to assist in building code

CMake 3 Tools Warning: These tools are being replaced by the Modern CMake. Some of the tools are still being maintained for now, but new projects shou

null 201 Dec 21, 2022
A fast build system that encourages the creation of small, reusable modules over a variety of platforms and languages.

Buck Buck is a build tool. To see what Buck can do for you, check out the documentation at http://buck.build/. Installation Since Buck is used to buil

Facebook 8.5k Jan 7, 2023
📦 CMake's missing package manager. A small CMake script for setup-free, cross-platform, reproducible dependency management.

Setup-free CMake dependency management CPM.cmake is a CMake script that adds dependency management capabilities to CMake. It's built as a thin wrapper

CPM.cmake 1.6k Jan 9, 2023
CMake scripts for painless usage of SuiteSparse+METIS from Visual Studio and the rest of Windows/Linux/OSX IDEs supported by CMake

CMake scripts for painless usage of Tim Davis' SuiteSparse (CHOLMOD,UMFPACK,AMD,LDL,SPQR,...) and METIS from Visual Studio and the rest of Windows/Lin

Jose Luis Blanco-Claraco 395 Dec 24, 2022
cmake-font-lock - Advanced, type aware, highlight support for CMake

cmake-font-lock - Advanced, type aware, highlight support for CMake

Anders Lindgren 39 Oct 2, 2022
cmake-avr - a cmake toolchain for AVR projects

cmake-avr - a cmake toolchain for AVR projects Testing the example provided The toolchain was created and tested within the following environment: Lin

Matthias Kleemann 163 Dec 5, 2022
Make CMake less painful when trying to write Modern Flexible CMake

Izzy's eXtension Modules IXM is a CMake library for writing Modern flexible CMake. This means: Reducing the amount of CMake written Selecting reasonab

IXM 107 Sep 1, 2022
CMake module to enable code coverage easily and generate coverage reports with CMake targets.

CMake-codecov CMake module to enable code coverage easily and generate coverage reports with CMake targets. Include into your project To use Findcodec

HPC 82 Nov 30, 2022
[CMake] [BSD-2] CMake module to find ICU

FindICU.cmake A CMake module to find International Components for Unicode (ICU) Library Note that CMake, since its version 3.7.0, includes a FindICU m

julp 29 Nov 2, 2022