> ~/Downloads/ClangBuildAnalyzer-mac --start clang_analyzer
Build tracing started. Do some Clang builds with '-ftime-trace', then run 'ClangBuildAnalyzer --stop clang_analyzer <filename>' to stop tracing and save session to a file.

> ~/Downloads/ClangBuildAnalyzer-mac --stop clang_analyzer clang_analyzer_report
Stopping build tracing and saving to 'clang_analyzer_report'...
ERROR: no .json files found under 'clang_analyzer'.

macOS Catalina 10.15.5 (19F101)

ClangBuildAnalyzerTest.zip

More often than not, I need to analyze the whole artifacts directory. Making session start step optional means that I can achieve this goal easier and no extra files need to be created in the process.

Unfortunately, when using with our project, I get Segmentation fault (core dumped) during --stop. I could provide you with the data you need, but I don't exactly know what to attach.

Our project is quite small - less than 100K LOC, number of files is not big either.

I noticed in htop, that the utility starts to use a lot of RAM, maybe this could help.

Looking at an older issue https://github.com/aras-p/ClangBuildAnalyzer/issues/45 I'm guessing LLVM clang 13 is not identified by this tool.

The error is

Stopping build tracing and saving to '1'... ERROR: no clang -ftime-trace .json files found under './'.

Clang version:

clang version 13.0.0 Target: x86_64-apple-darwin21.2.0 Thread model: posix

Example json file is :

{
    "traceEvents": [
        {
            "pid": 16764,
            "tid": 259,
            "ph": "X",
            "ts": 6343,
            "dur": 831,
            "name": "Source",
            "args": {
                "detail": "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include/sys/cdefs.h"
            }
        },
        {
            "pid": 16764,
            "tid": 259,
            "ph": "X",
            "ts": 3786,
            "dur": 242254,
            "name": "Frontend"
        },
        {
            "pid": 16764,
            "tid": 259,
            "ph": "X",
            "ts": 125,
            "dur": 258862,
            "name": "ExecuteCompiler"
        },
        {
            "pid": 16764,
            "tid": 260,
            "ph": "X",
            "ts": 0,
            "dur": 258861,
            "name": "Total ExecuteCompiler",
            "args": {
                "count": 1,
                "avg ms": 258
            }
        },
        {
            "pid": 16764,
            "tid": 261,
            "ph": "X",
            "ts": 0,
            "dur": 242253,
            "name": "Total Frontend",
            "args": {
                "count": 1,
                "avg ms": 242
            }
        },
        {
            "pid": 16764,
            "tid": 262,
            "ph": "X",
            "ts": 0,
            "dur": 217862,
            "name": "Total Source",
            "args": {
                "count": 28,
                "avg ms": 7
            }
        },
        {
            "pid": 16764,
            "tid": 263,
            "ph": "X",
            "ts": 0,
            "dur": 2,
            "name": "Total PerformPendingInstantiations",
            "args": {
                "count": 1,
                "avg ms": 0
            }
        },
        {
            "cat": "",
            "pid": 16764,
            "tid": 259,
            "ts": 0,
            "ph": "M",
            "name": "process_name",
            "args": {
                "name": "clang"
            }
        },
        {
            "cat": "",
            "pid": 16764,
            "tid": 259,
            "ts": 0,
            "ph": "M",
            "name": "thread_name",
            "args": {
                "name": ""
            }
        }
    ],
    "beginningOfTime": 1641965546191608
}

The analyzer can't read large json files due to this code:

diff --git a/src/main.cpp b/src/main.cpp
index 380b26f..fdfafbc 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -26,7 +26,7 @@ static std::string ReadFileToString(const std::string& path)
     if (!f)
         return "";
     fseek(f, 0, SEEK_END);
-    size_t fsize = ftell(f);
+    size_t fsize = _ftelli64(f);
     fseek(f, 0, SEEK_SET);
     std::string str;
     str.resize(fsize);

Maybe it should just use memory-mapped files.

Recently, when trying to use ClangBuildAnalyzer to analyze a firefox build, I got the following error and assertion failure on ClangBuildAnalyzer --stop:

ERROR: Failed to open directory (obj-x86_64-pc-linux-gnu-trace/dist/bin/chrome/en-US/locale/en-US/mozapps/downloads/settingsChange.dtd): No such file or directory.
ClangBuildAnalyzer: ../src/external/cute_files.h:427: int cf_dir_open(cf_dir_t*, const char*): Assertion `0' failed.

obj-x86_64-pc-linux-gnu-trace/dist/bin/chrome/en-US/locale/en-US/mozapps/downloads/settingsChange.dtd exists, but is a symbolic link to a file, not a directory:

[[email protected] fuzzy]$ stat obj-x86_64-pc-linux-gnu-trace/dist/bin/chrome/en-US/locale/en-US/mozapps/downloads/settingsChange.dtd
  File: obj-x86_64-pc-linux-gnu-trace/dist/bin/chrome/en-US/locale/en-US/mozapps/downloads/settingsChange.dtd -> /home/simon/work/fuzzy/toolkit/locales/en-US/chrome/mozapps/downloads/settingsChange.dtd
  Size: 88              Blocks: 8          IO Block: 4096   symbolic link
Device: fd02h/64770d    Inode: 34560445    Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/   simon)   Gid: ( 1000/   simon)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2020-10-21 18:42:33.948517287 +0200
Modify: 2020-10-14 18:48:03.012265746 +0200
Change: 2020-10-14 18:48:03.012265746 +0200
 Birth: 2020-10-14 18:48:03.012265746 +0200
[[email protected] fuzzy]$ stat /home/simon/work/fuzzy/toolkit/locales/en-US/chrome/mozapps/downloads/settingsChange.dtd
  File: /home/simon/work/fuzzy/toolkit/locales/en-US/chrome/mozapps/downloads/settingsChange.dtd
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: fd02h/64770d    Inode: 49026953    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/   simon)   Gid: ( 1000/   simon)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2020-10-22 09:43:44.048022446 +0200
Modify: 2020-10-22 09:43:34.870015064 +0200
Change: 2020-10-22 09:43:34.870015064 +0200
 Birth: 2020-10-22 09:43:29.366010639 +0200

I am not able to get simple trace running successfully through this tool. From a freshly cloned and built tree, I run:

# ./ClangBuildAnalyzer  --start /tmp
Build tracing started. Do some Clang builds with '-ftime-trace', then run 'ClangBuildAnalyzer --stop /tmp <filename>' to stop tracing and save session to a file.
# clang <various options...> -o /tmp/main.cc.o -c /tmp/main.cc -ftime-trace -ftime-report
ls /tmp
# ls /tmp/main*
/tmp/main.cc.json  /tmp/main.cc.o
# ./ClangBuildAnalyzer --stop /tmp /tmp/report
Stopping build tracing and saving to '/tmp/report'...
ERROR: no clang -ftime-trace .json files found under '/tmp'.

Seems to find the json file and fail while parsing it (within ParseBuildEvents in the code). The clang version is 11/head, downloaded and built today. Chrome can read and display the file successfully within chrome://tracing.

Any ideas? My apologies if I'm missing something obvious, and thank you for producing this tool and -ftime-trace. Incredibly useful.

Sometimes, you don't just want to know the most expensive instantiation of a template (i.e. tuple<int, float>), you want to know which templates as a whole are expensive (i.e. tuple). This new analysis does that.

Example output:

**** Template sets that took longest to instantiate: 175188 ms: std::tuple<$> (17428 times, avg 10 ms) 70354 ms: testing::internal::FunctionMocker<$>::Invoke (6166 times, avg 11 ms) 61916 ms: testing::internal::FunctionMocker<$> (4371 times, avg 14 ms) 61186 ms: testing::internal::FunctionMockerBase<$> (4017 times, avg 15 ms) 59616 ms: testing::internal::MockSpec<$> (3936 times, avg 15 ms)

Similarly, you don't always want to know which specific function took a long time to optimize, but which collection of template functions. Example output:

**** Function sets that took longest to compile / optimize: 16710 ms: public: static void __cdecl testing::internal::TuplePrefix<$>::ExplainMatchFailuresTo<$>(class std::tuple<$> const &, class std::tuple<$> const &, class std::basic_ostream<$> *) (6484 times, avg 2 ms) 11430 ms: private: void __thiscall testing::internal::TypedExpectation<$>::ExplainMatchResultTo(class std::tuple<$> const &, class std::basic_ostream<$> *) const (2999 times, avg 3 ms) 11161 ms: private: class testing::Action<$> const & __thiscall testing::internal::TypedExpectation<$>::GetCurrentAction(class testing::internal::FunctionMockerBase<$> const *, class std::tuple<$> const &) const (2999 times, avg 3 ms) 5071 ms: public: static bool __cdecl testing::internal::TypeParameterizedTest<$>::Register(char const *, struct testing::internal::CodeLocation, char const *, char const *, int) (999 times, avg 5 ms) 4025 ms: private: virtual class testing::internal::ExpectationBase const * __thiscall testing::internal::FunctionMockerBase<$>::UntypedFindMatchingExpectation(void const *, void const **, bool *, class std::basic_ostream<$> *, class std::basic_ostream<$> *) (2999 times, avg 1 ms)

I'm attempting to profile fuchsia compilations and am running into:

Analyzing build trace from '/tmp/capture-file'...
ERROR: corrupt input file '/tmp/capture-file' (checksum mismatch)

Do you have any ways of going about debugging this? This is from a fresh ClangBuildAnalyzer checkout and there didn't seem to be any issues with building or running it. I'm able to assert that -ftime-trace is working as intended since I can see the .json files in my build. The capture file I also made is about 470 MB.

since add_executable is used without sources the minimum required version has to be 3.11. see: https://cmake.org/cmake/help/v3.11/release/3.11.html#commands

With code like

template<typename T, unsigned N>
void foo(typename std::enable_if<(N < sizeof(T)), T>::type argument) {}

void bar() { foo<int, 2>(3); }

the parsing in collapseName breaks down, because < and > no longer match up. Then we go into retval.append(pos, new_pos); with pos > new_pos, which causes a std::length_error to be thrown in libstdc++.

Note that while C++11 syntax requires parantheses around comparisons in template arguments (like in the code above), they don't seem to appear in the string that we have here, which looks like std::enable_if<(2u) < (sizeof(int)), int>::type.

I have a case where I have a bunch of templated classes that have potentially expensive operator() implementations. However, due to how we collapse names, we never collapse any operators at all and we don't end up grouping these when highlight expensive template instantiation patterns. Relevant code is here:

...
static std::string_view CollapseName(const std::string_view& elt)
{
    // Parsing op<, op<<, op>, and op>> seems hard.  Just skip'm all
    if (elt.find("operator") != std::string::npos)
      return elt;
...

I'm wondering if we can change this to only prevent collapsing if we find either 'operator<' or 'operator>' in the name so that we can surface something like this.

Right now, you get an error when trying to use --start with a folder name that does not exist:

❯ clangbuildanalyzer --start artifacts
ERROR: failed to create session file at 'artifacts/ClangBuildAnalyzerSession.txt'.

Creating a folder manually solves this, but I think it should be done automatically (ideally recursively, so that it works when multiple folders need to be created):

~/Documents/Git/godotengine/godot master
❯ mkdir artifacts                                            

~/Documents/Git/godotengine/godot master
❯ clangbuildanalyzer --start artifacts
Build tracing started. Do some Clang builds with '-ftime-trace', then run 'ClangBuildAnalyzer --stop artifacts <filename>' to stop tracing and save session to a file.

ClangBuildAnalysis is a great tool and it has really helped me improve the build time of a large project I am working on.

My project is so large that the code crashes when loading the json files. I believe it is running out of memory while parsing the json dom. my project has 83GB of json files. I have run ClangBuildAnalyzer on all the files and I can analyze all of them if I do it in two groups, with the 51 largest json files in one group and the other 241 files in the other group.

While debugging, I changed the max tasks to 1 in main.cpp, hoping that parsing 1 json at a time would reduce the memory footprint. It may have gotten farther, but ClangBuildAnalyzer still crashed.

I tried using gdb to see the crash, but the app is killed due to out of memory, instead of core dumping, so gdb didn't give me any useful information.

I wondered if there was a memory leak or other bug in the json parsing. I manually copied the latest release of simdjson (0.9.4) into ClangBuildAnalyzer and rebuilt. I still got the same crashes.

I am currently using clang-11. I had more luck with ClangBuildAnalyzer on this project in the past and may have been using clang-10. I wonder if clang-10 writes fewer events? I do not see any warnings about unknown events while running ClangBuildAnalyzer with clang-11.

Could ClangBuildAnalyzer add a batch mode to support large projects? For example, I could run ClangBuildAnalyzer -all <subdirN> on the N subdirs of my build. Then, I would run ClangBuildAnalyzer --analyze with N file names instead of 1. ClangBuildAnalyzer would then read all N binary event files and concatenate all the data together before producing the statistics over the combined set of all events.

I have studied the LoadBuildEvents() method and believe it could be possible to have it append new files to the current event and names lists, but haven't had time to do it yet, especially making the event to name index is updated correctly.

I mistakenly passed my artefacts folder path to the "--analize" arg and ClangBuildAnalyzer crashed with std::bad_alloc from BufferedReader(FILE* f) when "fsize" was calculated as 9223372036854775807. I think it's better to add some additional checks for stupid errors like this. At least need to check that path is valid and it points to a file. Also, errors handling when working with fseek, ftello64 etc is completely skipped and ftello64 even returns signed int64_t and it also wasn't checked before being implicitly cast to size_t.

I changed this to build with MinGW:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 71f708f..56223a3 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -22,4 +22,4 @@ set(SRC
 )
 add_executable(ClangBuildAnalyzer "${SRC}")
 target_compile_features(ClangBuildAnalyzer PRIVATE cxx_std_17)
-target_link_libraries(ClangBuildAnalyzer -lrt -lpthread)
+target_link_libraries(ClangBuildAnalyzer -lpthread)
diff --git a/src/main.cpp b/src/main.cpp
index 3ef1fc1..68e46d6 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -96,7 +96,7 @@ static int RunStart(int argc, const char* argv[])
     return 0;
 }
 
-#ifdef _MSC_VER
+#ifdef _WIN32
 static time_t FiletimeToTime(const FILETIME& ft)
 {
     ULARGE_INTEGER ull;
@@ -128,7 +128,7 @@ struct JsonFileFinder
         if (!cf_get_file_time(f->path, &mtime))
             return;
         time_t fileModTime;
-#ifdef _MSC_VER
+#ifdef _WIN32
         fileModTime = FiletimeToTime(mtime.time);
 #else
         fileModTime = mtime.time;

This tool was very helpful in optimizing build times. However I would to be able to analyze the output a little more thoroughly. For example, how much time out of total is taken by each header or translation unit. It would greatly help with cost-benefit analysis when attempting any kind of refactoring. To do that I need raw aggregated data of the build analysis. Is it possible to get it from the tool at the moment. If not how hard would it be to add and where to look to do that?