Hi,

in our team which is gladly adopting FASTBuild via the VS-generated projects more and more we also often use non-VC++ projects our VS Installation, augmented with third party extensions understands (for example Python, WiX toolset to name only two).

If we want to have those in the same solution as the one generated by FASTBuild, the ~~VCXProject~~ new VSProjectExternal function needs to be able to reference external, existing projects without overwriting them. This pull request adds exactly this enhancement.

To use it, ~~the correct project GUID, as well as~~ the correct project type GUID (necessary for VS to correctly load those projects from the generated solution) ~~have~~ has to be specified, ~~and nothing more but the external flag~~ as in the example below:

-VCXProject( 'ExtProject-vsproj' )
+VSProjectExternal( 'ExtProject-vsproj' )
{
-    .ProjectIsExternal = true
-    .ProjectOutput     = 'Setup\Source\ExtProject\ExtProject.wixproj'
+    .ExternalProjectPath     = 'Setup\Source\ExtProject\ExtProject.wixproj'
-    .ProjectGuid       = '{32aa1ad4-da90-48b6-8d09-84bc693734d2}'
    .ProjectTypeGuid   = '{930c7802-8a8c-48f9-8165-68863bccd9dd}'       // set the correct WiX Toolset project type GUID
}

• ~~ProjectOutput~~ ExternalProjectPath matches the actual project file just ~~as in the case of generated (the default .ProjectIsExternal = false) projects~~ which has to exist;

~~- ProjectGuid becomes mandatory and should match the actual project GUID of the referenced Project (should - because FASTBuild will not check the match, but the resulted solution will be inconsistent and VS will not be too happy loading it);~~

• ProjectTypeGuid should actually be provided ~~unless the Project is of the type VC++ which FASTBuild already knows internally~~.

Best regards, Lucian

I guess this is by design, but I was wondering what the reasoning is behind the design decision to store the "timestamp of last built" and the dependencies found for each node into the serialized depgraph, instead of in a different file. If kept in a different file, I think you would no longer be in the situation that any .bff file change always causes a rebuild of everything

We're in the process of upgrading from FastBuild 0.93 to 0.96, and I'm running into a strange issue, which I haven't debugged or profiled yet, but maybe someone else will have seen it and can save me some time.

I have 72 hardware threads on this machine. Normally an entire FastBuild All with around 7000 object files takes ~2 minutes and change.

Object : 3595 73 0 73 73 1h:28m 19.713s

Here, I sat and watched it idle at about 1% cpu usage for a minute and a half...

I deleted fbuild.cache and tried again, same behavior.

I'll try to report more as I investigate (and have time to investigate)...

I finally managed to get compilation (https://ci.appveyor.com/project/jairbubbles/fastbuild) and analysis (https://sonarqube.com/overview?id=fastbuild) working.

I also put badges on the readme to look modern and cool ;-)

A few notes:

• Clang was failing and I didn't spend time to investigate. It's only x86 / x64 for now.
• Pull request with fast analysis should be working now. See here for more infos http://docs.sonarqube.org/display/PLUG/GitHub+Plugin
• Switching SDKs is not convenient and I had to update main bff. On a different project I had the issue too and I ended up creating a fbuild_vs2012.bff which was not very pretty. What would be great is to be able to switch between sub scripts that only change SDK, includes, libs and compiler files. These sub scripts could be shared between projects for people who are starting with FB.
• PowerShell is a pain in the ass. There are a lot of weird things and I didn't find a simple tutorial so it was a lot of back and forth.
• AppVeyor is really great. You have a full windows image so you can do everything you want.

You need to setup some en var in the AppVeyor settings :

# This assumes that the 2 following variables are defined:
# - SONAR_HOST_URL => should point to the public URL of the SQ server (e.g. : https://sonarqube.com)
# - SONAR_TOKEN    => token of a user who has the "Execute Analysis" permission on the SQ server
# - GITHUB_TOKEN    => token for commenting pull requests in GitHub

Closes #150

We're working from the FastBuild for UE4 wiki (https://github.com/fastbuild/fastbuild/wiki/fastbuild-for-Unreal-Engine-4), which we've modified a bit, but are still struggling to make work properly. We're with a branch of UE4 4.12.5.

We're getting this error:

29>  BFF file 'D:\dev\UE4\UnrealEngine\Engine\Intermediate\Build\fbuild.bff' has changed (reparsing will occur).
29>  D:\dev\UE4\UnrealEngine\Engine\Intermediate\Build\fbuild.bff(23,1): FASTBuild Error #1005 - DLL() - Unsupported node type in 'Libraries'. (Node: 'D:\dev\UE4\UnrealEngine\Engine\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-UnrealEd.lib', Type: 'Exe')
29>  DLL('DLL-2')

And here's the related excerpt from the fbuild.bff file:

DLL('DLL-1')
{
    .Linker                 = '$VSBasePath$/bin/amd64/lib.exe' 
    .LinkerOutput         =  'D:\dev\UE4\UnrealEngine\Engine\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-UnrealEd.lib' 
    .Libraries         = 'D:\dev\UE4\UnrealEngine\Engine\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-UnrealEd.lib.response' 
    .LinkerOptions       = ' /NOLOGO /errorReport:prompt /MACHINE:x64 /SUBSYSTEM:WINDOWS /DEF /NAME:"UE4Editor-UnrealEd.dll" /LIBPATH:"ThirdParty/Ogg/libogg-1.2.2/lib/Win64/VS2015" /LIBPATH:"ThirdParty/Vorbis/libvorbis-1.3.2/Lib/win64/VS2015/" /LIBPATH:"ThirdParty/Vorbis/libvorbis-1.3.2/Lib/win64/VS2015/" /LIBPATH:"ThirdParty/Windows/DirectX/Lib/x64" /LIBPATH:"ThirdParty/HACD/HACD_1.0/lib/Win64/VS2015/" /LIBPATH:"ThirdParty/VHACD/lib/Win64/VS2015/" /LIBPATH:"ThirdParty/FBX/2016.1.1/lib/vs2015/x64/release/" /LIBPATH:"ThirdParty/FreeType2/FreeType2-2.6/Lib/Win64/VS2015" /LIBPATH:"ThirdParty/PhysX/PhysX-3.3/lib/Win64/VS2015" /LIBPATH:"ThirdParty/PhysX/APEX-1.3/lib/Win64/VS2015" /LIBPATH:"../Binaries/ThirdParty/libsndfile/Win64" /NODEFAULTLIB:"LIBCMT" /NODEFAULTLIB:"LIBCPMT" /NODEFAULTLIB:"LIBCMTD" /NODEFAULTLIB:"LIBCPMTD" /NODEFAULTLIB:"MSVCRTD" /NODEFAULTLIB:"MSVCPRTD" /NODEFAULTLIB:"LIBC" /NODEFAULTLIB:"LIBCP" /NODEFAULTLIB:"LIBCD" /NODEFAULTLIB:"LIBCPD" @"%1" /OUT:"%2"'
}
DLL('DLL-2')
{
    .Linker                 = '$VSBasePath$/bin/amd64/link.exe' 
    .LinkerOutput         =  'D:\dev\UE4\UnrealEngine\Engine\Binaries\Win64\UE4Editor-Engine.dll' 
    .Libraries         = { 'DLL-1' } 
    .LinkerOptions       = ' /MANIFEST:NO /NOLOGO /DEBUG /errorReport:prompt /MACHINE:x64 /SUBSYSTEM:WINDOWS /FIXED:No /NXCOMPAT /STACK:5000000 /DELAY:UNLOAD /DLL /PDBALTPATH:%_PDB% /OPT:NOREF /OPT:NOICF /INCREMENTAL:NO /ignore:4199 /ignore:4099 /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"PhysX3PROFILE_x64.dll" /DELAYLOAD:"PhysX3CookingPROFILE_x64.dll" /DELAYLOAD:"PhysX3CommonPROFILE_x64.dll" /DELAYLOAD:"nvToolsExt64_1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"APEXFrameworkPROFILE_x64.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"libogg_64.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"libvorbis_64.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"libvorbisfile_64.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /DELAYLOAD:"vulkan-1.dll" /LIBPATH:"ThirdParty/PhysX/PhysX-3.3/lib/Win64/VS2015" /LIBPATH:"ThirdParty/PhysX/APEX-1.3/lib/Win64/VS2015" /LIBPATH:"D:\dev\UE4\UnrealEngine\Engine\Source\ThirdParty\Box2D\Box2D_v2.3.1\build\vs2015\bin\x64\Release" /LIBPATH:"ThirdParty/Ogg/libogg-1.2.2/lib/Win64/VS2015" /LIBPATH:"ThirdParty/Vorbis/libvorbis-1.3.2/Lib/win64/VS2015/" /LIBPATH:"ThirdParty/Vorbis/libvorbis-1.3.2/Lib/win64/VS2015/" /LIBPATH:"ThirdParty/libOpus/opus-1.1/win32/VS2012/x64/Release/" /LIBPATH:"ThirdParty/DirectShow/DirectShow-1.0.0/Lib/Win64/vs2015/" /NODEFAULTLIB:"LIBCMT" /NODEFAULTLIB:"LIBCPMT" /NODEFAULTLIB:"LIBCMTD" /NODEFAULTLIB:"LIBCPMTD" /NODEFAULTLIB:"MSVCRTD" /NODEFAULTLIB:"MSVCPRTD" /NODEFAULTLIB:"LIBC" /NODEFAULTLIB:"LIBCP" /NODEFAULTLIB:"LIBCD" /NODEFAULTLIB:"LIBCPD" @"D:\dev\UE4\UnrealEngine\Engine\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Engine.dll.response" /OUT:"%2" /IMPLIB:"D:\dev\UE4\UnrealEngine\Engine\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Engine.lib" /PDB:"D:\dev\UE4\UnrealEngine\Engine\Binaries\Win64\UE4Editor-Engine.pdb" /ignore:4078 %1'
}

Is there a better way to go about this? It seems like a DLL node should be able to use a .lib/Library as a dependency. Unfortunately, to make DLL('DLL-1') into a proper Library Node (e.g. Library('LIB-1')) looks like it would expect a compiler, and to actually compile the object files in the same FastBuild step as executing the librarian...

Also, it seems to only not like .lib files within the DLL Node construct because it uses lib.exe instead of link.exe + /DLL... It almost seems like it should work if not for whatever code is doing that heuristic/regex check?

I am building UE4 using the article from the wiki here. Sometimes when using remote workers the build fails with the following error:

cl: Command line error D8049 : cannot execute 'C:\tmp[...]\c1xx.dll': command line is too long to fit in debug record

I have observed the following:

• it happens only when cl.exe process is spawned on remote workers, when the file is compiled on the same machine where the build was initiated it's ok
• it happens when the arguments count sent to cl.exe exceeds around 20k characters
• when I execute CreateProcess in my own test aplication with the same compilation command it fails with the same error (which is very odd, because as I write above, it works fine from Fastbuild)
• using a response file doesn't help

I'm attaching the command I used to reproduce it in my test application. It's actually saved from Process.cpp (fullArgs parameter) in Fastbuild.

I'm on Win7, using VS2015 professional.

Command.txt

Note that this issue is similar to #87 but not the same.

Setup:

• on windows
• version 0.99 or 1.00
• custom emscripten target (compiles+links perfectly on mac/linux)
• emcc as compiler (emcc.bat)
• emcc as linker (emcc.bat)
• emcc as librarian (emcc.bat)
• CompilerFamily is set to clang
• LibrarianType is set to ar (also tried msvc)
• LinkerType doesn't have a clang option... (I tried msvc and the others)

The issue:

6> Lib: C:\dev\project\build\bin\bullet2\web-emcc-dev\bullet2.bc
c:\emsdk\upstream\emscripten\emcc.bat -o ........... +16239 characters of args
6> The command line is too long.
6> Failed to build Library. Error: 1 (0x01) Target: 'C:\dev\project\build\bin\bullet2\web-emcc-dev\bullet2.bc'
FBuild: Error: BUILD FAILED: lib-bullet2-web-emcc-dev
Time: 0.048s

This is the librarian step of a Library node. The issue here is (seemingly) coming from this error, which would make some sense since 1. it's a bat file 2. it's way more than 8192 characters 3. fastbuild is not (and can't) autodetect the need for a response file.

https://support.microsoft.com/en-us/help/830473/command-prompt-cmd-exe-command-line-string-limitation

Details: emscripten builds via emcc.bat, which simply calls emcc.py which does all the work. It forwards the args as is. emcc.py handles the use of response files, but there is no way for me to tell fastbuild to use them. It seems that the assumptions in fastbuild are quite limiting here. The assumption about the max length of a command line, as well as the assumption about which linkers can use response files.

It seems like the one solution proposed in #87 would be perfect for these situations (m_ForceUseResponseFile). e.g

LibrarianForceUseResponseFiles LinkerForceUseResponseFiles CompilerForceUseResponseFiles

I personally don't feel that hardcoding more assumptions (like emcc.bat and emcc) would be ideal, since this will come up time and time again for any compiler/linker/librarian that is not hardcoded.

Regardless this is a blocking issue, since I can't compile emscripten builds on windows with fastbuild, due to the missing use of a response file. I'd love to be able to be explicit and make that behaviour well defined, rather than the tool trying to guess :) p.s I expected forcing a LibrarianType to do something, but the behaviour was the same.

Thanks for the great tool as always, @ffulin

Export( .var ) is the counter part of Using( .var ). It will update the previous definition of .var in the first ancestor scope declaring it. If a previous declaration of .var is not found the parent scope will be used to declare a new .var.

This addition allows the following idiom :

.Defines = { '_DEBUG', 'SOMETHING=1' }
.CompilerOptions = ''

#if __WINDOWS__
ForEach( .name in .Defines )
{
   .CompilerOptions + ' /D "$name$"'
   Export( .CompilerOptions )
}
#endif

#if __LINUX__
ForEach( .name in .Defines )
{
   .CompilerOptions + ' -D$name$'
   Export( .CompilerOptions )
}
#endif

Print( "$CompilerOptions$" ) ; windows : ' /D "_DEBUG" /D "SOMETHING=1"'
                             ; linux   : ' /D_DEBUG -DSOMETHING=1'

I came across an error with this snippet :

.UntypedArray = {}
.Struct = []
.ArrayOfStructs = { .Struct }
.UntypedArray + .ArrayOfStructs

Which generates : fbuild.bff(4,10): FASTBuild Error #1034 - Operation not supported: 'ArrayOfStrings' + 'ArrayOfStructs'.

Empty arrays seem to be typed by default as arrays of string when declared with {}, which later causes a typing error when I try to concatenate it with an array of structs.

I'm trying to use fastbuild together with UE4, but I can't get it to work. When running the build process on pc 1, I can see in pc 2 in the UI of FBuildWorker "1 Connections" instead of 0.

However I'm getting the same error for all files:

Failed to build Object. Error: 4294967295 (0xffffffff)

I tried to add -distverbose flag, but I don't have where the output is - there isn't anymore info than without it.

I can't seem to find any info on what 4294967295 error code means either, although it seems like a generic error code for "unknown error".

How can I find out what the actual problem is?

Hi @ffulin,

I've hit an unexpected (to me) case with the cache. We've been using cache via SMB3 with a UNC Network Path for quite some time, with > 50 nodes. The nodes are able to store and retrieve cache entries successfully.

But, what I unfortunately haven't tested for quite some time was whether the cache entries are actually being shared between nodes (vs. each node storing and retrieving it's own exclusive copy).

I assumed the entries were being shared, and I know that at some point (far in the past) I did verify that it worked and entries were properly shared; but, I have direct evidence of multiple cases tonight where it is not occurring at the moment.

1. If I build from C:\dev and then build again from D:\dev, I just verified the cache entries to be exclusive.

2. If I build from two different machines, even if both machines use D:\dev, it seems that the entries are also exclusive.

This is with v0.99 with some modifications (but none having to do with the cache).

This is also with the LightCache enabled.

Is this behavior expected? Was there a point in the recent releases when the behavior of the cache with respect to this may have been changed?

Hi, ffulin! Thanks for your work!

Our project is compiling with different compilers for different platforms. We trying to make errors notification more readable in terms of our CI system. I guess, that option -monitor is really thats we need. Structured FastBuildLog.log allows us to do many useful things with compiler output. In particular, we want to parse compiler errors only (without warnings). During different experiments I noticed strange behavior. I compile with clang compiler foo.cpp file, that contains error and some warnings. Used options: -noprogress -wait -clean -summary -showcmds -j0 -monitor Also I used some -Werror compiler option: -Werror=unused-result (so IsWarningsAsErrorsClangGCC() should return true)

Compilation failed with error. The FastBuildLog.log looks like this:

133159458116500000 FINISH_JOB FAILED local "foo.cpp" "WARNING: foo.cpp(30,1): warning: 'FooOld' is deprecated foo.cpp(13,7): error: unknown type name 'i'"

On the one hand I see that compilation failed and on the other hand I see "WARNING" message. It is confusing a little. I expect that warning messages should not be shown and "WARNING" should be changed to "PROBLEM":

133159458116500000 FINISH_JOB FAILED local "foo.cpp" "PROBLEM: foo.cpp(13,7): error: unknown type name 'i'"

I guess that problem is in ObjectNode.cpp in ObjectNode::CompileHelper::SpawnCompiler method.

            if ( m_Err.IsEmpty() == false )
            {
                const bool treatAsWarnings = true; // change msg formatting
                DumpOutput( job, name, m_Err, treatAsWarnings );
            }

DumpOutput called here with treatAsWarnings=true without any conditions. In other places we call DumpOutput with treatAsWarnings=true only if IsWarningsAsErrorsClangGCC() return false. Possible fix:

                const bool treatAsWarnings = m_Result != 0 ? false : true; // change msg formatting 

Hi, ffulin! I checked gcc/clang compiler option that called: "Turn warnings into errors." It starts with a capital letter: -Werror So this place

            else if ( token.BeginsWith( "-werror" ) )
            {
                flags.Set( CompilerFlags::FLAG_WARNINGS_AS_ERRORS_CLANGGCC );
            }

should be (as I think) replaced by

            else if ( token.BeginsWith( "-Werror" ) )        // <= capital letter here!
            {
                flags.Set( CompilerFlags::FLAG_WARNINGS_AS_ERRORS_CLANGGCC );
            }

Hello, I'm trying to use FB with UE 5.1. It seems it works and I setup it correctly (probably), but when I want to build the project, it consume all my computer memory and it became unusable after couple of minutes and I need to hard reset the whole machine. It also happens on the other machines.

Is it possible to set some maximum memory what can be used by cl.exe? I've noticed there is a /Zm option for the cl.exe and I tried to set it into Compiler() { .Environment = '/Zm100' } or Settings.Environment = { "CL=/Zm100", }, but it doesn't seems it'd work. Maybe I use it incorrectly?

What also happens is the C: drive is being used 100%. Is that normal?

Thank you for the help! If there's anything more I could provide, let me know.

Hi! Please, check the file: Code/Tools/FBuild/FBuildCore/Graph/ObjectNode.h Seems, that there is mistake in method: bool IsWarningsAsErrorsMSVC() const { return ( ( m_Flags & FLAG_QT_RCC ) != 0 ); } I guess, this code should be like this: bool IsWarningsAsErrorsMSVC() const { return ( ( m_Flags & FLAG_WARNINGS_AS_ERRORS_MSVC ) != 0 ); }

I am using VS2022 to build the Unreal engine.The DLL Error has been fixed but the new Error occur:

7> - Blacklisted Worker: vm-donghaoran 7> - Node : D:\UnrealEngine\Engine\Intermediate\Build\Win64\UnrealEditor\Development\GeometryCore\Module.GeometryCore.3_of_3.cpp.obj 7>EXEC : - Job error Count: 1 / 3 7> - Details : 7>PROBLEM: D:\UnrealEngine\Engine\Intermediate\Build\Win64\UnrealEditor\Development\GeometryCore\Module.GeometryCore.3_of_3.cpp.obj 7>PROBLEM: C:\Users\donghaoran\AppData\Local\Temp.fbuild.tmp\0x00000000\core_1001\Module.GeometryCore.3_of_3.cpp.obj 7>Module.GeometryCore.3_of_3.cpp 7>D:\UnrealEngine\Engine\Intermediate\Build\Win64\UnrealEditor\Development\GeometryCore\Module.GeometryCore.3_of_3.cpp(3): fatal error C1083: Cannot open compiler generated file: '..\Intermediate\Build\Win64\UnrealEditor\Development\GeometryCore\Module.GeometryCore.3_of_3.cpp.dep.json': No such file or directory 7>EXEC : Failed to build Object. error : 2 (0x02) Target: 'C:\Users\donghaoran\AppData\Local\Temp.fbuild.tmp\0x00000000\core_1001\Module.GeometryCore.3_of_3.cpp.obj' 7>Got Result: vm-jiangjunji - D:\UnrealEngine\Engine\Intermediate\Build\Win64\UnrealEditor\Development\Zen\Module.Zen.cpp.obj 7>Remote System Failure!

I dont know how this error occur.Can someone knows what happend?

Looks like a bug. During jumbo build if continue a build after fixing code or build with key -clean there is an error message:

fatal error C1014: too many include files: depth = 1024

The cleaning intermediate and output files and full solution build is working fine.