Why can't it find the debug symbols when the .pdb files are in the same folder as the .exe file?

[1/30/2019 1:57:34 PM Warning] Could not locate debug symbols for 'C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe'. To make use of '--list_content' discovery, ensure that debug symbols are available or make use of '<ForceListContent>' via a .runsettings file.

    <DebugBreak>on</DebugBreak>
    <DiscoverCommandLine>--discover *</DiscoverCommandLine><!-- Note this custom discoverer is not part of the Catch2 framework -->
    <DiscoverTimeout>10000</DiscoverTimeout>
    <FilenameFilter>.*</FilenameFilter>
    <IncludeHidden>true</IncludeHidden>
    <Logging>Verbose</Logging>
    <MessageFormat>StatsOnly</MessageFormat>
    <StackTraceFormat>ShortInfo</StackTraceFormat>
    <StackTracePointReplacement>,</StackTracePointReplacement>
    <TestCaseTimeout>10000</TestCaseTimeout><!-- Milliseconds -->
    <WorkingDirectory></WorkingDirectory>
    <WorkingDirectoryRoot>Executable</WorkingDirectoryRoot>
    <ForceListContent>true</ForceListContent>
  </Catch2Adapter>

After quite a fight I can see tests listed in Test Explorer.

The problem know is that I can't click on a test in test explorer to navigate to the source code. A very useful feature.

How can I get the source code to the tests listed in test explorer? Is this a bug or missing feature?

I've been trying to get this adapter working, by tweaking the .runsettings file, but have yet to get it to even list the TEST_CASE in the Test Explorer. TE is showing MS-Test test cases in the same DLL.

This is the test file:

#include "stdafx.h"
//#define CATCH_CONFIG_MAIN  // This tells Catch to provide a main() - only do this in one cpp file
#define CATCH_CONFIG_RUNNER
#include "catch.hpp"

unsigned int Factorial(unsigned int number) {
	return number <= 1 ? number : Factorial(number - 1)*number;
}

TEST_CASE("Factorials are computed", "[factorial]") {
	REQUIRE(Factorial(1) == 1);
	REQUIRE(Factorial(2) == 2);
	REQUIRE(Factorial(3) == 6);
	REQUIRE(Factorial(10) == 3628800);
}

So, by chance can you post a fully working / configured Project that successfully uses your Adapter?

Thanks, djg

So I got this message after I run the tests.

[2019. 12. 06. 6:46:01.428 du.] ---------- Discovery started ---------- Test Adapter for Google Test: Test discovery starting... Test discovery completed, overall duration: 00:00:00.0418552 Started Catch2Adapter test discovery... Finished Catch2Adapter test discovery. No test is available in C:\Users\enkeyz\source\repos\chapter10\x64\Debug\Listing10-1.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. [2019. 12. 06. 6:46:02.132 du.] ========== Discovery finished: 0 tests found (0:00:00,6366892) ========== No tests found to run. No tests found to run.

In VS2019, when running test cases from the test explorer (both individually, and "Run All"), It appears the extension invokes the test executable with the test names unquoted. So if you have a TEST_CASE("foo works") for example, the extensions then runs tests foo works (which would run the test cases foo and works, but not foo works).

The test names should be quoted so that tests "foo works" is invoked, for example.

As a side note, the test cases are marked as passing, even though the output says "no tests ran". The arguments -w NoTests could be used to turn that into a warning at least -- maybe it should be an error, but a warning is still better than silently passing.

14/05/2019 15:08:20 Informational] ------ Discover test started ------ [14/05/2019 15:08:20 Error] Failed to load RunSettings file: 'C:\Work\LibDev\UnitTests\ReferenceTests.runsettings'. Please check if the file exists and is valid. [14/05/2019 15:08:20 Error] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Xml.XmlException: For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method. at System.Xml.XmlTextReaderImpl.Throw(Exception e)

We switched from an old Catch2 developer version labeled in catch_amalgated.hpp as

Generated: 2020-10-08 13:59:26.616931

to git commit 4436a60, which is quite recent.

This unfortunately broke this Test Adapter, because the XML output structure, which is processed by this adapter, changed: test group has been removed and element "Catch" renamed to "Catch2TestRun" in Catch2 commit e5938007f7dab5c9886ca561d09106ee4d7f2301 of September 7th this year.

Both Catch2 commits generate the same version string when calling the executable with argument --libidentify:

description:    A Catch2 test executable
category:       testframework
framework:      Catch Test
version:        3.0.0-preview.3

Therefore it is not possible for the Test Adapter yet to distinguish between new and old XML output and parse the results accordingly.

We attached the different outputs for one of our test projects. It only contains a single test case. The output has been generated by:

test_d *narrowcast* -s -r xml -o output.xml

Best regards, Paul

xml-output-old-and-new.zip

It seems that having [ , ] and , in test case names does not work anymore. Tests are still detected, but could not be executed in Visual Studio 2019.

Furthermore, :: causes some subgrouping, which is very unfortunate if you want to use for example, std::string somewhere in the middle of the test case name. It is sort of ok-ish if it belongs to the prefix. (: works fine)

With additional prints in my test main method, no test cases are discovered. Without additional prints in my test main method, all test cases are discovered.

Ideally, I like to output my own header.

int main(int argc, char** argv)
{
	std::cout << "Hello/n"; // No tests are discovered if not commented?
	
	// Catch setup.
	Catch::Session session;

	const int result = session.applyCommandLine(argc, argv);
	if (result != 0)
	{
		return result;
	}

	return session.run();
}

Minimal VS2019 example: Widget.rar.

Hi All, I downloaded catch2 from https://marketplace.visualstudio.com/items?itemName=JohnnyHendriks.ext01 but when I try to install it for vs2017 Pro, I have an error like "The extension cannot be installed to this product due to prerequisites that cannot be resolved" . Do you know why please ? We have Visual studio professional 2017 15.4 and it seems that catch2 is for vs2017 and vs2019. Thanks in advance.

I noticed in the walkthrough that nested sections are provided. I started using the GIVEN/WHEN/THEN capability in Catch2 and now find it hard to go back to sections!

If possible, it'd be awesome to support this alternate coding style in the test explorer.

Some background, I started to use the test adapter for catch2 for my own project and it worked wonderfully. One day I was fed up with the abundance of random std/catch code that was shown to me in the test coverage results so I sought to remove them.

After some time I managed to get them deleted using this .runsettings file (Only showing the important part):

<DataCollecttionRunSettings>
    <DataCollector friendlyName="Code Coverage">
        <Configuration>
            <CodeCoverage>
                <Functions>
                    <Exclude>
                        <Function>.*Catch.*</Function>
                        <Function>.*std.*</Function>
                    </Exclude>
                </Functions>
            </CodeCoverage>
        </Configuration>
    </DataCollector>
</DataCollecttionRunSettings>

The problem is that if I use this .runsettings file the results actually get filtered as expected, the problem is that the tests fail with Invalid test runner output. -------------------------..., but if I remove friendlyName=Code Coverage the tests resume to run correctly but the filtering fails.

I was wondering if it is a problem with the adapter or not, I wasn't able to find any explanation to this in the web nor in the documentation of the project sadly :(

I have been having trouble with discovery, but think I have diagnosed the problem. The "output directory" (in vs proj file) contains the exe, but not in runnable form. Our build system symlinks it into another directory with all its dependencies for running/debugging. This is handled with a custom Debug Command.

So.. "MyTests.exe --list-tests" will crash instead of returning anything useful.

Is there a way to control where the discoverer looks for the exe to discover the tests?

It seems that test-case discoverability depends on the case of the filename as referenced in the Solution file. Other test runners (e.g. GTest) do not fail on this.

If the case of the filename and the name of the project does not match, the discovery will fail and give an "Invalid source" error, e.g:

CheckSource name: coretests Invalid source. Accumulated Testcase count: 0

After some digging it turned out that the following line in my solution file was the culprit: Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "CoreTests", "coretests\coretests.vcxproj", "{BEC24047-EEE1-EE48-DAA5-85524471212A}"

After changing it to match the case of the project name it worked: Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "CoreTests", "coretests\CoreTests.vcxproj", "{BEC24047-EEE1-EE48-DAA5-85524471212A}"

Not entirely sure if this is an issue with the VS test runner or with the adapter, but even if it's in VS, perhaps this is something that could be worked around in the adapter.

Hi! I'm the creator of doctest which is a testing framework very similar to Catch. Would you be interested in adding support for it as well? I could help in any way necessary. What functionality needs to be exposed?

referring to the ResultsDirectory setting in runsettings, i don't see any result file after adding a TestResults folder to the solution directory and successfully running the reference tests e.g. Catch_Testset01 within visual studio 2017 resharper unit test explorer.

.\TestResults

Is it expected to have any test result output files? thank you for your time and help.

There is another TestAdapter solution for Catch2: https://github.com/xkbeyer/CatchTestAdapter Maybe you could get together and combine your solutions :-)