Visual Studio Test Adapter for Catch2

Overview

Test Adapter for Catch2

Within Visual Studio, the Test Explorer is a convenient way to run and debug unit tests. This test adapter adds support for the Catch2 C++ test framework. This adapter is for use in combination with Visual Studio 2017 and later.

How to get it

The Test Adapter for Catch2 is available via the Visual Studio Marketplace. It is available under the name "Catch Adapter for Catch2". The easiest way to get it is via "Extensions and Updates..." in the "Tools" menu. Of course you can also build it yourself.

A note on usage

Out of the box the extension does not discover tests. You need to add settings for the Test Adapter for Catch2 to a .runsettings file and use that as your test settings. This prevents the discovery mechanism from running non-Catch2 executables in your solution upon first use. See the Walkthrough for a detailed description on how to use the Test Adapter for Catch2.

Documentation

For documentation on the Test Adapter for Catch2 see the followng links.

Issues
  • Discoverer fails to find any tests

    Discoverer fails to find any tests

    Why can't it find the debug symbols when the .pdb files are in the same folder as the .exe file?

    [1/30/2019 1:57:34 PM Warning] Could not locate debug symbols for 'C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe'. To make use of '--list_content' discovery, ensure that debug symbols are available or make use of '<ForceListContent>' via a .runsettings file.

        <DebugBreak>on</DebugBreak>
        <DiscoverCommandLine>--discover *</DiscoverCommandLine><!-- Note this custom discoverer is not part of the Catch2 framework -->
        <DiscoverTimeout>10000</DiscoverTimeout>
        <FilenameFilter>.*</FilenameFilter>
        <IncludeHidden>true</IncludeHidden>
        <Logging>Verbose</Logging>
        <MessageFormat>StatsOnly</MessageFormat>
        <StackTraceFormat>ShortInfo</StackTraceFormat>
        <StackTracePointReplacement>,</StackTracePointReplacement>
        <TestCaseTimeout>10000</TestCaseTimeout><!-- Milliseconds -->
        <WorkingDirectory></WorkingDirectory>
        <WorkingDirectoryRoot>Executable</WorkingDirectoryRoot>
        <ForceListContent>true</ForceListContent>
      </Catch2Adapter>
    
    opened by ScottHutchinson 17
  • "No source available" in Text Explorer

    After quite a fight I can see tests listed in Test Explorer.

    The problem know is that I can't click on a test in test explorer to navigate to the source code. A very useful feature.

    image

    How can I get the source code to the tests listed in test explorer? Is this a bug or missing feature?

    opened by kervinpierre 8
  • Sample Project?

    Sample Project?

    I've been trying to get this adapter working, by tweaking the .runsettings file, but have yet to get it to even list the TEST_CASE in the Test Explorer. TE is showing MS-Test test cases in the same DLL.

    This is the test file:

    #include "stdafx.h"
    //#define CATCH_CONFIG_MAIN  // This tells Catch to provide a main() - only do this in one cpp file
    #define CATCH_CONFIG_RUNNER
    #include "catch.hpp"
    
    unsigned int Factorial(unsigned int number) {
    	return number <= 1 ? number : Factorial(number - 1)*number;
    }
    
    TEST_CASE("Factorials are computed", "[factorial]") {
    	REQUIRE(Factorial(1) == 1);
    	REQUIRE(Factorial(2) == 2);
    	REQUIRE(Factorial(3) == 6);
    	REQUIRE(Factorial(10) == 3628800);
    }
    

    So, by chance can you post a fully working / configured Project that successfully uses your Adapter?

    Thanks, djg

    opened by davegi 7
  • "Tests not found"

    So I got this message after I run the tests.

    [2019. 12. 06. 6:46:01.428 du.] ---------- Discovery started ---------- Test Adapter for Google Test: Test discovery starting... Test discovery completed, overall duration: 00:00:00.0418552 Started Catch2Adapter test discovery... Finished Catch2Adapter test discovery. No test is available in C:\Users\enkeyz\source\repos\chapter10\x64\Debug\Listing10-1.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. [2019. 12. 06. 6:46:02.132 du.] ========== Discovery finished: 0 tests found (0:00:00,6366892) ========== No tests found to run. No tests found to run.

    opened by enkeyz 6
  • Run Selected Tests doesn't quote names with spaces

    Run Selected Tests doesn't quote names with spaces

    In VS2019, when running test cases from the test explorer (both individually, and "Run All"), It appears the extension invokes the test executable with the test names unquoted. So if you have a TEST_CASE("foo works") for example, the extensions then runs tests foo works (which would run the test cases foo and works, but not foo works).

    The test names should be quoted so that tests "foo works" is invoked, for example.

    As a side note, the test cases are marked as passing, even though the output says "no tests ran". The arguments -w NoTests could be used to turn that into a warning at least -- maybe it should be an error, but a warning is still better than silently passing.

    opened by melak47 5
  • Not working for me for visual studio 2019

    Not working for me for visual studio 2019

    14/05/2019 15:08:20 Informational] ------ Discover test started ------ [14/05/2019 15:08:20 Error] Failed to load RunSettings file: 'C:\Work\LibDev\UnitTests\ReferenceTests.runsettings'. Please check if the file exists and is valid. [14/05/2019 15:08:20 Error] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Xml.XmlException: For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method. at System.Xml.XmlTextReaderImpl.Throw(Exception e)

    opened by snahmad 5
  • Catch2 XML reporter format change breaks Test Adapter

    Catch2 XML reporter format change breaks Test Adapter

    We switched from an old Catch2 developer version labeled in catch_amalgated.hpp as

    Generated: 2020-10-08 13:59:26.616931
    

    to git commit 4436a60, which is quite recent.

    This unfortunately broke this Test Adapter, because the XML output structure, which is processed by this adapter, changed: test group has been removed and element "Catch" renamed to "Catch2TestRun" in Catch2 commit e5938007f7dab5c9886ca561d09106ee4d7f2301 of September 7th this year.

    Both Catch2 commits generate the same version string when calling the executable with argument --libidentify:

    description:    A Catch2 test executable
    category:       testframework
    framework:      Catch Test
    version:        3.0.0-preview.3
    

    Therefore it is not possible for the Test Adapter yet to distinguish between new and old XML output and parse the results accordingly.

    We attached the different outputs for one of our test projects. It only contains a single test case. The output has been generated by:

    test_d *narrowcast* -s -r xml -o output.xml
    

    Best regards, Paul

    xml-output-old-and-new.zip

    opened by PaulFocus 4
  • Test case names

    Test case names

    It seems that having [ , ] and , in test case names does not work anymore. Tests are still detected, but could not be executed in Visual Studio 2019.

    Furthermore, :: causes some subgrouping, which is very unfortunate if you want to use for example, std::string somewhere in the middle of the test case name. It is sort of ok-ish if it belongs to the prefix. (: works fine)

    opened by matt77hias 4
  • Custom header results in no discovered test cases

    Custom header results in no discovered test cases

    With additional prints in my test main method, no test cases are discovered. Without additional prints in my test main method, all test cases are discovered.

    Ideally, I like to output my own header.

    int main(int argc, char** argv)
    {
    	std::cout << "Hello/n"; // No tests are discovered if not commented?
    	
    	// Catch setup.
    	Catch::Session session;
    
    	const int result = session.applyCommandLine(argc, argv);
    	if (result != 0)
    	{
    		return result;
    	}
    
    	return session.run();
    }
    

    Minimal VS2019 example: Widget.rar.

    opened by matt77hias 4
  • Can't intall testadapter catch2 because of prerequisites issue

    Can't intall testadapter catch2 because of prerequisites issue

    Hi All, I downloaded catch2 from https://marketplace.visualstudio.com/items?itemName=JohnnyHendriks.ext01 but when I try to install it for vs2017 Pro, I have an error like "The extension cannot be installed to this product due to prerequisites that cannot be resolved" . Do you know why please ? We have Visual studio professional 2017 15.4 and it seems that catch2 is for vs2017 and vs2019. Thanks in advance.

    opened by DamDre 4
  • Feature Request: GIVEN/WHEN/THEN support

    Feature Request: GIVEN/WHEN/THEN support

    I noticed in the walkthrough that nested sections are provided. I started using the GIVEN/WHEN/THEN capability in Catch2 and now find it hard to go back to sections!

    If possible, it'd be awesome to support this alternate coding style in the test explorer.

    enhancement 
    opened by iwubcode 4
  • Location of Runnable EXE Not As Assumed

    Location of Runnable EXE Not As Assumed

    I have been having trouble with discovery, but think I have diagnosed the problem. The "output directory" (in vs proj file) contains the exe, but not in runnable form. Our build system symlinks it into another directory with all its dependencies for running/debugging. This is handled with a custom Debug Command.

    So.. "MyTests.exe --list-tests" will crash instead of returning anything useful.

    Is there a way to control where the discoverer looks for the exe to discover the tests?

    opened by allan-bommer 3
  • Test discovery depends on filename case sensitivity

    Test discovery depends on filename case sensitivity

    It seems that test-case discoverability depends on the case of the filename as referenced in the Solution file. Other test runners (e.g. GTest) do not fail on this.

    If the case of the filename and the name of the project does not match, the discovery will fail and give an "Invalid source" error, e.g:

    CheckSource name: coretests Invalid source. Accumulated Testcase count: 0

    After some digging it turned out that the following line in my solution file was the culprit: Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "CoreTests", "coretests\coretests.vcxproj", "{BEC24047-EEE1-EE48-DAA5-85524471212A}"

    After changing it to match the case of the project name it worked: Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "CoreTests", "coretests\CoreTests.vcxproj", "{BEC24047-EEE1-EE48-DAA5-85524471212A}"

    Not entirely sure if this is an issue with the VS test runner or with the adapter, but even if it's in VS, perhaps this is something that could be worked around in the adapter.

    opened by jeroenakershoek 0
  • doctest support

    doctest support

    Hi! I'm the creator of doctest which is a testing framework very similar to Catch. Would you be interested in adding support for it as well? I could help in any way necessary. What functionality needs to be exposed?

    opened by onqtam 1
  • Question: How to set up TestResults folder

    Question: How to set up TestResults folder

    referring to the ResultsDirectory setting in runsettings, i don't see any result file after adding a TestResults folder to the solution directory and successfully running the reference tests e.g. Catch_Testset01 within visual studio 2017 resharper unit test explorer.

    .\TestResults

    Is it expected to have any test result output files? thank you for your time and help.

    opened by leonyu1010 1
  • Merge with https://github.com/xkbeyer/CatchTestAdapter

    Merge with https://github.com/xkbeyer/CatchTestAdapter

    There is another TestAdapter solution for Catch2: https://github.com/xkbeyer/CatchTestAdapter Maybe you could get together and combine your solutions :-)

    opened by fmuecke 1
  • Feature Request: Run tests in parallel

    Feature Request: Run tests in parallel

    I believe the test executables currently run one at a time. However, longer running tests can queue up the short-quick tests and make the overall test run take longer than it could if the tests were ran in parallel. It'd be nice if the test-explorer supported multiple tests to be ran at a single time. Maybe a MaxProcesses setting or something to control how many run at once?

    Thanks again!

    enhancement 
    opened by iwubcode 4
Releases(v1.8.0)
  • v1.8.0(Jan 29, 2022)

    This version introduces support for Catch2 v3, which has a small change in xml-reporting format and has improved support for test discovery.

    New Features

    • Add support for Catch2 v3 (issue #52)

    Extended Features

    • Enable support for environmental variables when running in debug mode.
    Source code(tar.gz)
    Source code(zip)
  • v1.7.0(Aug 2, 2021)

    New Features

    • Add support for Visual Studio 2022.
    • Add support for environmental variables (inspired by issue #37). Environmental variables to be set for the Catch2 executable process can now be configured with the <Environment> option (see Settings documentation for details).

    Bug fixes

    • Issue #46: A problem deleting temporary files, caused the test adapter to fail.
    Source code(tar.gz)
    Source code(zip)
  • v1.6.0(Mar 13, 2021)

    This version introduces some major changes in the way the backend works. Where before for each test case an instance of the test executable was started (still the default behavior), now it is possible to run multiple test cases in a single instance of the test executable, improving test execution time significantly. Related to this there is also a mechanism introduced to force a test case to be run in its own test executable instance, when using the new way of executing test cases.

    Furthermore, the test results are no longer written to memory, but to a temporary file before being processed. Though needed to make the above change possible, the positive side effect is that the test explorer now no longer has problems with tests that write information to the terminal in a way that is not intercepted by Catch2. In addition, the backend is better able to handle partial xml test result output, which typically occurs when the test executable instance needs to be killed due to a timeout, or when the test executable is prematurely terminated.

    Source code(tar.gz)
    Source code(zip)
  • v1.5.0(Feb 10, 2019)

    This version contains significant improvements in testcase discovery. No longer do you need a custom discovery mechanism to enable source links to the testcase. Also, the discovery mechanism has been made more robust.

    Extended Features

    • Issue #18: Added support for verbose default discovery. This enables creating a source file link to the position of a testcase for easy navigation via the Test Explorer, by using default discovery. This removes the need to add a custom Xml-based discovery mechanism to enable this testcase navigation feature in the Test Explorer.

    Changes to defaults

    • The default value for <DiscoverCommandLine> has been set to --verbosity high --list-tests *, this enables creating a source file link to the position of a testcase for easy navigation via the Test Explorer.

    Bug fixes

    • Issue #17: Testcases and tags with long names are not detected correctly using default discovery settings. Mostly fixed, there are a few corner cases that cannot be fixed.

    Documentation

    • Updated the walkthrough to reflect the availability of verbose default discovery.
    • Added a page on the discovery mechanism. This includes an updated and improved custom discovery example.
    Source code(tar.gz)
    Source code(zip)
  • v1.4.2(Dec 10, 2018)

    This version adds support for VS2019. It also updates the Microsoft.TestPlatform.ObjectModel to version 15.9.0.

    Bug fixes

    • Bug: When tests are run via "Debug Selected Tests" the working directory is set incorrectly. Fixed.
    Source code(tar.gz)
    Source code(zip)
  • v1.4.0(Oct 18, 2018)

    This is the v1.4.0 release. It is mostly a bug fix release.

    Extended Features

    • Improve Additional info output with respect to section. Is now more like standard Catch2 output.
    • Improve how is dealt with problematic test case names. Tests with such names are now marked as skipped, with an error message that suggests a possible solution.

    Bug fixes

    • Bug: Test durations in Test Explorer are wrong for systems setup with a decimal symbol that is not '.'. Fixed.
    Source code(tar.gz)
    Source code(zip)
  • v1.3.0(Aug 18, 2018)

    This is the v1.3.0 release. It is mostly a bug fix release.

    Extended Features

    • Add option <StackTracePointReplacement> to allow replacement of decimal points in StackTrace description with a custom string. This is related to a bug fix, where decimal points in the StackTrace description interferes with the displayed StackTrace link.

    • Handle test cases with names differing only in case more gracefully (e.g., "TestName", "TESTNAME", and "testname"). Test cases with names that only differ by case are always run together. Now the correct test result and failures are shown for those tests. Shown assertion statistics are of all tests that were run, and a note is added to the test result message to indicate this.

    Bug fixes

    • Bug: StackTrace link does not display correctly when decimal points are part of the description (e.g., when displaying floating point numbers). Fixed.
    • Bug: Test cases with names ending in '\' generate invalid output. Fixed.
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Jul 29, 2018)

    This is the v1.2.0 release. It contains the following changes with respect to v1.1.0.

    This version contains an important fix that enables stack trace links to source code. This is a significant usability improvement. Please update to this version and stop using the older ones. Documentation has been adapted to assume you are using this version.

    New Features

    • Added <MessageFormat> option to configure what information is shown in the message part of the test case detail view. The default is to only show the assertion statistics for the test. It is also possible to choose to show additional info in the message, or to not have a message at all.

    Extended Features

    • Improve error handling, specifically when a discovery timeout occurs it is now logged as a warning at logging level normal.
    • Enable stack trace links in the Test Explorer detail view. With the help of Microsoft I was able to figure out the correct string format to use for the Stacktrace info in order to turn it into a source link. As a result the <StackTraceFormat> option was also altered and now has the options Noneand ShortInfo, the latter being the default or fall-back value.

    Changes to defaults

    • The default value for <DiscoverCommandLine> has been set to --list-tests *, as having two settings with invalid defaults is not really usefull.
    • The default value for <DiscoverTimeout> has been set to 1000 ms. There were situations where 500 ms turned out to be too short, doubling that hopefully will give less problems.

    Bug fixes

    • Bug: CHECK_FALSE and REQUIRE_FALSE failed output expansion shows '!' in front of value results. E.g., "CHECK_FALSE( !true )", should be "CHECK_FALSE( true )". Fixed.
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Jul 14, 2018)

    This is the v1.1.0 release. It contains the following changes with respect to v1.0.0.

    New Features

    • Added disabled attribute to <Catch2Adapter> node for use in the .runsettings file.
    • Added <DebugBreak> setting that turns on or off Catch2's break on test failure feature (--break), when running tests via Debug Selected Tests.

    Extended Features

    • Added Debug level option to <Logging> setting and updated what is logged in the Verbose level.
    • Improve error handling, specifically when a duplicate testname is used, the potential resulting error is logged.
    • Add support for reporting Fatal Error Conditions

    Bug fixes

    • Bug: Warnings in Sections are not displayed when there are no failures. Fixed.
    • Bug: Get invalid test runner output error when Catch2 xml output produces additional text after xml report. Additional text is now ignored.
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jun 26, 2018)

    This is the first release of the Visual Studio Test Adapter for Catch2. It has all the basic features one might expect of a Visual Studio Test Adapter. It has proven to be useful for myself and a small group of beta-testers. As such I found it worthy for release.

    Capabilities

    • Discovery of Catch2 tests
    • Filename filter for test discovery
    • Enables use of custom discovery algorithms
    • Runs test with appropriate result messages
    • Can start a debug session for a test
    • Can set a working directory for running the tests
    • Can set a timeout for running a test case
    • Can set a timeout for test discovery
    • Can choose to include hidden tests or not

    Configuration

    The test adapter is configured via a .runsettings file.

    Known issues

    • Stack trace link does not work.
    • Not all Catch2 xml information can be displayed
    Source code(tar.gz)
    Source code(zip)
A D++ Discord Bot template for Visual Studio 2019 (x64 and x86)

D++ Windows Bot Template A D++ Discord Bot template for Visual Studio 2019 (x64 and x86, release and debug). The result of this tutorial. This templat

brainbox.cc 19 Jul 16, 2022
A Visual Studio template used to create Cobalt Strike BOFs

Introduction Cobalt Strike beacon object files (BOFs) is a feature that added to the beacon in order to allow rapid beacon extendibility in a more OPS

Securify 146 Jul 30, 2022
Sharpmake is an open-source C#-based solution for generating project definition files, such as Visual Studio projects and solutions, GNU makefiles, Xcode projects, etc.

Sharpmake Introduction Sharpmake is a generator for Visual Studio projects and solutions. It is similar to CMake and Premake, but it is designed for s

Ubisoft 750 Aug 1, 2022
Visual Studio Extension that installs additional color themes

Using this Extension Download and install the extension Restart Visual Studio Navigate to Tools > Options > Environment > General and select your colo

Microsoft 327 Jul 16, 2022
A Visual Studio extension containing a collection of tools to help contributing code to the Chromium project.

VsChromium VsChromium is a Visual Studio Extension containing a collection of tools useful for editing, navigating and debugging code. VsChromium was

The Chromium Project 253 Jul 24, 2022
Visual Studio native debugger extension to help debug native applications using Mono.

Unity Mixed Callstack UnityMixedCallstack is a Visual Studio 2017/2019 extension to help debug native applications embedding Mono, like Unity. If you

Unity Technologies 73 Aug 5, 2022
Useful UE4 Visual Studio extensions.

UE4 Smarter Macro Indenting This extension was designed to fix the unnecessary and annoying "smart" indenting that Visual Studio likes to do around va

Chris Pawlukowsky 241 Jun 14, 2022
Visual Studio extension for assembly syntax highlighting and code completion in assembly files and the disassembly window

Asm-Dude Assembly syntax highlighting and code assistance for assembly source files and the disassembly window for Visual Studio 2015, 2017 and 2019.

Henk-Jan Lebbink 4k Jul 28, 2022
A Visual Studio extension that provides enhanced support for editing High Level Shading Language (HLSL) files

HLSL Tools for Visual Studio This extension is for Visual Studio 2017 / 2019. Go here for the Visual Studio Code extension. HLSL Tools is a Visual Stu

Tim Jones 404 Aug 3, 2022
Half-Life Singleplayer SDK 2.3, updated to compile with Visual Studio 2019. Provided as-is with no further support. See the README for more information.

Half Life 1 SDK LICENSE Half Life 1 SDK Copyright© Valve Corp. THIS DOCUMENT DESCRIBES A CONTRACT BETWEEN YOU AND VALVE CORPORATION (“Valve”). PLEASE

Sam Vanheer 5 Apr 9, 2022
RRxIO - Robust Radar Visual/Thermal Inertial Odometry: Robust and accurate state estimation even in challenging visual conditions.

RRxIO - Robust Radar Visual/Thermal Inertial Odometry RRxIO offers robust and accurate state estimation even in challenging visual conditions. RRxIO c

Christopher Doer 56 Jul 26, 2022
Visual Leak Detector for Visual C++ 2008-2015

Visual Leak Detector Introduction Visual C++ provides built-in memory leak detection, but its capabilities are minimal at best. This memory leak detec

Arkady Shapkin 867 Jul 27, 2022
Firmware for the Boson Ethernet Adapter

Boson Ethernet Firmware This project contains firmware and gateware required to adapt the FLIR boson into an ethernet connection Directory Structure f

Gregory Davill 5 Jul 23, 2022
ControllaBLE - A retro-controllers to Bluetooth BLE adapter

ControllaBLE - A retro-controllers to Bluetooth BLE adapter This is an ESP32 based controller adapter that outputs as a dual joypad through Bluetooth

null 13 Dec 1, 2021
Nissan Pathfinder R51 Head Unit Adapter

Nissan Pathfinder R51 Head Unit Adapter This repo contains documentation and code to turn an Arduino into a CAN Bus climate control adapter for the Ni

Ryan Bourgeois 1 Apr 19, 2022
ESP based wifi adapter for Celestron telescopes

CelestronESPWifi ESP8266 based wifi adapter for Celestron telescopes The official Celestron Skyportal WiFi module is €190~. (aka SkyQ Link, aka Skylin

null 7 Jun 25, 2022
LoRa Driver for Semtech SX1262 on Linux (PineDio USB Adapter) and BL602 (PineDio Stack BL604)

LoRa Driver for Semtech SX1262 on Linux (PineDio USB Adapter) and BL602 (PineDio Stack BL604) Read the articles... "Build a Linux Driver for PineDio L

Lee Lup Yuen 7 Mar 15, 2022
MQB steering wheel adapter to retrofit into PQ46 platform (VW Passat b6/b7, CC)

MQB steering wheel adapter (retrofit MQB steering wheel into PQ46 platform - VW Passat b6/b7, CC) The adapter supports on-wheel buttons (media and cru

null 13 Aug 1, 2022
LLVM IR and optimizer for shaders, including front-end adapters for GLSL and SPIR-V and back-end adapter for GLSL

Licensing LunarGLASS is available via a three clause BSD-style open source license. Goals The primary goals of the LunarGLASS project are: Reduce the

LunarG, Inc. 151 Jun 18, 2022