I would like to suggest adding the following code just after line 752 in tap.c

if (retval == 0) {
printf("All tests passed!\n");
}

The effect of this would be to let the user know that no tests failed. This is helpful for console programs that produce a lot of output, so that the user doesn't have to scroll back through all of it to see what happened. In other cases, exit_status() prints informative information, so it would be good to have information printed about tests passing as well.

Motivation

A lot of my test files take the form of:

void test_literals(void) {
    cmp_ok(3, "==", 3, "should find that 3 equals 3");
}

void test_arithmetic(void) {
    cmp_ok(1+1, "==", 2, "should find that 1+1 equals 2");
    cmp_ok(2*2, "==", 4, "should find that 2*2 equals 4");
}

int main(int argc, char **argv) {
    plan(NO_PLAN);
    test_literals();
    test_arithmetic();
    done_testing();
}

However, the problem with this is that it doesn't know the number of tests in advance. The easy workaround is to run it once, determine the number of tests, and then modify the plan(NO_PLAN) like to use that number, but this is a little awkward, especially when the compiler could do the counting for us.

Implementation

AUTO_PLAN is an implementation of this idea of making the compiler count for us. It uses the special preprocessor variable __COUNTER__ to count all of the lines that could produce an output line. The way to use this feature is:

void test_literals(void) {
    cmp_ok(3, "==", 3, "should find that 3 equals 3");
}

void test_arithmetic(void) {
    cmp_ok(1+1, "==", 2, "should find that 1+1 equals 2");
    cmp_ok(2*2, "==", 4, "should find that 2*2 equals 4");
}

int main(int argc, char **argv) {
    plan(AUTO_PLAN);
    test_literals();
    test_arithmetic();
    done_testing();
}

The compiler will automatically determine that it should plan for 3 test cases (the 3 cmp_ok lines) and rewrite the plan(AUTO_PLAN) as plan(3). I decided to hide this feature behind an #ifdef USE_AUTO_PLAN because there is a possibility that someone's code already uses the __COUNTER__ variable.

One of the major limitations of this feature is that you can't split your tests up into multiple files and expect it to work, because the __COUNTER__ variable gets reset for each file. We could work around this problem by using Boost, but that's too large of a dependency for a simple library like this one.

Another limitation is that it will get the wrong count if you use pass() and fail() separately. This is because they each use ok() internally, but each ok() increases the number of tests, so if you had a construct like:

void test_pass_fail(void) {
    if (some_complicated_function()) {
        pass("Hooray");
    } else {
        fail("Boo");
    }
}

then this will be incorrectly counted as 2 tests instead of 1 (the pass() gets transformed into ok(1, "", "Hooray") and similarly for fail()). A workaround would be:

void test_pass_fail(void) {
    int rv;

    rv = some_complicated_function();
    cmp_ok(rv, "==", 0, "should do something correctly");
}

which will be correctly counted as 1 test.

Let me know if you have any questions or concerns with this feature.

I submitted libtap to the AUR, but I didn't know what the version of this library was so I made one up. It would help a lot if you would tag releases so they show up under "Releases" on github: git tag 0.1.0 or git tag v0.1.0. If you do that, I can make another PKGBUILD that targets released versions instead of development versions.

If my pull request (#17) is acceptable, I'll switch my PKGBUILD to point to this repository instead of my fork.

I've implemented the improvements you proposed in your comments for #10

• NULL for got and expected is handled
• reworked logic so memcmp() is not needed
• cosmetic fix in diagnostics

Hi Zorgnax,

I've added a cmp_mem test to easily compare arrays, memory regions, ... When I have time, I might add a version that takes an array literal instead of a variable as the "expected" parameter.

Thanks for libtap!

Hi,

even though I'm not a fan of the target platform in particular, I'd like to ask you to integrate this set of changes which makes libtap integrate well with VS C++ projects (tested with 2010) and provide a way of running prove against the actual generated tests by default.

Thank you for maintaining libtap! Alex

On NetBSD MAP_ANONYMOUS is already defined and libtap fails to build with:

$ uname -v
NetBSD 7.1.2 (GENERIC.201803151611Z)

$ gmake
cc -Wall -I. -fPIC   -c tap.c  -o tap.o
tap.c:302:0: warning: "MAP_ANONYMOUS" redefined [enabled by default]
 #define MAP_ANONYMOUS MAP_ANON
 ^
In file included from tap.c:297:0:
/usr/include/sys/mman.h:97:0: note: this is the location of the previous definition
 #define MAP_ANONYMOUS 0x1000 /* allocated from memory, swap space */
 ^
tap.c: In function 'tap_test_died':
tap.c:302:23: error: 'MAP_ANONYMOUS' undeclared (first use in this function)
 #define MAP_ANONYMOUS MAP_ANON
                       ^
tap.c:313:39: note: in expansion of macro 'MAP_ANONYMOUS'
                          MAP_SHARED | MAP_ANONYMOUS, -1, 0);
                                       ^
tap.c:302:23: note: each undeclared identifier is reported only once for each function it appears in
 #define MAP_ANONYMOUS MAP_ANON
                       ^
tap.c:313:39: note: in expansion of macro 'MAP_ANONYMOUS'
                          MAP_SHARED | MAP_ANONYMOUS, -1, 0);
                                       ^
Makefile:16: recipe for target 'tap.o' failed
gmake: *** [tap.o] Error 1

I started out with this approach:

#ifndef MAP_ANONYMOUS && defined MAP_ANON
#define MAP_ANONYMOUS MAP_ANON
#endif

but instead decided to heavily borrow an approach from libressl-portable.

I have been using libtap in my Introduction to Programming with C class at UCF this semester. Overall we are very happy with it. Thanks!

However, students complained that for one of my test suites all the tests passed, even though they had no code in a predicate function (i.e., one that should return a _Bool). Of course, that is an incorrect implementation, but often students are omitting the return statement in a function, and so not detecting that is a serious problem when not detected. I was eventually able to find that the tests improperly passed in this manner only if the testing code called plan(). If the call to plan() is omitted, then the tests work properly, checking the return value. I have boiled this down to the attached files, which I thiink clearly show the problem. This happens under Windows 10 using gcc (GCC) 5.4.0 running under cygwin and also under MingW gcc.

The attached zip file has 3 files. bfun.c has a Boolean-valued function named bfun in it that doesn't return a value. bfun.h is a header file for that. test_bfun.c has a main() function, which shows the problem. If you comment out the call to plan() in test_bfun.c, you get the expected detection of incorrect behavior in bfun().

bfun-example.zip

• create directories as necessary
• use -fPIC so shared objects work
• updated .gitignore

I'm building a PKGBUILD for ArchLinux and it works with these changes.

I ran into the issue of GCC optimization causing a test failure in t/diesok.c at the division-by-zero test, even after rebasing our internal package's source against 56e3123 (internal case CPANEL-41315). At first, I resorted to inline assembly, and I also considered using a pragma or function attributes to limit the disabling of optimization to just the code or file of that test, but this seemed like the most generic well-supported solution.

unlike is replaced with a skip macro on Windows. The macro definition is missing needed parenthesis and that results in Windows builds failing. This PR corrects the macro.