Organization of tests

Great advantage of Catch2 is that is allows for arbitrary titles of sections.

I think I figured out why you could not get the reporter to behave nicely. It turns out that Catch2 uses quite peculiar way of organization of tests: it does not follow the CW's-favorite describe/it/assert way of organization. Instead, it seems to resemble Julia's FactCheck, where Catch2's TEST_CASE is FactCheck's @facts, SECTION seems to correspond to @context, and REQUIRE is a @fact. As a result, it can be difficult to make a reporter which handles the organization artifacts with <DESCRIBE::> and <IT::>, because we will encounter similar problems as with Julia's setup, where we do not know what is an outer level, and what is the innermost level. Catch2's BDD mode does not seem to be helpful here either, because the GIVEN/WHEN/THEN just map to SECTION and we still do not know when we are in the innermost one. It could work if we forced some CW-specific convention, but I think it's not something what we want. Or, if the reporter can keep/pass state between events, it can try to deduce if it needs to emit an <IT::> at the point it attempts to emit <PASSED::> or <FAILED::>. I can give it a shot, but this this also feels a bit off (related: codewars/runner#262).

Assertion messages

It seems that Catch2 does not support assertion messages inside of its REQUIRE/CHECK macros. It is possible to provide custom messages with matchers, but depending on organization of tests, this can be cumbersome, especially in context of CW and the practice of random tests, if authors wanted to have an explicit feedback for a failing random case.

There is a feature request in Catch2's repository to provide such messages: catchorg/Catch2#2653 but it was already shot down once (catchorg/Catch2#929).

Reporter

The reporter is notoriously difficult to get to work for a couple of reasons:

• It is difficult to decide where to output an <IT::> marker (see Organization of tests above).
• Whenever a section starts, the framework re-reports all the events related to all sections above the tested one (i.e. when a section is started, all events related to ancestor sections and test cases are re-raised). This can be probably worked around by tracking the history of sections in the reporter (some example reporters already do this), but has to be evaluated further.
• Passed assertions are not reported by default. This might be not necessary, but can be helpful if assertions would be used to deduce whether an <IT::> should be emitted or not. To report passing assertions, the reporter needs to have m_preferences.shouldReportAllAssertions = true; in its constructor.
• The flag m_preferences.shouldRedirectStdOut = true;, should it be used, conflicts with the functionality of writing to stdout by the reporter. For now, the flag does not seem to be necessary and things written to stdout from inside of solution and from inside of tests appear in the report when the flag is set to false, but docs in some place suggested that the flag can be useful sometimes.

Still need to evaluate:

• reporting of generated test cases: nesting, failures and success,
• testing for (or for lack of) errors, exceptions, and failures,
• report in case of a fatal failure.

Build

Catch2, as opposed to header-only libraries like for example Snowhouse, is a library which needs to be linked with tests:

• it can be either pre-built and embedded in the runner image to be referenced by tests, or
• it can be rebuilt and linked on every run of tests, or
• its amalgamated version can be embedded into the source tree of a CW kata, and build together with the kata.

Locally i work with the amalgamated setup embedded in my local project, but I do not know which of three above options will be the best for CW runner. I expect prebuilding the library would save some time for user solutions.