The draft2019-09 and draft2020-12 directories contain various tests for 'format'. The ones located in format.json document the results of everything (even bad formats) as valid: true. This is because of the movement of format to annotations by default. That's fine. So, these tests should all return valid for these schema versions.

However, the tests in the optional/format directory are setup with 'valid: false'. As if the results of the tests should be an assertion. But there isn't any data that determines this distinction other than their location. When evaluating the files in optional/format, the test suite should consider format failures as assertions. When evaluating the tests outside of optional/format, it should consider format failures as annotations.

Makes it a bit hard to auto generate test implementations given the test suite JSON files.

It would be nice if there was some distinction in the test suite. Perhaps actually including $vocabulary: { "...format": true } in the schema node within the tests. Since this should be the thing that governs the behavior.