I've done similar in the past where I've used a JSON Schema to provide a full schema for other files to be used during CI. This would give us a few benefits:

• As documentation for the format of each file, can also contain description of the format.
• Can be used to test that the test files are correct during any changes in CI. This will prevent any unintentional breakage cause by PRs.

As an example (schemas/userhost-split.yaml):

type: object
properties:
 tests:
 type: array
 items:
 '$ref': '#!/definitions/test'
definitions:
 test:
 type: object
 properties:
 source:
 type: string
 description: source is the usthost
 atoms:
 type: object
 properties:
 nick:
 type: string
 user:
 type: string
 host:
 type: string
 required: [source, atoms]