Proposal: export HarnessEval-W case results as EvalPort ResultSets (portable evidence-tree records) #3

Hi — really enjoyed the paper and the design of the skill-routing pipeline. The fact that every case ships a skill_plan.json with per-skill reason fields for what ran and what was skipped, plus per-segment metadata.json and artifact_validation.json records, means HarnessEval-W's evaluation output is already unusually well-structured for something most benchmarks only keep as scalar CSVs. That's rare and it's worth being portable outside this repo.

I maintain EvalPort (Apache 2.0), an open interchange spec + SDK (evalport-sdk, Python/TS) for representing eval TestCase/Grader/ResultSet records so they can move between frameworks without hand-written glue. I think HarnessEval-W's per-case output maps onto it cleanly:

• A case like harnesseval_w_exploratory_0001_urban_crosswalk → an EvalPort TestCase (id, input = the case prompt, tags from taxonomy.primary_axis / taxonomy.probe_family).
• Each entry in skill_plan.json's selected_skills[] (e.g. render_quality_inspector, viewpoint_trajectory_verifier) → a Grader with type: "custom" and grader_id = skill_id. EvalPort's spec already has a "custom grader handling" rule (unsupported types clean-skip with metadata.skip_reason) that maps naturally onto your own skipped_skills[] + reason fields — the semantics are basically the same idea in both formats.
• A model run's per-segment metadata.json + artifact_validation.json (e.g. status: "passed", duration_seconds, expected{}, errors[]) → one ResultSet.results[] entry, with test_case_id = case_id, passed from artifact_validation.status, and a grader_results[] entry per skill carrying score/passed/reason.

Concrete sketch, using your real fields from segments/segment_001/{metadata.json,artifact_validation.json} and benchmark/plans/exploratory_transition/*.skill_plan.json:

# harnesseval_to_evalport.py — sketch, not a full converter
def segment_to_evalport_result(metadata: dict, artifact_validation: dict, skill_plan: dict) -> dict:
 return {
 "test_case_id": metadata["case_id"], # harnesseval_w_exploratory_0001_urban_crosswalk
 "actual_output": metadata["output_video"],
 "passed": artifact_validation["status"] == "passed",
 "grader_results": [
 {
 "grader_id": skill["skill_id"], # e.g. viewpoint_trajectory_verifier
 "type": "custom",
 "score": None, # HarnessEval's own case-card score, once emitted
 "passed": artifact_validation["status"] == "passed",
 "reason": skill["reason"],
 "metadata": {
 "role": skill["role"],
 "probe_family": skill_plan["taxonomy"]["probe_family"],
 },
 }
 for skill in skill_plan["selected_skills"]
 ],
 "metadata": {
 "model_slug": metadata["model_slug"],
 "duration_seconds": artifact_validation["duration_seconds"],
 "errors": artifact_validation["errors"],
 },
 }

That would let harnesseval verify run output (or the eventual per-case evidence tree / summary.json) be re-emitted as a spec-valid ResultSet and picked up by anything that already speaks EvalPort — for comparable prior art, see the huggingface-evaluate and lm-eval-harness adapters, which do the same kind of harness-result → ResultSet translation for other benchmark harnesses.

I know your README's "Extending HarnessEval" section is scoped to new cases/skills rather than tooling integrations, so I'm not assuming this is a priority — just flagging it since your evidence-tree format is a genuinely good fit and I'd be glad to build a standalone harnesseval-openeval-adapter package (following the shape of the adapters linked above: to_openeval(), tests against the real EvalPort validator, a README) and send it as a PR here or as a repo under adapters/ in the EvalPort repo, whichever you'd prefer. Totally fine to close this if it's not something you want to carry — no pressure either way.

— Sahi, independent contributor (not affiliated with this project)