forked from openai/symphony
-
Notifications
You must be signed in to change notification settings - Fork 4
[P2] Add memento-vault benchmark runner for Rondo harness evaluation #29
Open
Description
Context
The handoff's proving ground is memento-vault. The evaluation plan calls for running 8-12 real issues and measuring whether externalized knowledge/process reduces human steering and review failures.
Blocked by
- memento-vault benchmark task definitions.
- Rondo run ledger/checkpoints.
- Rondo adapter contract.
- Rondo gate runner.
- GitHub Issues tracker support for the pilot repo.
Proposal
Add a benchmark runner that consumes curated memento-vault tasks and records comparable run outcomes.
Scope
- Load benchmark task definitions with issue snapshot, acceptance criteria, required gates, risky areas, and rubric.
- Run one or more Rondo adapters against the task set.
- Collect metrics: interventions, gate failures, repeated mistakes, PR-readiness, time, and harness failures.
- Classify failures using the taxonomy from the evaluation plan.
- Produce a batch report after 3-5 tasks.
Acceptance criteria
- A benchmark batch can run against a configured task set without manual result collation.
- Results distinguish agent failure, harness failure, gate missing, knowledge missing/not retrieved, and ambiguous requirements.
- Batch reports identify repeated failures that should become guides, gates, or policy changes.
- The runner is adapter-neutral where practical.