Skip to content

Navigation Menu

Sign in
Sign up

[P2] Add memento-vault benchmark runner for Rondo harness evaluation #29

Open
Labels
enhancementNew feature or request harness-engineeringRondo/Beislid automation harness work. priority: P2Later work, usually after foundational dependencies land. status: unavailableBlocked by another repo, feature, or prerequisite.

Description

Context

The handoff's proving ground is memento-vault. The evaluation plan calls for running 8-12 real issues and measuring whether externalized knowledge/process reduces human steering and review failures.

Blocked by

  • memento-vault benchmark task definitions.
  • Rondo run ledger/checkpoints.
  • Rondo adapter contract.
  • Rondo gate runner.
  • GitHub Issues tracker support for the pilot repo.

Proposal

Add a benchmark runner that consumes curated memento-vault tasks and records comparable run outcomes.

Scope

  • Load benchmark task definitions with issue snapshot, acceptance criteria, required gates, risky areas, and rubric.
  • Run one or more Rondo adapters against the task set.
  • Collect metrics: interventions, gate failures, repeated mistakes, PR-readiness, time, and harness failures.
  • Classify failures using the taxonomy from the evaluation plan.
  • Produce a batch report after 3-5 tasks.

Acceptance criteria

  • A benchmark batch can run against a configured task set without manual result collation.
  • Results distinguish agent failure, harness failure, gate missing, knowledge missing/not retrieved, and ambiguous requirements.
  • Batch reports identify repeated failures that should become guides, gates, or policy changes.
  • The runner is adapter-neutral where practical.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request harness-engineeringRondo/Beislid automation harness work. priority: P2Later work, usually after foundational dependencies land. status: unavailableBlocked by another repo, feature, or prerequisite.

    Projects

    No projects

    Milestone

    No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      AltStyle によって変換されたページ (->オリジナル) /