SampleMoreStudio

License: MIT Python 3.11+ Node 20+ Status

Repo: github.com/Onur45500/SampleMoreStudio

At matched token cost, independent sampling tends to beat self-critique loops.

SampleMoreStudio is an open-source benchmark harness + live dashboard that tests that claim under equal completion-token budgets — first on single-turn reasoning tasks, then on short tool-using agent scaffolds. Local models (Ollama) are first-class so you can reproduce at ~0ドル.

This is an independent reproduction / extension inspired by the 2026 Sample More, Reflect Less finding. It is not a copy of that paper’s harness.

Delta Charts — strategy accuracy at matched budget

Delta Charts — equal-budget strategy comparison with hero Δ callout (preliminary data).

Prompt-engineering folklore and many agent scaffolds favor self-critique loops (Self-Refine, Reflexion). A competing idea is simpler: spend the same token budget on independent samples and aggregate (majority vote / best-of-N).

SampleMoreStudio makes that comparison auditable:

• Same model, same temperature, same total completion-token budget B
• Full transcripts (every sample, critique, tool call)
• Deterministic graders (no LLM-as-judge)
• Dashboard charts for the "reflect loses" delta, budget curves, and turn-by-turn inspection

Results are always scoped: on this suite, at these budgets, with these models.

• Equal-cost sweep harness — every (model ×ばつ strategy ×ばつ task ×ばつ budget ×ばつ seed) cell, resume-safe
• 8 strategies — Phase 1: CoT, Self-Refine, Best-of-N, Majority-Vote@K · Phase 2: ReAct, Plan-then-Act, Reflexion, Majority-of-N-agent
• ~120 tasks — math (offline GSM8K slice), code, logic, and fake file/JSON/calc tool tasks
• React dashboard — live monitor, leaderboard, delta charts, budget curves, run explorer, transcript viewer (+ side-by-side compare); see Dashboard features
• Static export — dump SQLite → JSON + frontend build for a backend-free publishable site

Open http://localhost:5173 after starting the API and frontend. The left sidebar has a Phase filter (Phase 1 / Phase 2 / Combined) that applies across all views, plus navigation to each page.

How to open a transcript

1. Sidebar → Run Explorer
2. Click the model link on any row → Transcript Viewer

How to compare two strategies on the same task

1. Run Explorer → click A on e.g. self_refine / math_003
2. Click B on majority_vote / same math_003
3. Click Compare A/B

You need finished runs in the DB first (python scripts/cli.py phase0 ... or a sweep). Live Monitor polls /api/progress every 2s (no SSE required).

Live Run Monitor — presets, progress, live leaderboard, cost gauges, log tail:

Live Run Monitor

Side-by-side compare — two runs on the same task (turn labels, tokens, truncation, correctness):

Side-by-side transcript compare

flowchart LR
 cfg[config YAML] --> runner[sweep_runner]
 runner --> strat[Strategy.run]
 strat --> budget[token_budget]
 strat --> model[Ollama or litellm]
 strat --> grade[deterministic graders]
 runner --> db[(SQLite transcripts)]
 db --> api[FastAPI]
 api --> ui[React dashboard]
 db --> export[static site]

The unifying abstraction is a Strategy:

run(task, model, token_budget) → transcript + final_answer + tokens_spent

Everything else (grading, storage, dashboard) is strategy-agnostic and shared across phases.

Primary budget = completion tokens. Prompt tokens and estimated $ cost are recorded alongside but do not consume B.

Invariant: never exceed B. Undershoot is normal (models stop early). Overshoot > 10% is flagged. Every generation records a truncated flag.

A fixed per-sample cap (e.g. always 128) silently truncates Best-of-N / Majority-Vote while CoT uses the full B. Defaults in config/budgets.yaml:

• sample_cap = max(128, B ×ばつ 0.25) → 256→128, 512→128, 1024→256, 2048→512
• selection_cap = max(64, B ×ばつ 0.125)

The leaderboard exposes n_or_k, per_sample_cap, and truncation_rate so fairness is auditable.

All strategies use the same temperature (default 0.7 in config/models.yaml).

git clone https://github.com/Onur45500/SampleMoreStudio.git
cd SampleMoreStudio

• Python 3.11+
• Node 20+
• Ollama with a local model matching config, e.g.:

ollama pull qwen2.5:7b-instruct-q4_K_M

Windows

.\setup.ps1
.\.venv\Scripts\Activate.ps1
$env:PYTHONPATH = (Get-Location).Path

Unix

chmod +x setup.sh && ./setup.sh
source .venv/bin/activate
export PYTHONPATH="$(pwd)"

python scripts/cli.py phase0 --tasks 20 --seeds 1 --budget 512

Runs Self-Refine vs Majority-Vote on math, dumps transcripts under results/transcripts/, and prints Δpp + budget integrity.

# Terminal 1
uvicorn backend.main:app --reload --port 8000
# Terminal 2
cd frontend
npm run dev

Open http://localhost:5173.

docker compose up --build

Ollama must be reachable from the backend container (host.docker.internal on Docker Desktop).

Get-NetTCPConnection -LocalPort 8000 | Select-Object OwningProcess
Stop-Process -Id <pid> -Force

Or use --port 8001.

Named presets live in config/sweeps.yaml:

python scripts/cli.py sweep --preset phase0
python scripts/cli.py sweep --preset phase1_publish # resume-safe; long-running
python scripts/cli.py sweep --preset phase2_publish

Overrides: --seeds, --budgets, --categories, --task-limit, --models, --strategies.
The dashboard Start preset control uses the same presets. Re-run to resume; completed cells are skipped.

python scripts/cli.py generate-tasks # rebuild task JSON from generators / offline GSM8K slice
python scripts/test_budget.py # budget + extraction unit tests
python scripts/summarize.py # quick leaderboard dump from SQLite

• Model — edit config/models.yaml (ollama/... local, or OpenRouter via .env OPENROUTER_API_KEY)
• Task — add JSON under tasks/phase1_reasoning/<cat>/ or tasks/phase2_agent/<cat>/ with id, prompt, answer, grader, grader_params
• Strategy — implement Strategy under backend/strategies/, register in config/strategies.yaml and backend/strategy_registry.py

python export/build_static_site.py --build

Writes a backend-free site to export/out/ (including transcripts).

Numbers below are from small local smokes on qwen2.5:7b-instruct-q4_K_M. Treat them as signal checks, not a published claim. Larger phase1_publish / phase2_publish sweeps (with bootstrap CIs) should replace these when complete — please open an issue or PR with updated tables rather than overstating early results.

Δ ≈ +16.7 pp (sampling over reflecting); CI wide at this n. Budget spent/target ≈ 1.0, 0 overshoot flags. At B=256 with thin slices, Self-Refine rarely emits a usable Answer:.

Majority-Vote beat Self-Refine on the claim under test. CoT won overall in part because fixed sample_token_cap=128 truncated multi-sample strategies — V2 scales caps with B to reduce that confound.

On short deterministic tool tasks, a single good plan can win; Majority-of-N needs enough tokens per attempt for a full Action/Observation loop.

Transcript dumps: results/transcripts/sweep_*/*.txt.

Limitations

• Compare strategies within a model; local vs API absolute skill differs
• Small n → wide CIs on hero Δpp
• Allocation formulas are necessarily somewhat arbitrary (documented above)
• Offline GSM8K slice may be contaminated for absolute accuracy; rankings are the target claim
• Code grading = subprocess + timeout — not a security boundary
• Run from repo root (PYTHONPATH); not published as a PyPI package

Non-goals (v1/v2)

• No LLM-judge grading
• No auth / multi-tenant SaaS
• No distributed multi-GPU orchestration
• No fine-tuning
• No claim of universal settlement of "sample vs reflect"

Contributions welcome — see CONTRIBUTING.md for setup, tests, and how to add models / tasks / strategies.

Good first directions: more offline tasks, preset docs, dashboard UX polish, reproducing a publish sweep and updating the preliminary findings table with CIs.

MIT