-
Notifications
You must be signed in to change notification settings - Fork 0
Add test verification workflow and safer generation - #3
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR extends the AutoCode MCP toolchain with a new test-verification step and broader support for custom-named solutions/generators, while hardening test generation to reduce the risk of unintended file deletion. It also adds stress-test statistics, updates workflow documentation, and bumps the package/plugin version to 0.7.0.
Changes:
- Add
problem_verify_teststool and register it in the MCP server/tooling docs/tests. - Add custom solution naming support across build/run, stress testing, test generation/verification; add generator
extra_argsand stress-test summary statistics. - Harden
problem_generate_testsoutput directory handling and only clear generated.in/.ansfiles; bump version to0.7.0.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/autocode_mcp/tools/test_verify.py |
New verification tool (file pairing, answer consistency, validator, empty-file checks). |
src/autocode_mcp/server.py |
Registers the new problem_verify_tests tool (tool count increases). |
src/autocode_mcp/tools/problem.py |
Adds output_dir + sol_name, validates output dir, and clears only generated test artifacts. |
src/autocode_mcp/tools/stress_test.py |
Adds sol_name/brute_name, generator types cycling, extra_args, and runtime statistics summary. |
src/autocode_mcp/tools/solution.py |
Adds name support to build/run and uses shared resolve_source. |
src/autocode_mcp/tools/{validator,generator,checker,interactor}.py |
Refactors duplicated source resolution into resolve_source, adds canonical path + binary size reporting. |
src/autocode_mcp/tools/mixins.py |
Introduces ResolvedSource + resolve_source() helper. |
scripts/workflow_guard.py |
Updates inferred state for custom-named solutions (prefix match) and adds workflow text mentioning verification. |
tests/test_tools/test_problem.py |
Adds regression tests for safer output dir handling and custom sol_name. |
tests/test_e2e_mcp.py |
Updates expected tool count for MCP list-tools. |
pyproject.toml, src/autocode_mcp/__init__.py, uv.lock, .claude-plugin/plugin.json, packaging tests |
Version bump to 0.7.0. |
CHANGELOG.md, CLAUDE.md |
Workflow/docs updates reflecting new tool and parameters. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Copilot
AI
Apr 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_resolve_tests_dir() uses os.path.abspath/commonpath checks, which can be bypassed via symlinks (e.g., output_dir points to a symlink under problem_dir that resolves outside). That can still cause _clear_generated_tests() to delete .in/.ans files in external locations. Consider validating with os.path.realpath() (or Path.resolve()) for both problem_dir and tests_dir and/or rejecting symlinked output directories before clearing.
Copilot
AI
Apr 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_check_file_count() computes expected_indices as range(1, len(numeric_indices)+1), which under-reports gaps when filenames skip ahead (e.g., 01.in and 100.in would only report missing 2). If you want to enforce contiguous numbering, expected_indices should be based on max(numeric_indices) (and possibly also validate the set starts at 1).
Copilot
AI
Apr 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
workflow_guard introduces state["tests_verified"] and advertises a required problem_verify_tests step, but the state is never updated in post_tool() and pre_tool() still allows problem_pack_polygon after only tests_generated. This makes the new verification step unenforced in the guard. Update post_tool() to set tests_verified based on problem_verify_tests output (e.g., data.get("passed")) and gate problem_pack_polygon on tests_verified.
Copilot
AI
Apr 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment says this is saved to a "canonical/standard location that other tools depend on", but canonical_path is now derived from effective_name. Other parts of the codebase (e.g., Polygon packing XML uses solutions/sol.cpp & solutions/brute.cpp) still assume the standard filenames, so building with name!=solution_type can break downstream workflow. Consider either always also writing/copying to solutions/{solution_type}.cpp for backward compatibility, or updating downstream tools (e.g., problem_pack_polygon) to use the chosen name.
Summary
Verification