1
0
Fork
You've already forked propose
0
No description
  • Python 100%
William Waites 21b524e378 moved
2026年01月26日 02:20:32 +00:00
examples fix example path 2026年01月23日 11:55:57 +00:00
propose session update command, prompt tweaks, and acceptance threshold from the command line 2026年01月23日 19:50:35 +00:00
notes.org add notes, tweak readme 2026年01月23日 12:04:44 +00:00
pyproject.toml Initial revision of working adversarial proposal editor 2026年01月21日 12:29:21 +00:00
README.md moved 2026年01月26日 02:20:32 +00:00

Multi-Agent Document Optimisation

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! !!
!! THIS PROJECT HAS MOVED !!
!! !!
!! IT IS NOW AT !!
!! !!
!! https://codeberg.org/wwaites/persevere !!
!! !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

A lightweight multi-agent LLM pipeline for iteratively drafting and improving documents through adversarial optimization.

Note that this is a Proof of Concept Demonstration and at least partly an exercise in learning MCP and PydanticAI. It needs refactored. It also needs a decent test harness.

Overview

This system automates the process of creating high-quality documents (cover letters, CVs, proposal summaries, etc.) to a specification (job adverts, call for proposals, etc) by using multiple AI agents that simulate the application review process. The workflow creates an adversarial optimization loop where:

  1. An applicant agent drafts application materials from source documents
  2. A fact-checker agent verifies all claims against supporting materials
  3. A reviewer agent scores and critiques the draft from the hiring manager's perspective
  4. The cycle repeats until quality targets are met or iteration limits are reached

The fact-checker ensures honesty by rejecting unsubstantiated claims, while the reviewer provides critical feedback to drive improvement. A fourth compressor agent manages token budgets by condensing conversation history when necessary. The fact checker also means that the application is really only as good as the input documents: the system is good for refining and polishing and not useful for idea generation and brainstorming.

Key Features

  • Document visibility scoping: Separate applicant-visible and reviewer-visible document sets prevent information leakage
  • Adversarial optimization: Reviewer agent simulates competitive selection pressure
  • Fact-checking: Prevents hallucination and unsubstantiated claims
  • Session-based workspaces: Organize materials for multiple applications
  • Cost instrumentation: Track token usage and API costs
  • Context window management: Automatic semantic history compression preserves system prompts while staying within limits
  • Reproducibility: SQLite-backed persistence enables auditing and iteration (not, however, reproducibility in the scientific sense)

Architecture

Tech Stack

  • Python 3.11+
  • PydanticAI Graphs: Multi-agent orchestration
  • Model Context Protocol (MCP): Tool/document exposure
  • SQLAlchemy + SQLite: Document storage and session management
  • OpenAI / Anthropic APIs: LLM providers

Document Model

Documents are stored with structured metadata:

  • Type: cv, job_description, cover_letter, publication, report, guidance, criticism, other
  • Visibility:
    • applicant: Visible to applicant agent only
    • reviewer: Visible to reviewer agent only
    • public: Visible to both agents
    • draft: Stores work-in-progress, not visible to agents
    • feedback: Stores reviews/critiques, not visible to agents
  • Content source: user, llm, url
  • Session association: Documents belong to a named session workspace

Agent Graph

┌─────────────┐ ┌─────────────┐
│ Applicant │──────┐ │ Context │
│ Agent │ │ │ Compressor │
└─────────────┘ │ └─────────────┘
 ↑ ↓
 │ ┌──────────────┐
 │ │ Fact-Checker │
 │ │ Agent │
 │ └──────────────┘
 │ │
 │ ↓ (if substantiated)
 │ ┌──────────────┐
 └──────│ Reviewer │
 │ Agent │
 └──────────────┘
 │
 └─→ Score + Feedback → Iterate or Terminate

Each agent has access to a specialized MCP endpoint exposing only documents with appropriate visibility. This prevents "tool-based cheating" where agents read documents they shouldn't see, even when strongly instructed not to.

Compressor Agent

Before each agent run, the compressor checks conversation history token count. If it exceeds a configurable threshold, it summarizes the conversation while preserving the initial system prompt. This maintains context while preventing window overflow.

Model Selection

Development uses cost-efficient models:

  • OpenAI gpt-5-mini (gpt-5-nano lacks sufficient tool-use capability)
  • Anthropic claude-haiku-4-5 (significantly better performance)

Production-quality runs use:

  • OpenAI gpt-5.2
  • Anthropic claude-sonnet-4-5 (best results when it doesn't time out)

Cost: 8–10 rounds of optimization typically cost 1ドル–2ドル with Haiku or ChatGPT 5.2, significantly more with Sonnet (when it doesn't time out)

Design Rationale

Why visibility scoping?

Agents instructed not to read certain documents will often do so anyway if the tools are available. The MCP self-documentation acts like instructions, making it too tempting to "cheat." Separate MCP endpoints per agent role enforce information boundaries programmatically.

Why a fact-checker?

LLMs readily hallucinate accomplishments, publications, and experiences. The fact-checker, with access to the same source materials as the applicant, acts as a gatekeeper to maintain honesty.

Why adversarial review?

Simulating a reviewer agent creates competitive pressure to improve. The reviewer sees only what a real hiring manager would see (job description, public info) and scores accordingly, driving the applicant to strengthen weak areas.

Why history compression?

With many documents and multiple iterations, token counts can exceed context windows (especially for cheaper models). A dedicated compressor agent preserves information density while staying within limits, unlike naive truncation.

System Prompts

High-quality system prompts are critical. Agents must be:

  • Grounded: Instructed to stick to provided materials
  • Role-aware: Understand their adversarial/cooperative position
  • Tool-disciplined: Use only appropriate tools for their visibility scope
  • Structured: Return outputs in predictable formats (Pydantic models)

Prompt engineering proved more important than model selection for preventing hallucination and off-task behavior.

Workflow Example

  1. Create a session:

    session_create(user="alice", name="Software Engineer @ Acme", setting="job")

  2. Add materials:

    document_create(session_id="...", visibility="applicant", 
 document_type="cv", content_source="user", 
 title="My CV", content="...")
document_create(session_id="...", visibility="public",
 document_type="job_description", content_source="url",
 title="Job Advert", content="...")

  3. Run optimization loop (internal agent graph):

    • Applicant drafts cover letter, cv, or proposal → draft visibility
    • Fact-checker validates → pass/fail
    • Reviewer scores and critiques → feedback visibility
    • Loop continues until score threshold met or max iterations

  4. Retrieve final draft:

    draft_document_list(session_id="...")
document_get(document_id="best-draft-id")

  5. Export by updating visibility to applicant or extracting content

MCP Tool Documentation

Session Management

session_create

Create a new proposal generation session.

Parameters:

  • user (string, required): User identifier
  • name (string, required): Session name
  • setting (enum, required): One of job, grant, procurement, other

Returns: Session object with id, created_at, updated_at

session_list

List sessions for a user.

Parameters:

  • user (string, required): User identifier
  • limit (integer, optional): Max results (default: 20)

Returns: Array of session objects

session_get

Retrieve a session by ID.

Parameters:

  • session_id (string, required): Session identifier

Returns: Session object

session_rename

Rename an existing session.

Parameters:

  • session_id (string, required): Session identifier
  • name (string, required): New session name

session_delete

Delete a session and all associated documents.

Parameters:

  • user (string, required): User identifier
  • session_id (string, required): Session identifier

Document Management

document_create

Add a document to a session.

Parameters:

  • session_id (string, required): Target session
  • visibility (enum, required): One of public, reviewer, applicant, draft, feedback
  • document_type (enum, required): One of job_description, call_for_proposals, organisation_info, evaluation_criteria, guidance, cv, cover_letter, case_for_support, executive_summary, skill_summary, report, publication, project_ideas, biography, instructions, criticism, other
  • content_source (enum, required): One of user, llm, url
  • title (string, required): Document title
  • content (string, required): Full document text (plain text or markdown)

Returns: Document object with id, summary, length, created_at

document_create_from_url

Add a document from a URL (extracts text from HTML or PDF).

Parameters:

  • session_id (string, required): Target session
  • visibility (enum, required): Visibility scope
  • document_type (enum, required): Document type
  • plain_text (boolean, required): Whether to convert HTML to plain text
  • url (string, required): Source URL

Returns: Document object

document_get

Retrieve a document by ID, including full content.

Parameters:

  • document_id (string, required): Document identifier

Returns: Complete document object with content field

document_update

Update document metadata.

Parameters:

  • document_id (string, required): Document identifier
  • visibility (enum, optional): New visibility
  • document_type (enum, optional): New type
  • content_source (enum, optional): New source
  • title (string, optional): New title
  • summary (string, optional): New summary

document_delete

Delete a document.

Parameters:

  • document_id (string, required): Document identifier

Document Queries

document_list

List all documents in a session (unrestricted; main MCP endpoint only).

Parameters:

  • session_id (string, required): Session identifier
  • doctype (enum, optional): Filter by document type
  • visibility (enum, optional): Filter by visibility

Returns: Array of document summaries (excludes content)

applicant_document_list

List documents visible to the applicant agent (applicant + public visibility).

Parameters:

  • session_id (string, required): Session identifier

Returns: Array of document summaries

reviewer_document_list

List documents visible to the reviewer agent (reviewer + public visibility).

Parameters:

  • session_id (string, required): Session identifier

Returns: Array of document summaries

draft_document_list

List draft documents (work-in-progress, not visible to agents).

Parameters:

  • session_id (string, required): Session identifier

Returns: Array of draft document summaries

feedback_document_list

List feedback/review documents (not visible to agents).

Parameters:

  • session_id (string, required): Session identifier

Returns: Array of feedback document summaries

Cost Tracking

cost_summary

Summarize API costs for a session.

Parameters:

  • session_id (string, required): Session identifier

Returns: Cost breakdown by model/provider

Acknowledgments

Built using the PydanticAI Graphs multi-agent example as a starting point, extended significantly for production use with MCP integration, visibility scoping, fact-checking, and cost management.