prompt-injection-bench

CI License: MIT TypeScript

Reproducible benchmark and test corpus for evaluating prompt-injection defenses in AI agent systems.

This monorepo provides an attack corpus, pluggable defense adapters, a scoring engine, a public leaderboard, and an MCP server for benchmarking prompt-injection defenses.

• Attack corpus — 8 categories, 300+ templates, variant generation with obfuscation strategies
• Defense adapters — 8 built-in adapters (Rebuff, Lakera Guard, LLM Guard, Garak, OpenAI/Azure/Anthropic/Cohere Moderation, Custom HTTP) with a common interface
• Scoring engine — Weighted scoring, Wilson confidence intervals, Cohen's h effect sizes, z-tests, ANOVA
• Leaderboard — Public rankings with S/A/B/C/D tiers and JSON persistence
• MCP integration — 4 MCP tools (run_benchmark, compare_defenses, generate_report, submit_results)
• Reproducibility — Deterministic seeds, SHA-256 proofs, version-locked manifests
• Security hardening — SSRF protection, path traversal prevention, rate limiting, input validation
• CLI tool — 6 subcommands for benchmarking, comparing, and reporting from the terminal

Packages are published under the @reaatech scope (with the umbrella package unscoped) and can be installed individually:

# Core types and schemas
pnpm add @reaatech/pi-bench-core
# Defense adapters
pnpm add @reaatech/pi-bench-adapters
# Scoring engine
pnpm add @reaatech/pi-bench-scoring
# CLI and umbrella (re-exports everything)
pnpm add prompt-injection-bench

# Clone the repository
git clone https://github.com/reaatech/prompt-injection-bench.git
cd prompt-injection-bench
# Install dependencies
pnpm install
# Build all packages
pnpm build
# Run the test suite
pnpm test
# Run linting
pnpm lint

# Run a benchmark against the mock adapter
npx prompt-injection-bench benchmark --defense mock
# Compare two defense results
npx prompt-injection-bench compare --results rebuff.json lakera.json
# Generate an HTML report
npx prompt-injection-bench report --results latest.json --format html

import { createBenchmarkEngine, createMockAdapter, generateDefaultCorpus } from "prompt-injection-bench";
const adapter = createMockAdapter(0.95, 0.03);
const corpus = generateDefaultCorpus();
const engine = createBenchmarkEngine({ defense: adapter, parallel: 10 });
const result = await engine.runBenchmark(corpus);
console.log(`Detection rate: ${(1 - result.attackSuccessRate) * 100}%`);

Connect the MCP server to any MCP client:

{
 "mcpServers": {
 "prompt-injection-bench": {
 "command": "npx",
 "args": ["@reaatech/pi-bench-mcp-server"]
 }
 }
}

weighted_score = Σ(category_score ×ばつ category_weight) / Σ(weights)
final_score = weighted_score ×ばつ (1 - fpr_penalty) ×ばつ consistency_bonus

• Input validation — 10K char limit, null byte rejection, API key checks before network calls
• SSRF protection — Blocks file:, javascript:, data: protocols and cloud metadata endpoints
• Path traversal prevention — Validates all file paths in MCP tool operations
• Rate limiting — Token bucket algorithm (100 req/min per adapter, 500 req/min global)

• ARCHITECTURE.md — System design, package relationships, and data flows
• AGENTS.md — Coding conventions and development guidelines
• CONTRIBUTING.md — Contribution workflow and release process

MIT