Agentic test data harness for Python — deterministic in CI, AI-powered in dev.
PyPI version Downloads Python 3.11+ pytest plugin MIT License
# ❌ Everyone does this. It's brittle and misses real-world edge cases. user = User(name="Test User", email="test@test.com", bio="Lorem ipsum...") # ❌ factory_boy is great — but it's static. No surprises, no edge cases. UserFactory.create(role="admin") # ❌ Writing 500 of them by hand? Not happening.
Hardcoded fixtures rot. AI-generated fixtures are unpredictable in CI.
FixtureForge solves both — same codebase, two behaviors:
Dev mode → AI generates rich, realistic, edge-case-aware fixtures
CI mode → same fixtures, frozen with seed=42, 100% reproducible
pip install fixtureforge
from fixtureforge import Forge from pydantic import BaseModel class User(BaseModel): id: int name: str email: str bio: str forge = Forge() # auto-detects AI provider from env vars users = forge.create_batch(User, count=50, context="SaaS platform users")
FixtureForge routes each field to the cheapest generator automatically:
id→ sequential counter (free)name,email→ Faker (free)bio→ single batched AI call for all 50 records (1 API call, not 50)
No AI key? No problem. Pure Faker mode works out of the box:
forge = Forge(use_ai=False, seed=42) # deterministic, zero network, CI-safe users = forge.create_batch(User, count=500)
Every field is classified into a tier. Only semantic fields hit the AI:
| Tier | Fields | Generator | Cost |
|---|---|---|---|
| Structural | id, user_id, order_id |
Counters + FK registry | Free |
| Standard | name, email, phone, address |
Faker | Free |
| Computed | @computed_field |
Pydantic | Free |
| Semantic | bio, description, review, message |
LLM (batched) | API tokens |
100 users with 2 semantic fields = 2 API calls, not 200.
# .github/workflows/test.yml - name: Run tests env: FORGE_SEED: 42 # identical output every run # No AI key needed — FixtureForge auto-detects and falls back to Faker run: pytest
In dev, export any provider key and AI kicks in automatically:
export ANTHROPIC_API_KEY=sk-ant-... # → Claude export OPENAI_API_KEY=sk-... # → GPT export GOOGLE_API_KEY=... # → Gemini export GROQ_API_KEY=... # → Groq (fast + cheap)
# Step 1: generate customers customers = forge.create_batch(Customer, count=10) # Step 2: Order.customer_id auto-resolves to a real customer.id orders = forge.create_batch(Order, count=100) # → every order.customer_id is valid. No manual wiring.
Generate multiple models in parallel. The first model warms the AI cache;
every subsequent model inherits it — ~90% cheaper per additional model.
results = forge.swarm( models=[User, Order, Product, Payment], counts=[10, 50, 100, 30], contexts=["SaaS users", "E-commerce orders", None, None], ) # { # "User": [...10 users...], # "Order": [...50 orders...], # "Product": [...100 products...], # "Payment": [...30 payments...], # }
5 models ≈ cost of 1.5 models.
forge.memory.add_rule("financial", "Users under 18 get restricted account type") forge.memory.add_rule("user", "Israeli phone numbers use format 05x-xxx-xxxx") forge.memory.add_rule("orders", "Max 3 active loans per customer at any time") # Rules inject into AI prompts automatically on every generation call users = forge.create_batch(User, count=50, context="Israeli SaaS platform")
Rules survive across sessions. Update a rule — next call respects it immediately.
Skeptical Memory validates stored rules against the live schema before every call.
# conftest.py from fixtureforge.pytest_plugin import forge_fixture, forge_swarm_fixture from pydantic import BaseModel class User(BaseModel): id: int; name: str; email: str forge_fixture(User, count=10, seed=42) # → fixture: "users" forge_swarm_fixture([User, Order], counts=[5, 20], seed=42) # → "swarm_data"
# test_users.py def test_signup(users): for user in users: assert "@" in user.email def test_full_flow(swarm_data): users = swarm_data["User"] orders = swarm_data["Order"]
The forge fixture is auto-available in every test with zero config.
# Be explicit forge = Forge(provider_name="anthropic", model="claude-haiku-4-5-20251001") forge = Forge(provider_name="openai", model="gpt-4o-mini") forge = Forge(provider_name="gemini", model="gemini-2.0-flash") forge = Forge(provider_name="groq", model="llama-3.3-70b-versatile") forge = Forge(provider_name="ollama", model="llama3.2") # local, zero cost forge = Forge(use_ai=False) # pure Faker
# Seed + Interpolation: generates ~1 000 unique AI values, tiles to 100 000 dataset = forge.create_large(Order, count=100_000, seed_ratio=0.01) # Streaming — one record at a time, never loads all into memory for user in forge.create_stream(User, count=1_000_000, filename="users.json"): process(user)
from fixtureforge.core.exporter import DataExporter users = forge.create_batch(User, count=100) DataExporter.to_json(users, "users.json") DataExporter.to_csv(users, "users.csv") DataExporter.to_sql(users, "users.sql", table_name="users")
| FixtureForge | factory_boy | Faker | hypothesis | |
|---|---|---|---|---|
| AI-powered context | ✅ | ❌ | ❌ | ❌ |
| Deterministic (seed=) | ✅ | ✅ | ✅ | ✅ |
| FK relationships | Auto | Manual | ❌ | ❌ |
| Batched AI calls | ✅ | ❌ | ❌ | ❌ |
| Coverage gap analysis | ✅ | ❌ | ❌ | Partial |
| Large datasets (100k+) | ✅ | Manual | Manual | ❌ |
| pytest plugin | ✅ | ❌ | ❌ | ❌ |
| Multi-LLM support | ✅ | ❌ | ❌ | ❌ |
| Permission gates | ✅ | ❌ | ❌ | ❌ |
| CI-safe (zero network) | ✅ | ✅ | ✅ | ✅ |
FixtureForge is not a replacement for Faker — it uses Faker internally for standard fields.
It adds the layer between "I need realistic data" and "I need it to feel like production."
# Core (deterministic mode, no AI) pip install fixtureforge # With your preferred provider pip install "fixtureforge[anthropic]" # Claude pip install "fixtureforge[openai]" # GPT pip install "fixtureforge[gemini]" # Gemini pip install "fixtureforge[all]" # All providers
Requirements: Python 3.11+ · pydantic ≥ 2.5 · faker ≥ 22.0
For teams with compliance requirements — GDPR, SOC2, HIPAA, multi-tenant SaaS:
| Community | Enterprise | |
|---|---|---|
| AI generation + Faker | ✅ | ✅ |
| pytest plugin | ✅ | ✅ |
| Deterministic seeding | ✅ | ✅ |
| Cryptographic Provenance Envelope | ❌ | ✅ |
| PII Airgap — fail-closed scanner | ❌ | ✅ |
| Contextual Tenant Enclaves | ❌ | ✅ |
| Cross-tenant FK violation detection | ❌ | ✅ |
| Presidio / custom scanner support | ❌ | ✅ |
from fixtureforge.enterprise import ForgeEnterprise forge = ForgeEnterprise(use_ai=False) users = forge.create_batch(User, count=10) users[0].model_dump() # { "id": 1, "name": "...", "bio": "...", # "forge_metadata": { # "forge_id": "abc-123", # "provenance_hash": "sha256:029773ed...", ← immutable audit stamp # "tenant_id": "tenant-acme", # "source": "faker", # ... # } # } with forge.isolate_tenant("tenant-acme"): acme_users = forge.create_batch(User, count=5) # FK references from tenant-acme can NEVER resolve to tenant-xyz records
Access: yaniv2809@gmail.com
| Component | Status |
|---|---|
Core (Forge, create_batch) |
✅ Stable |
| DataSwarms | ✅ Stable |
| ForgeMemory | ✅ Stable |
| pytest plugin | ✅ v2.2.0 |
| Anthropic / OpenAI / Gemini / Groq / Ollama | ✅ |
assert_semantic_match |
✅ v2.2.0 |
SmartFailureAnalyzer |
✅ v2.2.0 |
| Enterprise Edition | ✅ (access by request) |
| ForgeDream (coverage analysis) | 🔜 Feature-flagged |
| Async support | 🔜 Planned |
- Docs: https://yaniv2809.github.io/fixtureforge/
- PyPI: https://pypi.org/project/fixtureforge/
- Repository: https://github.com/Yaniv2809/fixtureforge
- Issues: https://github.com/Yaniv2809/fixtureforge/issues
- Discussion: #1
Issues and PRs welcome.
git clone https://github.com/Yaniv2809/fixtureforge cd fixtureforge pip install -e ".[dev]" PYTHONPATH=src python -m pytest tests/
MIT © Yaniv2809
If FixtureForge saved you time, give it a ⭐ — it helps others find it.