JsonRemedy

The Problem

// LLM output with mixed issues
```json
{
 users: [
 {name: 'Alice Johnson', active: True, scores: [95, 87, 92,]},
 {name: "Bob Smith", active: False /* incomplete
 ],
 metadata: None

# Legacy Python system output
{'users': [{'name': 'Alice', 'verified': True, 'data': None}]}

// Copy-paste from JavaScript console
{name: "Alice", getValue: function() { return "test"; }, data: [1,2,3]}

// Streaming API with connection drop
{"status": "processing", "results": [{"id": 1, "name": "Alice"

// Human input with common mistakes
{name: Alice, "age": 30, "scores": [95 87 92], active: true,}

Comprehensive Repair Capabilities

🧹 Content Cleaning (Layer 1)

🏗️ Structural Repairs (Layer 2)

✨ Syntax Normalization (Layer 3)

🔧 Hardcoded Patterns (ported from json_repair Python library)

🚀 Fast Path Validation (Layer 4)

🛟 Tolerant Parsing (Layer 5) ⏳ FUTURE

🧠 Context-Aware Intelligence

# ✅ PRESERVE: Comma inside string content
{"message": "Hello, world", "status": "ok"}
# ✅ REMOVE: Trailing comma
{"items": [1, 2, 3,]}
# ✅ PRESERVE: Numbers stay numbers 
{"count": 42}
# ✅ QUOTE: Unquoted keys get quoted
{name: "Alice"}
# ✅ PRESERVE: Boolean content in strings
{"note": "Set active to True"}
# ✅ NORMALIZE: Boolean values
{"active": True}
# ✅ PRESERVE: Escape sequences in strings
{"path": "C:\\Users\\Alice"}
# ✅ PARSE: Unicode escapes
{"unicode": "\\u0048\\u0065\\u006c\\u006c\\u006f"}

Quick Start

def deps do
 [
 {:json_remedy, "~> 0.1.4"}
 ]
end

Basic Usage

# Simple repair and parse
malformed = ~s|{name: "Alice", age: 30, active: True}|
{:ok, data} = JsonRemedy.repair(malformed)
# => %{"name" => "Alice", "age" => 30, "active" => true}
# Get the repaired JSON string
{:ok, fixed_json} = JsonRemedy.repair_to_string(malformed)
# => "{\"name\":\"Alice\",\"age\":30,\"active\":true}"
# Track what was repaired
{:ok, data, repairs} = JsonRemedy.repair(malformed, logging: true)
# => repairs: [
# %{layer: :syntax_normalization, action: "quoted unquoted key 'name'"},
# %{layer: :syntax_normalization, action: "normalized boolean True -> true"}
# ]

Real-World Examples

# LLM output with multiple issues
llm_output = """
Here's the user data you requested:

```json
{
 // User information
 users: [
 {
 name: 'Alice Johnson',
 email: "alice@example.com",
 age: 30,
 active: True,
 scores: [95, 87, 92,], // Test scores
 profile: {
 city: "New York",
 interests: ["coding", "music", "travel",]
 },
 },
 {
 name: 'Bob Smith',
 email: "bob@example.com", 
 age: 25,
 active: False
 // Missing comma above
 }
 ],
 metadata: {
 total: 2,
 updated: "2024-01-15"
 // Missing closing brace
```

That should give you what you need!
"""
{:ok, clean_data} = JsonRemedy.repair(llm_output)
# Works perfectly! Handles code fences, comments, quotes, booleans, trailing commas, missing delimiters

# Legacy Python-style JSON
python_json = ~s|{'users': [{'name': 'Alice', 'active': True, 'metadata': None}]}|
{:ok, data} = JsonRemedy.repair(python_json)
# => %{"users" => [%{"name" => "Alice", "active" => true, "metadata" => nil}]}
# JavaScript object literals
js_object = ~s|{name: "Alice", getValue: function() { return 42; }, data: [1,2,3]}|
{:ok, data} = JsonRemedy.repair(js_object)
# => %{"name" => "Alice", "data" => [1, 2, 3]} (function removed)
# Streaming/incomplete data
incomplete = ~s|{"status": "processing", "data": [1, 2, 3|
{:ok, data} = JsonRemedy.repair(incomplete)
# => %{"status" => "processing", "data" => [1, 2, 3]}
# Human input with common mistakes
human_input = ~s|{name: Alice, age: 30, scores: [95 87 92], active: true,}|
{:ok, data} = JsonRemedy.repair(human_input)
# => %{"name" => "Alice", "age" => 30, "scores" => [95, 87, 92], "active" => true}

Examples

📚 Basic Usage Examples

mix run examples/basic_usage.exs

🔧 Hardcoded Patterns Examples ✨ NEW

mix run examples/hardcoded_patterns_examples.exs

🌍 Real-World Scenarios

mix run examples/real_world_scenarios.exs

⚡ Performance Examples

mix run examples/quick_performance.exs

🔬 Stress Testing

mix run examples/simple_stress_test.exs

📊 Example Output

=== JsonRemedy Real-World Scenarios ===
Example 1: LLM/ChatGPT Output with Code Fences
==============================================
Input (LLM response with code fences and explanatory text):
Here's the user data you requested:
```json
{
 "users": [
 {name: "Alice Johnson", age: 32, role: "engineer"},
 {name: "Bob Smith", age: 28, role: "designer"}
 ],
 "metadata": {
 generated_at: "2024-01-15",
 total_count: 2,
 active_only: True
 }
}

Final repaired JSON:

All examples include detailed output showing:
- **Input analysis**: What's wrong with the JSON
- **Layer-by-layer processing**: Which layers made repairs
- **Final output**: Clean, valid JSON
- **Repair summary**: Detailed log of all fixes applied
- **Performance metrics**: Timing and throughput data
### 🎯 **Custom Examples**
Create your own examples using the same patterns:
```elixir
# examples/my_custom_example.exs
defmodule MyCustomExample do
 def test_my_json do
 malformed = ~s|{my: 'problematic', json: True}|
 
 case JsonRemedy.repair(malformed, logging: true) do
 {:ok, result, context} ->
 IO.puts("✓ Repaired successfully!")
 IO.puts("Result: #{Jason.encode!(result, pretty: true)}")
 IO.puts("Repairs: #{length(context.repairs)}")
 {:error, reason} ->
 IO.puts("✗ Failed: #{reason}")
 end
 end
end
MyCustomExample.test_my_json()

🔧 Example Status & Known Issues

Known Issue: Performance Benchmarks

# These work fine:
mix run examples/performance_benchmarks.exs # May hang on large datasets
# Alternatives that complete successfully:
mix run examples/quick_performance.exs # Lightweight performance testing
mix run examples/simple_stress_test.exs # Stress testing without hanging

Recent Fixes (v0.1.1)

Implementation Status

🗺️ Roadmap

The 5-Layer Architecture

defmodule JsonRemedy.LayeredRepair do
 def repair(input) do
 input
 |> Layer1.content_cleaning() # Cleaning: Remove wrappers, comments, normalize encoding
 |> Layer2.structural_repair() # State machine: Fix delimiters, nesting, structure 
 |> Layer3.syntax_normalization() # Char parsing: Fix quotes, booleans, commas
 |> Layer4.validation_attempt() # Jason.decode: Fast path for clean JSON
 |> Layer5.tolerant_parsing() # Custom parser: Handle edge cases gracefully (FUTURE)
 end
end

🧹 Layer 1: Content Cleaning

🏗️ Layer 2: Structural Repair

✨ Layer 3: Syntax Normalization

🚀 Layer 4: Validation

🛟 Layer 5: Tolerant Parsing ⏳ FUTURE

API Reference

Core Functions

# Main repair function
JsonRemedy.repair(json_string, opts \\ [])
# Returns: {:ok, term} | {:ok, term, repairs} | {:error, reason}
# Repair to JSON string
JsonRemedy.repair_to_string(json_string, opts \\ []) 
# Returns: {:ok, json_string} | {:error, reason}
# Repair from file
JsonRemedy.from_file(path, opts \\ [])
# Returns: {:ok, term} | {:ok, term, repairs} | {:error, reason}

Options

[
 # Return detailed repair log as third tuple element
 logging: true,
 
 # How aggressive to be with repairs
 strictness: :lenient, # :strict | :lenient | :permissive
 
 # Stop after successful layer (for performance)
 early_exit: true,
 
 # Maximum input size (security)
 max_size_mb: 10,
 
 # Processing timeout
 timeout_ms: 5000,
 
 # Custom repair rules for Layer 3
 custom_rules: [
 %{
 name: "fix_custom_pattern",
 pattern: ~r/special_pattern/,
 replacement: "fixed_pattern",
 condition: nil
 }
 ]
]

Advanced APIs

# Layer-specific processing (for custom pipelines)
JsonRemedy.Layer1.ContentCleaning.process(input, context)
JsonRemedy.Layer2.StructuralRepair.process(input, context) 
JsonRemedy.Layer3.SyntaxNormalization.process(input, context)
# Individual repair functions
JsonRemedy.Layer3.SyntaxNormalization.normalize_quotes(input)
JsonRemedy.Layer3.SyntaxNormalization.fix_commas(input)
JsonRemedy.Layer3.SyntaxNormalization.normalize_escape_sequences(input)
# Health checking
JsonRemedy.health_check()
# => %{status: :healthy, layers: [...], performance: {...}}

Streaming API

# Process large files efficiently
"huge_log.jsonl"
|> File.stream!()
|> JsonRemedy.repair_stream()
|> Stream.map(&process_record/1)
|> Stream.each(&store_record/1)
|> Stream.run()
# Real-time stream processing with buffering
websocket_stream
|> JsonRemedy.repair_stream(buffer_incomplete: true, chunk_size: 1024)
|> Stream.each(&handle_json/1)
|> Stream.run()
# Batch processing with error collection
inputs
|> JsonRemedy.repair_stream(collect_errors: true)
|> Enum.reduce({[], []}, fn
 {:ok, data} -> {[data | successes], errors}
 {:error, err} -> {successes, [err | errors]}
end)

Performance Characteristics

Benchmarks

Input Type | Throughput | Memory | Notes
------------------------------|---------------|-----------|------------------
Valid JSON (Layer 4 only) | TODO: | TODO: | Jason.decode fast path
Simple malformed | TODO: | TODO: | Layers 1-3 processing 
Complex malformed | TODO: | TODO: | Full pipeline
Large files (streaming) | TODO: | TODO: | Constant memory usage
LLM output (typical) | TODO: | TODO: | Mixed complexity

Performance Strategy

mix run bench/comprehensive_benchmark.exs
mix run bench/memory_profile.exs

Real-World Use Cases

🤖 LLM Integration

defmodule MyApp.LLMProcessor do
 def extract_structured_data(llm_response) do
 case JsonRemedy.repair(llm_response, logging: true, timeout_ms: 3000) do
 {:ok, data, []} -> 
 {:clean, data}
 {:ok, data, repairs} -> 
 Logger.info("LLM output required #{length(repairs)} repairs")
 maybe_retrain_model(repairs)
 {:repaired, data}
 {:error, reason} -> 
 Logger.error("Unparseable LLM output: #{reason}")
 {:unparseable, reason}
 end
 end
 
 defp maybe_retrain_model(repairs) do
 # Analyze repair patterns to improve LLM prompts
 serious_issues = Enum.filter(repairs, &(&1.layer == :structural_repair))
 if length(serious_issues) > 3, do: schedule_model_retraining()
 end
end

📊 Data Pipeline Healing

defmodule DataPipeline.JSONHealer do
 def process_external_api(response) do
 response.body
 |> JsonRemedy.repair(strictness: :lenient, max_size_mb: 50)
 |> case do
 {:ok, data} -> 
 validate_and_transform(data)
 {:error, reason} -> 
 send_to_deadletter_queue(response, reason)
 {:error, :unparseable}
 end
 end
 
 def heal_legacy_export(file_path) do
 file_path
 |> JsonRemedy.from_file(logging: true)
 |> case do
 {:ok, data, repairs} when length(repairs) > 0 ->
 Logger.warn("Legacy file required healing: #{inspect(repairs)}")
 maybe_update_source_system(file_path, repairs)
 {:ok, data}
 result -> result
 end
 end
end

🔧 Configuration Recovery

defmodule MyApp.ConfigLoader do
 def load_with_auto_repair(path) do
 case JsonRemedy.from_file(path, logging: true) do
 {:ok, config, []} -> 
 {:ok, config}
 {:ok, config, repairs} ->
 Logger.warn("Config file auto-repaired: #{format_repairs(repairs)}")
 maybe_write_fixed_config(path, config, repairs)
 {:ok, config}
 {:error, reason} ->
 {:error, "Config file unrecoverable: #{reason}"}
 end
 end
 
 defp maybe_write_fixed_config(path, config, repairs) do
 if mostly_syntax_fixes?(repairs) do
 backup_path = path <> ".backup"
 File.cp!(path, backup_path)
 
 fixed_json = Jason.encode!(config, pretty: true)
 File.write!(path, fixed_json)
 
 Logger.info("Auto-fixed config saved. Backup at #{backup_path}")
 end
 end
end

🌊 Stream Processing

defmodule LogProcessor do
 def process_json_logs(file_path) do
 file_path
 |> File.stream!(read_ahead: 100_000)
 |> JsonRemedy.repair_stream(
 buffer_incomplete: true,
 collect_errors: true,
 timeout_ms: 1000
 )
 |> Stream.filter(&valid_log_entry?/1)
 |> Stream.map(&enrich_log_entry/1)
 |> Stream.chunk_every(1000)
 |> Stream.each(&bulk_insert_logs/1)
 |> Stream.run()
 end
 
 def process_realtime_stream(websocket_pid) do
 websocket_pid
 |> stream_from_websocket()
 |> JsonRemedy.repair_stream(
 buffer_incomplete: true,
 max_buffer_size: 64_000,
 early_exit: true
 )
 |> Stream.each(&handle_realtime_event/1)
 |> Stream.run()
 end
end

🔬 Quality Assurance

defmodule QualityControl do
 def analyze_data_quality(source) do
 results = source
 |> stream_data()
 |> JsonRemedy.repair_stream(logging: true)
 |> Enum.reduce(%{total: 0, clean: 0, repaired: 0, failed: 0, repairs: []}, 
 fn result, acc ->
 case result do
 {:ok, _data, []} -> 
 %{acc | total: acc.total + 1, clean: acc.clean + 1}
 {:ok, _data, repairs} -> 
 %{acc | total: acc.total + 1, repaired: acc.repaired + 1, 
 repairs: acc.repairs ++ repairs}
 {:error, _} -> 
 %{acc | total: acc.total + 1, failed: acc.failed + 1}
 end
 end)
 
 generate_quality_report(results)
 end
 
 defp generate_quality_report(%{total: total, clean: clean, repaired: repaired, 
 failed: failed, repairs: repairs}) do
 %{
 summary: %{
 quality_score: (clean + repaired) / total * 100,
 clean_percentage: clean / total * 100,
 repair_rate: repaired / total * 100,
 failure_rate: failed / total * 100
 },
 top_issues: repair_frequency_analysis(repairs),
 recommendations: generate_recommendations(repairs)
 }
 end
end

Comparison with Alternatives

Advanced Features

Custom Repair Rules

# Define domain-specific repair rules
custom_rules = [
 %{
 name: "fix_currency_format",
 pattern: ~r/\$(\d+)/,
 replacement: ~S({"amount": 1,円 "currency": "USD"}),
 condition: &(!JsonRemedy.LayerBehaviour.inside_string?(&1, 0))
 },
 %{
 name: "normalize_dates",
 pattern: ~r/(\d{4})-(\d{2})-(\d{2})/,
 replacement: ~S("1円-2円-3円T00:00:00Z"),
 condition: nil
 }
]
{:ok, data} = JsonRemedy.repair(input, custom_rules: custom_rules)

Health Monitoring

# System health and performance monitoring
health = JsonRemedy.health_check()
# => %{
# status: :healthy,
# layers: [
# %{layer: :content_cleaning, status: :healthy, avg_time_us: 45},
# %{layer: :structural_repair, status: :healthy, avg_time_us: 120},
# # ...
# ],
# performance: %{
# cache_hit_rate: 0.85,
# avg_repair_time_us: 850,
# memory_usage_mb: 12.3
# }
# }
# Performance statistics
stats = JsonRemedy.performance_stats()
# => %{success_rate: 0.94, avg_time_us: 680, cache_hits: 1205}

Error Analysis

# Detailed error analysis for debugging
case JsonRemedy.repair(malformed_input, logging: true) do
 {:ok, data, repairs} ->
 analyze_repair_patterns(repairs)
 {:success, data}
 
 {:error, reason} ->
 case JsonRemedy.analyze_failure(malformed_input) do
 {:analyzable, issues} -> 
 Logger.error("Repair failed: #{inspect(issues)}")
 {:partial_analysis, issues}
 {:unanalyzable, _} -> 
 {:complete_failure, reason}
 end
end

Limitations and Design Philosophy

What JsonRemedy Excels At

What JsonRemedy Doesn't Do

Design Philosophy

Security Considerations

# Built-in security features
JsonRemedy.repair(input, [
 max_size_mb: 10, # Prevent memory exhaustion
 timeout_ms: 5000, # Prevent infinite processing
 max_nesting_depth: 50, # Prevent stack overflow
 disable_custom_rules: true # Disable user rules in untrusted contexts
])

Contributing

# Development setup
git clone https://github.com/nshkrdotcom/json_remedy.git
cd json_remedy
mix deps.get
# Run test suites
mix test # All tests
mix test --only unit # Unit tests only 
mix test --only integration # Integration tests
mix test --only performance # Performance validation
mix test --only property # Property-based tests
# Quality assurance
mix credo --strict # Code quality
mix dialyzer # Type analysis
mix format --check-formatted # Code formatting
mix test.coverage # Coverage analysis
# Benchmarking
mix run bench/comprehensive_benchmark.exs
mix run bench/memory_profile.exs

Architecture Overview

lib/
├── json_remedy.ex # Main API
├── json_remedy/
│ ├── layer_behaviour.ex # Common interface for all layers
│ ├── layer1/
│ │ └── content_cleaning.ex # ✅ Code fences, comments, wrappers
│ ├── layer2/
│ │ └── structural_repair.ex # ✅ Delimiters, nesting, state machine
│ ├── layer3/
│ │ └── syntax_normalization.ex # ✅ Quotes, booleans, char-by-char parsing
│ ├── layer4/
│ │ └── validation.ex # Jason.decode optimization
│ ├── layer5/ # ⏳ PLANNED
│ │ └── tolerant_parsing.ex # ⏳ Custom parser with error recovery
│ ├── pipeline.ex # Layer orchestration
│ ├── performance.ex # Monitoring and health checks
│ └── config.ex # Configuration management

Adding New Repair Capabilities

# 1. Add repair rule to Layer 3
@repair_rules [
 %{
 name: "fix_my_pattern",
 pattern: ~r/custom_pattern/,
 replacement: "fixed_pattern",
 condition: &my_condition_check/1
 }
 # existing rules...
]
# 2. Add test cases
test "fixes my custom pattern" do
 input = "input with custom_pattern"
 expected = "input with fixed_pattern"
 
 {:ok, result, context} = SyntaxNormalization.process(input, %{repairs: [], options: []})
 assert result == expected
 assert Enum.any?(context.repairs, &String.contains?(&1.action, "fix_my_pattern"))
end
# 3. Add to API documentation
@doc """
Fix my custom pattern in JSON strings.
"""
@spec fix_my_pattern(input :: String.t()) :: {String.t(), [repair_action()]}
def fix_my_pattern(input), do: apply_rule(input, @my_pattern_rule)

Roadmap

Version 0.2.0 - Enhanced Capabilities

Version 0.3.0 - Ecosystem Integration

Version 0.4.0 - Advanced Features

License