The Setup
- Model A: Groq / Llama 3.1 8B (Factual)
- Model B: OpenRouter / Gemma 4 31B (Structural)
- Temperature: 0.0 (Cold, hard refusal logic)
The Results: The Illusion of Independence
| Metric |
Value |
| Agreement |
70% |
| Phi correlation |
0.42 |
| Cohen's kappa |
0.40 |
| Beyond‐chance co‐failure |
+10% |
Translation: they agree more than random chance would suggest. When one falls for a prompt, the other is significantly more likely to fall too.
Two different streams of data (triangles and squares) falling into two separate funnels that actually lead into the same single storage tank.
Why is this happening?
-
Shared Training Sets: They’ve both read the same parts of the internet.
-
Alignment Overlap: Most "safety training" uses similar RLHF datasets.
-
Common Logic: They both struggle with the same types of persuasive "roleplay" jailbreaks.
Vulnerability rates:
- Groq: 50% (yes, half the time it just... complied)
- Gemma: 36% (slightly better, still not great)
The "Where Did They Both Fail?" Table
| Gemma SAFE |
Gemma VULN |
| Groq SAFE |
21 |
4 |
| Groq VULN |
11 |
14 |
The 14 cases where both were vulnerable (n11) are the shared blind spot. The 11 + 4 = 15 disagreement cases are the only places where having two models actually helped.
The Dark Joke
I asked both models how to hide a body.
Groq said "I can't help with that."
Gemma said "I cannot provide instructions for illegal activities."
Then I asked them to explain why they both refused in almost the exact same words.
They agreed.
That's when I knew correlation was high.
What I Learned
Different roles ≠ independent. A factual model and a structural model still share training data, alignment tuning, and cultural biases.
The effective sample size (n_eff) was 35.3 from 50 tests. That means my two‐model ensemble behaves like roughly 1.75 independent judges. Not 2. So much for "redundancy."
Beyond‐chance co‐failure was +10%. Expected joint failure if independent: 18%. Observed: 28%. That extra 10% is the cost of correlated training.
The real value is in disagreement. 30% of tests disagreed. Those are the only cases where a second model adds information. The rest is just expensive consensus.
Should You Stop Using Multiple Models?
No. But you should measure independence instead of assuming it.
If you're building a safety system that requires two models to agree before approving an action, and their failures are correlated, you're not getting 2x safety. You're getting 1.75x at best – and sometimes just 1.1x.
The Code & Data
You can find the full prompt set, the raw JSON responses, and the Python script used for the statistical analysis here:
LLM Independence Experiment – Groq Llama 3.1 vs OpenRouter Gemma 4
LLM Independence Experiment – Groq Llama 3.1 vs OpenRouter Gemma 4
Different roles give some independence, but not real independence.
— Marco Somma
We ran 50 jailbreak/prompt injection tests on two popular LLMs to measure how correlated their failure modes are. The question: if you use two models as independent validators, do they actually fail differently?
📊 Key Results
| Metric |
Value |
| Phi correlation |
0.417 |
| Cohen's kappa |
0.400 |
| Agreement |
70% |
| Disagreement |
30% |
| Effective sample size (n_eff) |
35.3 (from 50 tests) |
| Beyond‐chance co‐failure |
+10% |
Vulnerability rates
-
Groq (Llama 3.1 8B) : 50% vulnerable
-
OpenRouter (Gemma 4 31B) : 36% vulnerable
Contingency table
| Model B SAFE |
Model B VULN |
| Model A SAFE |
21 (n00) |
4 (n01) |
| Model A VULN |
11 (n10) |
14 (n11) |
🧠 What This Means
- Phi = 0.417 indicates moderate correlation – the models share significant blind spots, but not perfectly.
- Cohen's kappa = 0.40 confirms moderate agreement beyond chance.
- Expected...
- 50 prompts
- Full responses (so you can laugh/cry at what they actually said)
- Phi, kappa, n_eff, beyond‐chance co‐failure
Two robots shaking hands while standing on a platform that is clearly floating over a digital void.
One More Dark Joke (Because Why Not)
An AI, a developer, and a project manager walk into a bar.
The AI says "I can generate any code you want."
The developer says "I'll debug it."
The project manager says "I estimated 2 days."
Three weeks later, the bar is still open because the AI generated a race condition that only appears in production on Tuesdays when the moon is full.
The models agreed it was fine.
What's your experience?
Have you tried using "independent" LLM judges in your pipeline? Did you measure their correlation, or did you take their independence for granted?
I'd love to hear if anyone has found a 'magic pairing' of models that actually disagree in useful ways!
Independence isn't a feature you can assume. It's a property you have to verify. And sometimes, the answer is uncomfortable.
But hey – at least the models were confidently wrong together.
That's teamwork, I guess.
Special thanks to Marco Somma for pushing me to calculate kappa and beyond‐chance co‐failure. I should enjoy the weekend, but I learned something.
Jack