Bug

Evaluation/metrics/answer_accuracy.py::calculate_factuality parses the classifier reply with bare json.loads(response.content) inside try/except: return 0.0. The default judge (gpt-4o-mini, temperature=0, no JSON mode — as generation_eval.py builds it) returns the JSON wrapped as Output: {...}\n\nReasoning: ..., so json.loads raises JSONDecodeError and factuality silently returns 0.0 for every sample. ACC = 0.75·factuality + 0.25·cosine therefore degenerates to 0.25·cosine and cannot separate correct from incorrect answers.

This is the only LLM-parse site in the metrics not using JSONHandler.parse_with_fallbacks — generate_statements (same file), coverage.py, and faithfulness.py all route through it and are unaffected.

Fix

Route the factuality parse through the same handler (already imported at the top of the file):

parsed = await JSONHandler().parse_with_fallbacks(response.content)
classification = ClassificationWithReason(**parsed)

Verification

compute_answer_correctness on a perfect-match pair, with the repo's own config (gpt-4o-mini + BAAI/bge-large-en-v1.5, CPU):

compute_answer_correctness(
 "What did the cat do?",
 "The cat sat on the mat.",
 "The cat sat on the mat.", # answer == ground_truth
)