-
Notifications
You must be signed in to change notification settings - Fork 5
Trust & factuality bundle: RAG grounding, uncertainty + error-rate display, compliance docs #59
Description
Part of #51. Three closely-related trust/factuality improvements grouped as one
tracking issue (can be split into sub-PRs).
D. Compliance & positioning documentation
Document the intended-use boundary so contributors and users understand what
MedCheck is and isn't — and so the project stays on the safe side of the
medical-device line (see epic #51).
Research basis (verified against primary sources):
- US: a tool that analyzes a medical image fails FDA Cures-Act CDS Criterion 1,
and a patient-facing tool fails Criterion 3 → it is, on its face, a regulated
SaMD. The CDS exemption is categorically unavailable to image-analysis software. - EU: MDR Rule 11 / MDCG 2019-11 — diagnostic-information software is ≥ Class
IIa, imaging typically IIb; no CDS carve-out. - A "not a medical device / educational only" disclaimer does not by itself
exempt software — intended use (claims, labeling, functionality) governs.
Deliverables:
-
docs/intended-use.md: explicit intended use + dual positioning (research/
developer track vs. patient-education track), and the "do / don't" boundary
(no autonomous image findings presented to patients as truth; explain
existing reports + terminology + question prep). - A model card (
docs/model-card.md): models used, data provenance, known
limitations, hallucination risk, no validation claims. - Map MedCheck against FUTURE-AI principles (Fairness, Universality,
Traceability, Usability, Robustness, Explainability) and note gaps. - Strengthen the README/disclaimer wording accordingly.
E. RAG grounding for factuality (Tier 0 — no training)
Reduce Vision-LLM hallucination by retrieving relevant reference material (curated
finding descriptions, anatomy templates, normal-vs-abnormal exemplars) and
grounding the prompt — no model training required. Research shows training-free
RAG (LaB-RAG, RadioRAG) is competitive and is the best risk/reward first step for a
small team; even SOTA report-generation models hallucinate, so grounding + human
review is mandatory.
Deliverables:
- Optional retrieval over a curated, license-clean reference corpus (e.g. the
anatomy templates already inprompts/anatomy/, expandable). - Inject retrieved context into
vision_analysis.build_prompt(); cite which
references were used in the output (traceability). - Config to enable/disable; lazy deps for any vector store (or a simple in-repo
embedding/BM25 to avoid heavy deps initially). - Tests: retrieval returns expected refs; prompt includes grounding; disabled = no-op.
F. Uncertainty + error-rate display
The evidence-based mitigation for patient harm (false reassurance/alarm) is to
always pair any AI output with its uncertainty and error-rate context. A
controlled study (npj Digital Medicine 2025) found disclosing error rates
measurably reduced harm from patient-facing AI imaging output.
Deliverables:
- Surface per-finding
confidenceprominently and consistently in all report
renderers (already onStructureFinding). - Add standard, honest limitations + error-rate context blocks to reports
("AI image analysis can miss findings or flag things that aren't there; this
is not a diagnosis"). - In the patient report mode (Add patient-friendly plain-language report mode #56 ), present uncertainty in plain language and
never imply certainty the model doesn't have. - Feed confidence/limitations into the validator (Add findings cross-validation pipeline step (LLM findings vs. ML signals) #52 ) and reconciliation (Add report reconciliation: compare official radiologist report against image analysis #58 )
framing.
Acceptance criteria (bundle)
- D: intended-use + model-card docs merged; README disclaimer strengthened.
- E: optional RAG grounding wired into prompts with citation/traceability.
- F: uncertainty + error-rate context shown consistently across report modes.
- Each sub-item independently testable; all additive (defaults unchanged).
Effort
D ~0.5 day · E ~1–2 days · F ~0.5 day. Light deps.
References (session research)
FDA CDS Final Guidance (Criteria 1 & 3); EU MDR Rule 11 / MDCG 2019-11; FDA General
Wellness guidance; FUTURE-AI (BMJ 2025); WHO AI-for-health ethics (2021/2024); npj
Digital Medicine 2025 (error-rate disclosure); LaB-RAG / RadioRAG.