-
Notifications
You must be signed in to change notification settings - Fork 0
PDF/OCR ingestion (M2) #1
Open
Description
PDF / OCR ingestion (M2, deferred)
Real clinical source documents arrive as PDFs and scans. The ingestion boundary
currently accepts text; there is no extraction path.
Requirements
- Text-layer first: extract embedded text (pdfminer/pypdf) when present — no
OCR needed (the TCGA-Reports corpus is already OCR-cleaned this way). - OCR fallback only when there is no text layer (tesseract via
pytesseract),
behind an optional extra so the core stays thin; skip-if-missing in CI. - OCR/extracted text MUST pass through the pseudonymization boundary before it
reaches the casebank/store — OCR output is PHI-bearing and must not bypass
harness/ingest. - Capture per-element provenance (source file, page, text-layer vs OCR, OCR
confidence) so the input contract can later correlate performance with input
quality.
Verify
- A tiny bundled text-layer PDF fixture round-trips to text (offline-testable).
- Property test: extracted text with known synthetic identifiers is pseudonymized
before anystore/write (no raw identifier reaches the store).
Scope note
External binary (tesseract) + real documents mean this is not fully offline-
verifiable beyond the text-layer fixture. Gate OCR behind [ocr] and a
skip-if-missing test.