Skip to content

Navigation Menu

Sign in
Sign up

PDF/OCR ingestion (M2) #1

Open
Open
Labels
enhancementNew feature or request

Description

PDF / OCR ingestion (M2, deferred)

Real clinical source documents arrive as PDFs and scans. The ingestion boundary
currently accepts text; there is no extraction path.

Requirements

  • Text-layer first: extract embedded text (pdfminer/pypdf) when present — no
    OCR needed (the TCGA-Reports corpus is already OCR-cleaned this way).
  • OCR fallback only when there is no text layer (tesseract via pytesseract),
    behind an optional extra so the core stays thin; skip-if-missing in CI.
  • OCR/extracted text MUST pass through the pseudonymization boundary before it
    reaches the casebank/store — OCR output is PHI-bearing and must not bypass
    harness/ingest.
  • Capture per-element provenance (source file, page, text-layer vs OCR, OCR
    confidence) so the input contract can later correlate performance with input
    quality.

Verify

  • A tiny bundled text-layer PDF fixture round-trips to text (offline-testable).
  • Property test: extracted text with known synthetic identifiers is pseudonymized
    before any store/ write (no raw identifier reaches the store).

Scope note

External binary (tesseract) + real documents mean this is not fully offline-
verifiable beyond the text-layer fixture. Gate OCR behind [ocr] and a
skip-if-missing test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      AltStyle によって変換されたページ (->オリジナル) /