feat(inference): extract geospatial workloads from backend - #673

Summary

Extract geospatial triangulation and timezone lookup from the database-facing backend into an independently deployed inference service.

• Backend calls inference over authenticated internal HTTP; no Redis, shared cache, or local fallback.
• Inference receives no PostgreSQL, S3, JWT, temporal-model, risk-service, or superadmin credentials.
• Inference is attached only to an internal compute network; it has no database network route and no published port.
• Validation-worker failures continue through the existing lease/retry/dead-letter path.
• Delete, label, and unmatch operations precompute the resulting alert state before any mutation; inference failure returns HTTP 503 with database state unchanged.
• Notification timezone lookup stays best-effort: lookup failure is logged and notification delivery is skipped.

Architecture

flowchart LR
 U["Clients / cameras"] -->|"published API :5050"| B["Backend\nDB + storage + JWT credentials\ndata and compute networks"]
 B -->|"data network"| DB[("PostgreSQL")]
 B -->|"data network"| S3["S3 / LocalStack"]
 B -->|"compute network\nBearer token"| I["Inference\ntriangulation + timezone\n1 worker by default"]
 I -. "no route" .-> X[("Database blocked")]
 I -. "no credentials" .-> Y["S3 / JWT / backend secrets"]
 classDef isolated fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
 classDef blocked fill:#ffebee,stroke:#c62828,stroke-dasharray:5 5;
 class I isolated;
 class X,Y blocked;

The backend remains the sole authority for persistence and orchestration. Inference is a stateless compute boundary with three endpoints:

• POST /v1/triangulate
• POST /v1/timezone
• GET /status (health only; unauthenticated)

The two compute endpoints validate bearer authentication, coordinate bounds, finite numeric values, unique sequence IDs, timestamps, classifications, and canonical response shape.

Dependency and image impact

flowchart LR
 BEFORE["Backend before\n26 direct\n35 transitive-only\n61 total\n581.6 MB"]
 AFTER["Backend after\n19 direct\n29 transitive-only\n48 total\n299.3 MB"]
 INF["Inference after\n7 direct\n17 transitive-only\n24 total\n385.7 MB"]
 BEFORE -->|"remove 7 direct / 6 transitive-only\n-282.2 MB (-48.5%)"| AFTER
 BEFORE -->|"extract native geospatial closure"| INF

Dependency closures are counted per image from uv export --only-group <group> --no-dev --no-hashes --no-emit-project; "transitive-only" is the closure minus direct declarations. Packages shared by both groups are counted in both images. Raw image sizes therefore must not be summed as host disk usage because shared Docker layers may be deduplicated.

Image measurements use docker image inspect .Size on Linux/arm64 images built from the same checkout and python:3.11-slim Dockerfile:

• baseline backend: 581,552,675 bytes
• extracted backend: 299,322,028 bytes
• inference: 385,675,379 bytes

Package decisions

Failure semantics

sequenceDiagram
 participant W as Validation worker
 participant B as Backend
 participant I as Inference
 participant D as Database
 W->>B: validate sequence
 B->>I: authenticated triangulation
 alt inference succeeds
 I-->>B: canonical groups + location
 B->>D: persist alert state
 else timeout / 4xx / 5xx / malformed response
 I--xB: unavailable
 B-->>W: raise
 W->>D: retain due marker / retry lease
 end

Interactive mutations use the same ordering: fetch state → call inference → mutate only after success. There is no local compute fallback that could silently diverge.

Verification

• 637 backend tests passed in the Docker Compose stack.
• 12 inference tests passed, covering authentication, validation, deterministic output, timezone fallback, empty/singleton/relaxed-time/same-pose/same-mast/mixed/dateline behavior, and responsive health checks during serialized compute.
• Ruff, ty, lock check, and dependency-sync verification passed.
• Both production Dockerfiles build successfully.
• Backend image cannot import geopy, networkx, numpy, pandas, pyproj, shapely, or timezonefinder.
• Local isolation inspection: inference had only pyronear_compute, no sensitive environment variables, and could not resolve db.
• Authenticated backend-to-inference smoke: triangulation 200; timezone 200 (Europe/Paris).

Synthetic in-process triangulation benchmark (median of 3 runs, Linux/arm64 dependency environment):

Deployment and rollback

Release automation builds and publishes both images, uploads the current Compose definition, provisions INFERENCE_API_TOKEN from GitHub Secrets, starts healthy inference first, then switches backend. CPU work runs off the event loop behind a per-process lock, so triangulations remain serialized while /status and timezone lookup stay responsive. INFERENCE_WORKERS defaults to 1 and can scale independently later.

Rollback remains backend-only: restore the previous backend image; the stateless inference container may remain running.

Production acceptance still required

• Configure a strong repository secret named INFERENCE_API_TOKEN for VPS rollout.
• Run authenticated triangulation and timezone notification smoke tests.
• Confirm inference has no database credentials or connectivity in production.
• Confirm end-to-end alert creation on the VPS.