-
-
Notifications
You must be signed in to change notification settings - Fork 0
Document what the observability stack costs in RPS #187
Description
There is currently nothing in the docs about what turning an instrument on costs. It is a lot, and the ordering is not what people expect.
Measured on a trivial async def endpoint returning {"ok": True}, uvicorn single worker, ab -k -c 16 -n 20000. Apple M2, CPython 3.14.7, fastapi 0.141.1, sentry-sdk 2.67.1, opentelemetry-sdk 1.44.0, prometheus-fastapi-instrumentator 8.1.0.
| config | RPS | vs bare |
|---|---|---|
| bare FastAPI | 7978 | - |
| full lite-bootstrap stack (otel + prometheus + structlog + sentry) | 2389 | −70% |
| same stack, tuned | 4187 | −48% |
Per instrument, measured alone, driving the ASGI app in-process so only library cost shows (baseline 15.8 μs/req):
| instrument | +μs/req | share |
|---|---|---|
LoggingInstrument (configured, no logs emitted) |
+0.1 | ~0% |
PrometheusInstrument |
+17.5 | 8% |
SentryInstrument (tracing off) |
+58.5 | 27% |
OpenTelemetryInstrument |
+120.1 | 55% |
| all four | +217.1 |
Things a page like this should say, all of which surprised me:
- OpenTelemetry is twice Sentry, and is the dominant cost in the stack. Most people assume Sentry is the expensive one.
LoggingInstrumentis free until you actually log. The cost is per record (~8 μs with Sentry on), not per request.- The Sentry knobs people reach for do nothing.
attach_stacktrace=False,max_breadcrumbs=0and dropping the default integrations all measured within noise. The entire cost is the ASGI integration, and over half of that is aTransactionbuilt and discarded becausesentry_traces_sample_rateis unset. sentry_traces_sample_rate=1.0costs +274 μs/request on top of that (2486 RPS, −68% vs bare). Worth stating next to the setting.- Each saving has a price, and the page should be explicit about it: no breadcrumbs on error events, no release health, no Sentry-side trace correlation, sampled-away OTel traces.
Suggested home: a "Performance" or "What observability costs" page under Introduction, linked from each instrument's docs, with a short "if you are RPS-constrained, start here" section.
Caveats to include: these ratios are for a handler that does nothing. A service doing real work per request (a DB round trip, a downstream call) will see a much smaller relative hit - the absolute per-request cost is what transfers, not the percentage.
The benchmark harness that produced these numbers is reproducible and I can attach it if useful. Sibling issues cover the two OpenTelemetry knobs and the Sentry ones.