Skip to content

Navigation Menu

Sign in
Sign up

Document what the observability stack costs in RPS #187

Open
Labels
documentationImprovements or additions to documentation

Description

There is currently nothing in the docs about what turning an instrument on costs. It is a lot, and the ordering is not what people expect.

Measured on a trivial async def endpoint returning {"ok": True}, uvicorn single worker, ab -k -c 16 -n 20000. Apple M2, CPython 3.14.7, fastapi 0.141.1, sentry-sdk 2.67.1, opentelemetry-sdk 1.44.0, prometheus-fastapi-instrumentator 8.1.0.

config RPS vs bare
bare FastAPI 7978 -
full lite-bootstrap stack (otel + prometheus + structlog + sentry) 2389 −70%
same stack, tuned 4187 −48%

Per instrument, measured alone, driving the ASGI app in-process so only library cost shows (baseline 15.8 μs/req):

instrument +μs/req share
LoggingInstrument (configured, no logs emitted) +0.1 ~0%
PrometheusInstrument +17.5 8%
SentryInstrument (tracing off) +58.5 27%
OpenTelemetryInstrument +120.1 55%
all four +217.1

Things a page like this should say, all of which surprised me:

  • OpenTelemetry is twice Sentry, and is the dominant cost in the stack. Most people assume Sentry is the expensive one.
  • LoggingInstrument is free until you actually log. The cost is per record (~8 μs with Sentry on), not per request.
  • The Sentry knobs people reach for do nothing. attach_stacktrace=False, max_breadcrumbs=0 and dropping the default integrations all measured within noise. The entire cost is the ASGI integration, and over half of that is a Transaction built and discarded because sentry_traces_sample_rate is unset.
  • sentry_traces_sample_rate=1.0 costs +274 μs/request on top of that (2486 RPS, −68% vs bare). Worth stating next to the setting.
  • Each saving has a price, and the page should be explicit about it: no breadcrumbs on error events, no release health, no Sentry-side trace correlation, sampled-away OTel traces.

Suggested home: a "Performance" or "What observability costs" page under Introduction, linked from each instrument's docs, with a short "if you are RPS-constrained, start here" section.

Caveats to include: these ratios are for a handler that does nothing. A service doing real work per request (a DB round trip, a downstream call) will see a much smaller relative hit - the absolute per-request cost is what transfers, not the percentage.

The benchmark harness that produced these numbers is reproducible and I can attach it if useful. Sibling issues cover the two OpenTelemetry knobs and the Sentry ones.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      AltStyle によって変換されたページ (->オリジナル) /