DRIVERS-3454 Support trace context propagation to the server - #1966

Assigned vbabanin for team dbx-spec-maintainers-connection-establishment because blink1073 is out of office.

@jyemin

There are no tests. Is that deliberate or do you plan to add a test plan?

TL;DR: No plan to add tests. The wire-level assertions aren't expressible cross-driver: the section is invisible to command monitoring, and server spans are only observable with special mongod startup parameters driver CI doesn't provision.

• How Java (reference implmenetation) was tested locally

Server spans are only observable by controlling the mongod process. It must be started with --setParameter opentelemetryTraceDirectory=<dir> plus --setParameter featureFlagOtelTraceSampling=true and --setParameter 'openTelemetryTracingSampling={defaultSampling: {samplingFactor: 1.0}}'. Spans are then exported as OTLP JSON files written to that directory on the server's filesystem, flushed on the batch-export interval (default 1s).

Verification means running the traced operation, waiting past the flush interval, then reading those files and matching the exported span's traceId/parentSpanId against the client span.

@jyemin

so a malformed section fails those tests on 9.0+, and a missing wire version gate in the driver will presumably cause a hard failure on servers that don't support the new section at all.

The rejection is server-side and safe for the server: pre-9.0 servers error on the unknown section kind and close that connection (the existing OP_MSG behavior for unknown payload types), and 9.0+ servers uassert on a malformed payload. These are ordinary request-scoped errors, the server does not crash, and other connections are unaffected.

A driver that never attaches the section at all.

The section is optional by design, so there's no functional harm in either direction: against pre-9.0 servers nothing is sent, and on 9.0+ OP_MSG simply permits zero telemetry sections.

A section attached to commands that should not include one (auth, monitoring).

That would be a driver violating the existing OTel spec, which already prohibits span creation for security-sensitive commands (and monitoring connections carry no spans). If a driver misbehaves anyway, the server still parses the traceparent and may link a server span to it, subject to its sampling/rate limiting — nothing breaks, but the spans shouldn't exist.

A valid-but-wrong traceparent, e.g. the operation span's context instead of the command span's.

The server can't detect this: any structurally valid (W3C-conformant) traceparent is accepted and used as the parent. The damage shows up in the collector, a server span parented to an id that was never exported (orphaned), or attached to the operation span instead of the command span. That's a driver spec violation invisible to everything except end-to-end span inspection.

Retries reusing one parent-id across attempts instead of one per attempt.

This is precisely why the spec attaches server spans to command spans rather than the operation: the normal case is one operation issuing multiple commands (retry attempts, getMores etc.), each of which should fold under the operation as its own subtree. A driver reusing one parent-id across attempts would hang every server span off a single node flattening the tree and misreport causality, again, only observable from the exported spans.

That's true now, but it doesn't have to remain so. We could create a mongo-orchestration/mongo-runner configuration that enables OTel tracing and invoke it in CI.

Agreed, already thought about it: I have a WIP in drivers-evergreen-tools adding an opt-in --otel flag to drivers_orchestration.py. It launches every mongod/mongos with the file-exporter (while HTTP requires every driver to embed a listener plus an OTLP-protobuf decoder)

Sample usage once it lands:

# provision an OTel-enabled cluster (drivers-evergreen-tools)
OTEL=1 MONGODB_VERSION=latest TOPOLOGY=replica_set bash .evergreen/run-orchestration.sh
source mo-expansion.sh # exports OTEL_TRACE_DIR
# driver test suite gates the prose test on the variable (skipped when unset)
./gradlew :driver-sync:test --tests '<ProseTest>' \
 -Dorg.mongodb.test.uri="$MONGODB_URI" \
 -Dorg.mongodb.test.otel.trace.dir="$OTEL_TRACE_DIR"

I'd track this as separate tickets in both repos (drivers-evergreen-tools feature and one for the spec's prose test that consumes it) so this PR's isn't blocked on tooling.

The section is optional by design, so there's no functional harm in either direction: against pre-9.0 servers nothing is sent, and on 9.0+ OP_MSG simply permits zero telemetry sections

I think this misses the point. The harm is that a driver has not implemented the specification properly. The point of the tests is to catch those violations so that in incomplete or incorrect implementation is not released.

The same reasoning applies to all your replies.

I'd track this as separate tickets in both repos (drivers-evergreen-tools feature and one for the spec's prose test that consumes it) so this PR's isn't blocked on tooling.

If tooling changes are required to properly test implementations, then blocking on tooling is an acceptable outcome.