-
Notifications
You must be signed in to change notification settings - Fork 344
feat(sdk): support multiple OTLP span exporters (dual-write alongside Langfuse) #1716
Description
Summary
We would like Langfuse SDKs to support exporting spans to more than one destination—specifically, keep the default Langfuse OTLP export and optionally forward the same spans to an additional OTLP endpoint (or custom SpanExporter).
This would follow the standard OpenTelemetry pattern of attaching multiple SpanProcessors to a single TracerProvider, without requiring users to replace Langfuse's built-in exporter.
Problem
Today, when instrumenting with the Langfuse Python SDK (@observe(), LangChain callbacks, etc.), spans are exported through Langfuse's internal OTLP pipeline to the configured Langfuse host.
Some deployments need parallel export to another OTLP-compatible backend—for example:
- An internal observability or logging pipeline (same VPC / on-prem)
- A secondary sink during migration or validation
- Environment-specific routing (e.g. dev/staging copies to a local collector)
PR #1618 added optional span_exporter, which is helpful for customization, but it replaces the default Langfuse exporter rather than adding a second one. Setting generic OTEL_EXPORTER_OTLP_* env vars also does not appear to dual-write when using Langfuse's own TracerProvider initialization.
Proposed behavior
Support one or more additional exporters while preserving the default Langfuse export path.
Option A — constructor / client API
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
Langfuse(
additional_span_exporters=[
OTLPSpanExporter(endpoint="https://internal-collector.example.com/v1/traces"),
],
)
Option B — environment variables (no code changes)
# Existing Langfuse config unchanged
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...
LANGFUSE_HOST=...
# Additional OTLP sink
LANGFUSE_ADDITIONAL_OTLP_ENDPOINT=https://internal-collector.example.com
LANGFUSE_ADDITIONAL_OTLP_HEADERS=Authorization=Bearer ...
Requirements we care about:
- Default behavior unchanged when additional exporters are not configured
- Additional exporters receive the same spans as Langfuse (no silent filtering unless explicitly configured later)
- get_client() / singleton initialization should preserve additional exporters
- Clear documentation on auth headers, timeouts, and failure isolation (one exporter failing should not break the other)
Alternatives considered
- Replace exporter via span_exporter (feat(client): add custom span exporter support #1618 ) — loses default Langfuse export unless manually re-wired
- User-owned OTEL Collector sidecar — works but adds operational overhead; SDK-level dual export is simpler for many teams
- Application-level double instrumentation — fragile across frameworks and easy to break on upgrades
Questions for maintainers
- Is dual / multi-exporter export something the Langfuse SDK team would consider?
- Would you prefer an explicit additional_span_exporters API, env-based configuration, or both?
- Should this live in langfuse-python first, with parity in langfuse-js?
- Any concerns about performance, backpressure, or support burden we should address in a PR?
Happy to contribute a PR if there is alignment on the API shape. Thanks!