How Netflix Scaled its Real-Time Service Map

Follow us on

Netflix has described how it redesigned the streaming pipeline behind Service Topology, its real-time map of service dependencies, to support its production scale. The system now uses three stages to separate intermediary resolution from enrichment and persistence, propagates backpressure to Kafka rather than dropping records, and uses server-sent events instead of gRPC for high-volume internal transfers.

The account follows Netflix’s earlier description of Service Topology’s multi-source graph and focuses on the production engineering required to keep the network-flow pipeline up to date at scale.

Service Topology combines separately stored views from eBPF network flows, inter-process communication (IPC) metrics, and distributed traces. Engineers can query the layers independently or merge them for a broader view of service dependencies. Netflix says teams use it for incident investigation, blast-radius analysis, dependency understanding, and production change management.

The new post focuses on the network-flow ingestion path. Raw flow records show network hops rather than the logical application dependency: traffic can pass through load balancers, NAT gateways, API gateways, or proxies. Netflix therefore processes the data in three stages. The first consumes multi-region Kafka streams, filters invalid records, batches data into five-minute windows, and creates initial aggregators. The second resolves intermediaries into direct application-to-application edges and redistributes those results. The final stage enriches nodes with information such as health, ownership and metadata before persisting them to the graph database.

Netflix separates initial aggregation, intermediary resolution and graph persistence into three stages. (source)

Netflix says its earlier design concentrated work for popular destinations. Because intermediary resolution requires bringing relevant flows together, popular destinations and their intermediaries could make instances hot. The company reports that some instances experienced up to 100 times the typical traffic while also performing I/O-heavy enrichment work. Separating resolution from enrichment and persistence allowed the system to redistribute that work.

The pipeline uses Apache Pekko Streams to manage backpressure. When graph storage cannot keep up, Netflix says pressure travels upstream through the processing stages until the Kafka consumer pauses, leaving records in Kafka until capacity returns. The result is delayed freshness under load rather than dropped data or an incomplete map; Netflix considers that preferable to batch-generated maps that may already be stale during an incident.

When graph writes slow, demand signals travel upstream through the pipeline to the message stream. (source)

Netflix also replaced gRPC between pipeline stages with server-sent events (SSE). It reports that serialisation, connection-pool management and streaming-response memory pressure became expensive at its volume; it describes SSE as lighter-weight and compatible with reactive backpressure. This internal transport is separate from the gRPC API exposed to Service Topology clients.

Netflix’s processing fleet expands and contracts with demand. Each instance reads the same current list of healthy instances from the service registry, then uses consistent hashing to decide which instance owns each aggregator. When an instance joins or leaves, the updated list automatically moves only the affected aggregators to new owners, without a separate rebalancing process.

The IPC pipeline does not need this extra redistribution step. Its metrics already describe application-level calls and are partitioned by application from the start, allowing them to be aggregated in a single stage.

The post also describes historical reconstruction. Rather than retaining complete graph snapshots or replaying an event log, Service Topology keeps time-windowed aggregator snapshots and property-level mutation history. Netflix says it can reconstruct a topology at a specified point in time, enabling engineers to examine changes in dependencies around an incident.

Related Topics:

• Architecture & Design
• Distributed Systems
• Microservices
• eBPF
• gRPC
• Netflix
• Apache Kafka
• Distributed Tracing

We protect your privacy.