TPack

NSDI '27 artifact reviewers: start at docs/AE.md, and evaluate the nsdi27-ae tag.

TPack is a generative compression framework for distributed traces. Instead of forwarding every span across the network, an edge collector fits a compact statistical model on the local trace stream, transmits only the model parameters, and a backend collector regenerates synthetic traces that preserve the queries operators actually run, e.g. rate, error rate, latency percentiles, and service dependency structure.

The core idea: traces are a probability graph. TPack learns the graph (start table, edge probabilities), the durations conditioned on graph position (Gaussian mixtures + regression), and the dependent attributes (categorical predictors). At generation time it takes a random walk through that graph, materializes durations and attributes, and emits the result as OTLP — drop-in compatible with any tracing backend.

git clone https://github.com/ProjectASAP/TPack.git
cd TPack/examples/basic
docker compose up --build

After ~70 seconds, browse to:

• http://localhost:3001 — Grafana over the original tracegen workload
• http://localhost:3002 — Grafana over the TPack-regenerated workload

The two should look similar in shape, with the TPack side reconstructed from a model that was ~kilobytes in size.

See examples/basic/README.md for what the logs should say and how to interpret the side-by-side dashboards.

TPack has two distinct audiences. Pick yours:

Drop the TPack exporter into your edge collector and the TPack receiver into your backend collector. The exporter accumulates spans for flush_interval_seconds, trains a model, and broadcasts it over gRPC; the receiver subscribes, regenerates traces, and forwards them to your existing pipeline.

Edge collector (compresses):

exporters:
 tpack:
 flush_interval_seconds: 60
 max_buffered_traces: 50000
 model_server_port: 9090
 primary_attributes:
 - service.name
 - span.kind
 - operation.name
 - status.code
 dependent_attributes:
 - http.status_code
service:
 pipelines:
 traces/compress:
 receivers: [otlp]
 exporters: [tpack]

Backend collector (decompresses):

receivers:
 tpack:
 source_type: grpc
 model_server_endpoint: edge-collector:9090
 continuous_generation: true
exporters:
 otlp_grpc:
 endpoint: tempo:4317 # or your jaeger / tempo / OTLP backend
 tls:
 insecure: true
service:
 pipelines:
 traces/decompress:
 receivers: [tpack]
 exporters: [otlp_grpc]

For details see docs/DEVELOPER.md, docs/CONFIG.md, and examples/basic/.

The paper evaluates TPack on three datasets (OTel Demo, RE2, Uber) across 11 experiment parts: main result, feature/node/graph ablations, RCA, scalability, TVAE baseline, and figure generation.

uv sync --frozen # install Python eval framework from uv.lock
make build # build tpack-eval Go binary
bash scripts/download_data.sh otel-demo # ~60 MB → 1.4 GB unpacked into data/otel-demo/
bash scripts/run_all_experiments.sh otel-demo # run one part
bash scripts/run_all_experiments.sh # run everything
uv run plot_paper --mode query_fidelity --report output/otel-demo/report.json
uv run scorecard --input output/otel-demo/report.json --approaches tpack_default

See docs/AE.md for environment setup (Go 1.25, Python 3.13, Docker), per-dataset prerequisites, expected runtimes, and how each paper claim maps to a command.

TPack/
├─ pkg/tpackmodel/ # core models + algorithms (Go)
├─ exporter/tpackexporter/ # OTel Collector exporter (compressor)
├─ receiver/tpackreceiver/ # OTel Collector receiver (decompressor)
├─ cmd/
│ ├─ otelcol-tpack/ # custom OTel Collector binary
│ └─ tpack-eval/ # standalone evaluation CLI
├─ examples/basic/ # docker compose end-to-end demo
├─ tpack_eval/ # Python evaluation framework (paper figures, scorecard, TVAE baseline)
├─ configs/ # per-dataset YAML configs + ablations
├─ scripts/ # experiment driver, smoke test
└─ docs/ # architecture, reproduction, config reference

Per-OS install commands are in docs/AE.md.

The paper evaluates on three public traces. None ship with the repo; download instructions live in docs/DATASETS.md.

For each 1-minute bucket of traces, TPack trains four sub-models in parallel:

1. Start table — exact counts of each root span signature.
2. Topology — edge probabilities P(child signature | parent signature, child position).
3. Root timing — Gaussian mixture (up to 3 components) over root span duration.
4. Child timing + dependent attributes — OLS regression for gap/duration ratios + per-pair categorical distributions for high-cardinality attributes.

Generation is a level-batched BFS over a random walk through the topology graph, parallelized across CPU cores. Models serialize to gzipped protobuf — tens of kilobytes per one-minute bucket, roughly two orders of magnitude smaller than the raw OTLP they replace. Exact sizes depend on the dataset and the configured attribute set; run scripts/run_all_experiments.sh otel-demo to measure them for yourself.

For more depth see docs/ARCHITECTURE.md.

@inproceedings{tpack2026,
 title = {T-PACK: Structure-Aware Generative Compression for Efficient Distributed Tracing},
 author = {Chin, Yen-Ru and Srivastava, Milind and Zhou, Yajie and Fanti, Giulia and Sekar, Vyas},
 booktitle = {22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI '27)},
 publisher = {USENIX Association},
 year = {2027},
}

See CITATION.cff for machine-readable metadata.

MIT — see LICENSE.

See CONTRIBUTING.md.