Skip to content

Navigation Menu

Sign in
Sign up

[Cisco] Fix fboss2 interface traffic hangs and missing rates - #1626

Open
selekkala wants to merge 3 commits into
facebook:main from
selekkala:fix/fboss2-interface-traffic-rates
Open

[Cisco] Fix fboss2 interface traffic hangs and missing rates #1626
selekkala wants to merge 3 commits into
facebook:main from
selekkala:fix/fboss2-interface-traffic-rates

Conversation

@selekkala

@selekkala selekkala commented Sep 11, 2026

Copy link
×ばつ 8 counters) were exported; - ran 70 no-traffic/below-threshold invocations and 30 active-traffic invocations: 100/100 returned successfully, with zero hangs or timeouts; - observed command latency between 32 ms and 64 ms; - generated a controlled traffic burst on `eth1/1/1` and observed: ```text hwagent0 eth1/1/1.out_bytes.rate.60 52195 Interface Name Peer Intvl InMbps InPct InKpps OutMbps OutPct OutKpps eth1/1/1 -- 0:60 0.00 0.00% 0.00 0.42 0.00% 0.04 Total -- -- 0.00 0.00% 0.00 0.42 0.00% 0.04 ``` - Confirmed `fboss_sw_agent`, `fboss_hw_agent@0`, and `qsfp_service` remained active after validation. " data-view-component="true"> Copy Markdown

Summary

Fix fboss2 show interface traffic on OSS multi-switch systems where the command can hang indefinitely or return an empty/zero table while ports are carrying traffic.

This change:

  • uses deterministic synchronous RPCs for the command's sequential data collection;
  • exports the SUM and RATE types requested by MonotonicCounter callers, making port .rate.60 counters available from the HW agent;
  • fetches only the traffic-rate counter families needed by this command;
  • skips down ports, which may legitimately have no HW-agent rate counters;
  • reports a missing counter on an up port instead of silently converting it to zero; and
  • preserves fractional Kpps values and guards per-port zero bandwidth.

The aggregate zero-bandwidth fix from #1560 is already present on main and remains unchanged.

Root cause

There were three contributing problems:

  1. CmdShowInterfaceTraffic::queryClient() mixed future-based RPCs attached to a short-lived local IOThreadPoolExecutor with blocking HW-agent fanout, followed by an unconditional wait(). If the future did not reach a terminal state on a cold/multi-switch path, the command had no forward progress and remained blocked. The operation is sequential at the point the model is created, so the local executor did not provide useful concurrency.
  2. The OSS MonotonicCounter implementation created its backing stat with getLockableStatNoExport() and ignored both requested export types. Port counters were updated internally, but .rate.60 was therefore absent from FB303 output.
  3. The CLI read rate counters with std::map::operator[]. Missing counters were inserted as zero, making unavailable telemetry indistinguishable from a real zero traffic rate. It also attempted to read current rate counters for down ports.

Fix details and compatibility

  • The Thrift APIs and output schema are unchanged.
  • Both monolithic and multi-switch paths remain supported.
  • MonotonicCounter now honors its existing constructor contract. Its swap() implementation also transfers the state, name, and lockable stat so clear/rename operations retain the correct counter lifecycle.
  • The FB303 regex is anchored to exact .rate.60 suffixes, reducing the counter payload compared with fetching every agent counter.
  • Down ports are excluded before rate lookup. An up port without required telemetry fails explicitly instead of displaying misleading zeros.
  • Existing error-counter reporting is preserved for both up and down ports.

Unit-test coverage

  • synchronous query path and exact .rate.60 regex;
  • missing rate counter on an up port;
  • down port without rate counters;
  • fractional Kpps conversion;
  • zero-bandwidth utilization;
  • monotonic counter rate export, removal, and port-name reinitialization; and
  • existing empty-traffic aggregate output coverage from [Nexthop] Fix nan% issue in "show interface traffic" CLI output #1560 .

Test Plan

  • pre-commit run --from-ref upstream/main --to-ref HEAD — passed, including clang-format and repository checks.
  • Built a full C8501 FBOSS image carrying the equivalent backport: fboss-gr2-stable-20260903-fbc276b.1-1-ge405635.
  • Validated on a Cisco C8501 / Morgan800CC lab switch:
    • enabled all 128 physical ports; two connected ports came up;
    • verified all 1,024 expected physical-port traffic .rate.60 counters (128 ports ×ばつ 8 counters) were exported;
    • ran 70 no-traffic/below-threshold invocations and 30 active-traffic invocations: 100/100 returned successfully, with zero hangs or timeouts;
    • observed command latency between 32 ms and 64 ms;
    • generated a controlled traffic burst on eth1/1/1 and observed:
hwagent0 eth1/1/1.out_bytes.rate.60 52195
Interface Name Peer Intvl InMbps InPct InKpps OutMbps OutPct OutKpps
eth1/1/1 -- 0:60 0.00 0.00% 0.00 0.42 0.00% 0.04
Total -- -- 0.00 0.00% 0.00 0.42 0.00% 0.04
  • Confirmed fboss_sw_agent, fboss_hw_agent@0, and qsfp_service remained active after validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /