-
Notifications
You must be signed in to change notification settings - Fork 188
Cluster peer never discovered: browser loses peer's mDNS replies (macOS Internet Sharing drops inbound IPv4 multicast; legacy-unicast replies hit a closed socket) #44
Description
Bug: cluster peer never discovered — the browser loses the peer's mDNS replies (macOS Internet Sharing drops inbound IPv4 multicast; legacy-unicast replies land on a closed socket)
Correction (2026年09月07日). This issue originally blamed the
cluster-uuid=TXT value. After a packet capture, an A/B test and socket-level probes on the affected pair, that hypothesis is wrong:cluster-uuid=is correct. The symptom is real, the root cause is in mDNS ingestion on the browsing host. The original report is kept at the bottom for history.
Environment
- PAIR 0.91.7 (same on both nodes)
- Node A: macOS (Apple Silicon), PAIR from .dmg. Internet Sharing enabled,
net.inet.ip.forwarding=1; shares its connection to Node B over a direct Ethernet link (en0192.168.2.1/24) - Node B (
home-wire): Debian 12, kernel 6.1, PAIR Linux desktop app;enp7s0192.168.2.2/24 via DHCP from Node A;avahi-daemonrunning - Other cluster members are on Node A's second interface (
en1, 10.0.0.x)
Symptom
Node B is a paired cluster member. Its hardware card shows, dns-sd -Z _nvpair-node._tcp local. on Node A lists its record, yet Node A's PAIR never lists it in discovery:nodes-changed, the node stays trusted:false, engine-manager logs node <uuid> is not a discovered ec peer every ~10 s, and the Ollama/LM Studio panels for Node B stay on "Initializing..." forever. Node B discovers Node A fine (reverse direction works).
What cluster-uuid= actually is
It is the node's cluster principal: the UUID in its cluster certificate (node.crt SAN urn:nvpair:node:<uuid>), the name peers pin under cluster/trusted/<uuid>.json (services/shared/noderec/noderec.go, services/nvpair-node-scanner/daemon.go, services/shared/clustertrust/mesh.go). It is not clusterId. uuid= is resolved from the same identity, so uuid == cluster-uuid is the normal state of every clustered node. Publishing clusterId there would never match a pin and would break trust/mTLS for every peer.
Why the avahi workaround appeared to work
Not because of the TXT value (HasPin(<clusterId>) is false everywhere). It worked because the record was finally received: avahi publishes over IPv4 and IPv6, PAIR's responder over IPv4 only. The moment the avahi record arrived the peer flipped to clustered:true; trusted:true followed ~9 s later from the periodic HTTP node-info sweep, which reads the real principal from the peer's ni port.
Root cause (verified)
Capture on Node A en0 with the workaround off, plus probes:
- Peer's multicast reply reaches the wire but no socket. Node B's PAIR responder answers Node A's re-sent PTR query by multicast:
PTR + Extra{SRV, TXT "cluster-uuid=<principal>", A 192.168.2.2}, TTL 255, complete and correct. A probe socket on Node A bound to0.0.0.0:5354, joined to224.0.0.251onen0, receives multicast sent by Node A itself but nothing from the peer, while unicast from the peer to the same socket arrives. IPv4 multicast onen1and IPv6 multicast onen0both work. With Internet Sharing on, the macOS kernel does not deliver inbound IPv4 multicast on the shared interface to local UDP sockets. - Legacy-unicast replies land on a closed socket.
shared/discoveryre-sends the PTR query from a throwaway ephemeral-port socket (the Windows send workaround) and closes it at once. Avahi/Bonjour answer a query from a non-5353 port by unicast to that port (RFC 6762 §6.7); those answers were lost too.
A/B test: workaround off, PAIR on Node A restarted; for 2 min 17 s with only PAIR's record on the wire Node B never gained clustered. 4 s after avahi returned: clustered:true, then trusted:true.
Proposed fix (tested)
In services/shared/discovery/discovery.go:
- set the unicast-response (QU) bit on the re-sent PTR query (RFC 6762 §5.4);
- keep the per-interface query socket open for the scan window and parse its unicast replies with the same semantics as the zeroconf browse, so a node keys identically either way.
PAIR's own responder already honours QU, so a peer answers straight back to the querying socket, the one delivery path that does not depend on multicast reception. Verified live: a patched nvpair-node-scanner on Node A discovers Node B with its correct cluster-uuid= within one scan, with avahi stopped; the stock 0.20.3 scanner as negative control does not. Tests added (QU bit, PTR/SRV/TXT/A assembly, foreign/addressless records ignored, unicast replies collected), green with -race. Patch attached in the comment below.
Suggested version bumps: nvpair-node-scanner, nvpair-cluster-manager (both browse). cluster-uuid= semantics unchanged; worth documenting in the scanner README.
Original report (superseded)
(superseded) Bug: Node re-broadcasts its own UUID as cluster-uuid in mDNS TXT record — peer never becomes a "discovered ec peer", remote engine proxying (Ollama/LM Studio) stuck on "Initializing..." forever
Environment
- PAIR version: v0.40.2 (same version on both nodes)
- Node A: macOS (Apple Silicon), PAIR installed via .dmg
- Node B: Debian 12 (kernel 6.1.180-1), PAIR installed as a Linux desktop app, GNOME session
- Both nodes on the same LAN segment, directly reachable (ping/mDNS both succeed)
- Cluster has 4 members total; the issue only affects one specific node (Node B)
Summary
A cluster member (Node B) shows up correctly in the UI (hardware card, GPU info, cluster membership), but its Ollama/LM Studio engines are permanently stuck on "Initializing...". The backend logs show a continuous stream of:
node <uuid> is not a discovered ec peer
repeating every ~10s indefinitely, on both sides of the pair (each node reports the other's UUID as "not a discovered ec peer").
Root cause (as far as I could determine)
Node B's _nvpair-node._tcp mDNS TXT record advertises a cluster-uuid field that is identical to its own node uuid, instead of the actual cluster ID:
$ dns-sd -Z _nvpair-node._tcp local.
home-wire._nvpair-node._tcp TXT "v=1" "uuid=<node-uuid>" "cluster-uuid=<node-uuid>" "ip=192.168.2.2" "ni=14318" "lm=1234" "er=14319" "wl=14320" "cl=14321" "em=14322" "ec=14323"
Note cluster-uuid == uuid. The real cluster ID (confirmed present and correct in cluster/members.json, cluster/admission.json, and configs/ui-config.json on both nodes, including Node B itself) is a different UUID entirely, and it is correctly readable from cluster/admission.json's clusterId field — so this isn't a case of missing/malformed data on disk, the scanner just isn't using it.
Because of this mismatch, the peer that receives the mDNS announcement can never match the advertised cluster-uuid against its locally pinned trust record for that node, so:
- the node's entry in the
discovery:nodes-changedbroadcast stays"trusted": false(all correctly-clustered peers show"trusted": true, "clustered": true) - the engine-manager never registers it as a usable peer for the encrypted engine-proxy channel, hence the persistent
is not a discovered ec peerRPC error - Ollama/LM Studio panels for that node never leave "Initializing..." in the UI
What I tried (none fixed it)
- Restarting the whole app on both nodes — TXT record is regenerated identically every time, still self-referential.
- Killing and letting the supervisor respawn just the discovery/scanner service on the affected node — note: this actually took down the entire backend service tree on that node (all cli-bin child processes exited), it wasn't a clean single-service restart. Had to fully relaunch the app.
- Removing the node from the cluster and re-pairing it (fresh PIN pairing, new admission epoch) — the on-disk trust/pin files (
cluster/trusted/<uuid>.json) update correctly and match the live cert fingerprint on both sides, but the mDNS TXT record keeps advertising the wrongcluster-uuid, so the underlying issue persists across re-pairing. - Manually adding the node via
manual-nodes.json(to route around the underlying reachability layer) fixes basic HTTP reachability/hardware display, but does not affect this specificcluster-uuidTXT bug, since the engine-proxy "ec peer" channel is unrelated to manual-node reachability. - Checked
nvpair-node-scanner --helpfor a CLI override (e.g. an explicit--cluster-uuidflag) — none exists; it only takes--cluster-dir,--ipc,--log-level, TLS-related flags, and--version. So there's no supported way to override this from the outside.
Expected behavior
The cluster-uuid field in the mDNS TXT record should reflect the actual cluster ID the node joined (as already stored correctly in its own cluster/admission.json), not its own node UUID. Once that's correct, the peer should be recognized as trusted/clustered and the engine-manager should be able to open the encrypted peer channel for Ollama/LM Studio proxying.
Additional notes
- This looks like it could be in the discovery/TXT-record-publishing path of
nvpair-node-scanner(possibly related to identity/cluster info not being re-read correctly aftercluster-managercompletes admission — i.e. the scanner may be started with, or falls back to, the node's own UUID instead of pulling the cluster UUID once membership is established). - Happy to provide full
nvpair.jsonllog excerpts (with hostnames/UUIDs redacted) or run further diagnostics if useful.