fleet: make the hub mirror publish non-blocking (bounded hung-hub tail latency)

jmz commented

2026-09-11 00:01:26 +00:00

Owner

From the fleet whole-branch review (v0.22.0). hub_mirror publishes to the local bus first (unguarded) then best-effort to the hub; NatsConn.publish blocks on .result(timeout=5). If the hub connection is HUNG (not cleanly failed), the hub publish can add up to a bounded ~5s to the TAIL of the transition/notify call before being swallowed. Local publish + local incident persistence run first and are byte-for-byte unaffected (invariant holds), and nats_pub is the last notifier in the composite — so this is bounded + local-safe, not a correctness bug. Follow-up: make the hub mirror fire-and-forget / async (or a shorter hub-publish timeout) so a hung hub adds ~0 to the local notify tail. Non-blocking; watch only if hub-network hangs appear in prod.

From the fleet whole-branch review (v0.22.0). `hub_mirror` publishes to the local bus first (unguarded) then best-effort to the hub; `NatsConn.publish` blocks on `.result(timeout=5)`. If the hub connection is HUNG (not cleanly failed), the hub publish can add up to a bounded ~5s to the TAIL of the transition/notify call before being swallowed. Local publish + local incident persistence run first and are byte-for-byte unaffected (invariant holds), and nats_pub is the last notifier in the composite — so this is bounded + local-safe, not a correctness bug. Follow-up: make the hub mirror fire-and-forget / async (or a shorter hub-publish timeout) so a hung hub adds ~0 to the local notify tail. Non-blocking; watch only if hub-network hangs appear in prod.

jmz commented

2026-09-11 14:39:13 +00:00

Author

Owner

Fixed in v0.24.1. The fleet hub mirror hop is now fire-and-forget: NatsConn.publish_nowait submits the hub publish to the loop via run_coroutine_threadsafe WITHOUT .result(), so a hung/slow hub can no longer add the ~5s tail to local incident processing; failures log via a done-callback, never raised. The LOCAL publish stays the blocking, ordered path (byte-for-byte unchanged). Whole-branch review: APPROVE.

jmz closed this issue

2026-09-11 14:39:13 +00:00

jmz referenced this issue from a commit

2026-09-11 14:39:14 +00:00

plan: fleet hub mirror non-blocking (#11)

jmz referenced this issue from a commit

2026-09-11 14:39:14 +00:00

feat(mesh): fire-and-forget fleet hub mirror hop (OSS #11 T1)

jmz referenced this issue from a commit

2026-09-11 14:39:14 +00:00

chore(mesh): bump to 0.24.1 + document fire-and-forget hub mirror (OSS #11 T2)