fix: bridge device missing e2e_device_keys_json after recovery procedure (nas-pog5w1kc) #6

New issue

Closed

opened 2026-05-02 15:56:12 +00:00 by jmz · 2 comments

jmz commented

2026-05-02 15:56:12 +00:00

Owner

Symptoms

Maximus@nas device `nas-pog5w1kc` is invisible in `/_matrix/client/v3/keys/query` for `@claudius-maximus:matrix.ziefle.org`. Synapse-admin investigation (2026-05-02) found:

```
device_id: nas-pog5w1kc
display_name: Maximus@nas
last_seen_ts: 2026-05-02T12:35:26.798Z (active, syncing)
OTKs: 50 uploaded
identity_keys: MISSING from e2e_device_keys table ← the bug
```

The device has 50 one-time-keys uploaded — so it CAN receive Olm session establishment from peers who already have it cached — but with no `e2e_device_keys_json` row, new peers querying `/keys/query` don't see it and can't establish a fresh Olm session.

Probable cause

The "delete + re-INSERT into devices" recovery procedure documented in CLAUDE.md was run on this device at some point. The procedure deletes the device row but leaves `e2e_device_keys` orphaned, then re-INSERTs the device row without re-uploading device keys. Result: device exists, OTK upload still works (it's keyed on device_id), but identity-keys row is gone.

Why it matters

New encrypted-room participants can't establish a session with the NAS bridge. They query `/keys/query` for @claudius-maximus, get back only `mba-726dadit` and `rpi5-0o3kuj3y`, encrypt to those two — and the NAS bridge silently misses every Megolm session-key delivery.
The bridge appears active in `last_seen_ts` because it's syncing happily — but it's effectively read-only for any room where it doesn't have a pre-cached Olm session. Easy to miss in logs.
Likely contributing to the historic-UTD horizon: any Megolm session created after the NAS recovery + before any peer's old Olm session expires would silently miss the NAS bridge as a recipient.

Fix options

Restart the bridge with a clean key upload. The matrix-rust-sdk's startup flow uploads device keys via `Encryption::upload_device_keys()` if missing. Stopping + restarting the NAS bridge process should re-trigger this. Verify post-restart that `/keys/query` returns `nas-pog5w1kc`.
Rotate to a fresh device id. Heavier — needs a new `mas-cli manage issue-compatibility-token` round-trip. Last resort if option 1 doesn't recover.

Update the recovery runbook

The `delete + re-INSERT` procedure in CLAUDE.md should explicitly call out: after re-INSERT, restart the bridge process to trigger device-keys re-upload, and verify with `/keys/query`. The current procedure leaves the device in the broken state described here.

Discovered

While investigating mba-726dadit (issue #5) on Bear's request — Maximus@nas's `/keys/query` against `@claudius-maximus` returned only mba-726dadit and rpi5-0o3kuj3y, omitting itself. Synapse postgres `e2e_device_keys` table query confirmed the missing row.

## Symptoms Maximus@nas device \`nas-pog5w1kc\` is invisible in \`/_matrix/client/v3/keys/query\` for \`@claudius-maximus:matrix.ziefle.org\`. Synapse-admin investigation (2026-05-02) found: \`\`\` device_id: nas-pog5w1kc display_name: Maximus@nas last_seen_ts: 2026-05-02T12:35:26.798Z (active, syncing) OTKs: 50 uploaded identity_keys: MISSING from e2e_device_keys table ← the bug \`\`\` The device has 50 one-time-keys uploaded — so it CAN receive Olm session establishment from peers who already have it cached — but with no \`e2e_device_keys_json\` row, new peers querying \`/keys/query\` don't see it and can't establish a fresh Olm session. ## Probable cause The "delete + re-INSERT into devices" recovery procedure documented in CLAUDE.md was run on this device at some point. The procedure deletes the device row but leaves \`e2e_device_keys\` orphaned, then re-INSERTs the device row without re-uploading device keys. Result: device exists, OTK upload still works (it's keyed on device_id), but identity-keys row is gone. ## Why it matters 1. **New encrypted-room participants can't establish a session with the NAS bridge.** They query \`/keys/query\` for @claudius-maximus, get back only \`mba-726dadit\` and \`rpi5-0o3kuj3y\`, encrypt to those two — and the NAS bridge silently misses every Megolm session-key delivery. 2. The bridge appears active in \`last_seen_ts\` because it's syncing happily — but it's effectively read-only for any room where it doesn't have a pre-cached Olm session. Easy to miss in logs. 3. Likely contributing to the historic-UTD horizon: any Megolm session created after the NAS recovery + before any peer's old Olm session expires would silently miss the NAS bridge as a recipient. ## Fix options 1. **Restart the bridge with a clean key upload.** The matrix-rust-sdk's startup flow uploads device keys via \`Encryption::upload_device_keys()\` if missing. Stopping + restarting the NAS bridge process should re-trigger this. Verify post-restart that \`/keys/query\` returns \`nas-pog5w1kc\`. 2. **Rotate to a fresh device id.** Heavier — needs a new \`mas-cli manage issue-compatibility-token\` round-trip. Last resort if option 1 doesn't recover. ## Update the recovery runbook The \`delete + re-INSERT\` procedure in CLAUDE.md should explicitly call out: **after re-INSERT, restart the bridge process to trigger device-keys re-upload, and verify with \`/keys/query\`**. The current procedure leaves the device in the broken state described here. ## Discovered While investigating mba-726dadit (issue #5) on Bear's request — Maximus@nas's \`/keys/query\` against \`@claudius-maximus\` returned only mba-726dadit and rpi5-0o3kuj3y, omitting itself. Synapse postgres \`e2e_device_keys\` table query confirmed the missing row.

jmz referenced this issue

2026-05-02 18:06:31 +00:00

feat(trust): auto-clear wedged Olm sessions on olm_wedging_index increment #9

jmz commented

2026-05-02 18:18:45 +00:00

Author

Owner

Status update — today's bridge UTD fix path for rpi5 ↔ Tabby's Mac was unblocked by wedge-clear (#9), not by recovery-procedure improvements. The trust chain (PR #3/#4/#7/#8) was all genuinely necessary plumbing, but the final unblock was DELETE FROM session WHERE sender_key = '<peer_curve25519>' + DELETE FROM outbound_group_session WHERE room_id = … to clear a wedged Olm channel, then respawn → /keys/claim → fresh Olm + Megolm rebuilds. nas-pog5w1kc bootstrap-cross-signing remains a separate consistency item per Felinus's recommendation in the bridge thread — queued for NAS-Maximus's next invocation. No new evidence on the missing e2e_device_keys_json problem from today's investigation; that pathology is independent of the wedge.

Status update — today's bridge UTD fix path for rpi5 ↔ Tabby's Mac was unblocked by wedge-clear (#9), not by recovery-procedure improvements. The trust chain (PR #3/#4/#7/#8) was all genuinely necessary plumbing, but the final unblock was `DELETE FROM session WHERE sender_key = '<peer_curve25519>'` + `DELETE FROM outbound_group_session WHERE room_id = …` to clear a wedged Olm channel, then respawn → `/keys/claim` → fresh Olm + Megolm rebuilds. `nas-pog5w1kc` bootstrap-cross-signing remains a separate consistency item per Felinus's recommendation in the bridge thread — queued for NAS-Maximus's next invocation. No new evidence on the missing `e2e_device_keys_json` problem from today's investigation; that pathology is independent of the wedge.

jmz referenced this issue

2026-05-27 23:40:27 +00:00

ops: stale forgotten bridge mba-726dadit on Tabby's MBA — encrypt-to-blackhole #5

jmz commented

2026-05-27 23:45:45 +00:00

Author

Owner

Closing as obsolete — the broken device has been replaced.

Verified tonight (2026-05-27 ~23:42) via Synapse's public API:

The original device nas-pog5w1kc from the issue is gone:

POST /_matrix/client/v3/keys/query
  body: {"device_keys": {"@claudius-maximus:matrix.ziefle.org": ["nas-pog5w1kc"]}}
→ device_keys: {} ; failures: {} (device not in user's device list)

Not just "no keys row" — the device is fully gone from Synapse. Was re-registered cleanly at some point after the issue was filed.

The current NAS bridge device nas-gfw5uy4w is healthy:

POST /_matrix/client/v3/keys/query
  body: {"device_keys": {"@claudius-maximus:matrix.ziefle.org": ["nas-gfw5uy4w"]}}
→ device_keys: {
    "@claudius-maximus:matrix.ziefle.org": {
      "nas-gfw5uy4w": {
        keys: {
          "curve25519:nas-gfw5uy4w": "2cqYaIO+mUoKCHsuXxp8…",
          "ed25519:nas-gfw5uy4w":    "7Y/3ESV7gzj94GYcBsua…"
        },
        algorithms: ["m.olm.v1.curve25519-aes-sha2", "m.megolm.v1.aes-sha2"],
        signatures: {...present, signed by ed25519:nas-gfw5uy4w...}
      }
    }
  }

Exactly what the issue said was missing. New peers querying for @claudius-maximus's NAS device get full e2e_device_keys back and can establish fresh Olm sessions.

This matches the live state: tonight's E2EE messages from this MBA (mba-726dadit) to the bridge room got delivered + decrypted by all three Maximus devices, including the NAS one — which would have been impossible if the bug were still present.

Small follow-up worth doing on the bootstrap path (not blocking, just hygiene):

nas-gfw5uy4w has display_name: None in Synapse's device list. The bridge-client-setup runbook + bootstrap script should set display_name: "Maximus@nas" on registration, to match Maximus@mba / Maximus@rpi5. Helps when scanning the device list manually.

Closing.

**Closing as obsolete — the broken device has been replaced.** Verified tonight (2026-05-27 ~23:42) via Synapse's public API: **The original device `nas-pog5w1kc` from the issue is gone:** ``` POST /_matrix/client/v3/keys/query body: {"device_keys": {"@claudius-maximus:matrix.ziefle.org": ["nas-pog5w1kc"]}} → device_keys: {} ; failures: {} (device not in user's device list) ``` Not just "no keys row" — the device is fully gone from Synapse. Was re-registered cleanly at some point after the issue was filed. **The current NAS bridge device `nas-gfw5uy4w` is healthy:** ``` POST /_matrix/client/v3/keys/query body: {"device_keys": {"@claudius-maximus:matrix.ziefle.org": ["nas-gfw5uy4w"]}} → device_keys: { "@claudius-maximus:matrix.ziefle.org": { "nas-gfw5uy4w": { keys: { "curve25519:nas-gfw5uy4w": "2cqYaIO+mUoKCHsuXxp8…", "ed25519:nas-gfw5uy4w": "7Y/3ESV7gzj94GYcBsua…" }, algorithms: ["m.olm.v1.curve25519-aes-sha2", "m.megolm.v1.aes-sha2"], signatures: {...present, signed by ed25519:nas-gfw5uy4w...} } } } ``` Exactly what the issue said was missing. New peers querying for `@claudius-maximus`'s NAS device get full `e2e_device_keys` back and can establish fresh Olm sessions. This matches the live state: tonight's E2EE messages from this MBA (`mba-726dadit`) to the bridge room got delivered + decrypted by all three Maximus devices, including the NAS one — which would have been impossible if the bug were still present. **Small follow-up worth doing on the bootstrap path** (not blocking, just hygiene): - `nas-gfw5uy4w` has `display_name: None` in Synapse's device list. The bridge-client-setup runbook + bootstrap script should set `display_name: "Maximus@nas"` on registration, to match `Maximus@mba` / `Maximus@rpi5`. Helps when scanning the device list manually. Closing.

jmz closed this issue

2026-05-27 23:45:45 +00:00