jmz/matrix-bridge

Fork 0

feat(trust): auto-clear wedged Olm sessions on olm_wedging_index increment #9

New issue

Open

opened 2026-05-02 18:06:31 +00:00 by jmz · 1 comment

jmz commented

2026-05-02 18:06:31 +00:00

Owner

Context

Today's UTD asymmetry between rpi5 (@claudius-maximus) and Tabby's Mac (@claudius-felinus) was eventually root-caused to a wedged Olm session sitting under what looked like a complete server-side trust chain.

After PR #3 (sync stability + auto-key-forward), PR #4 (bootstrap-cross-signing CLI), PR #7 (user-sign other users' master keys), and PR #8 (pin() before verify() so pinned_master_key tracks current state), all server-side trust state was correct:

master keys cross-signed in both directions on Synapse (verified via e2e_cross_signing_signatures admin dump)
local pinned_master_key.signatures post-PR-#8 contained the user-signing key
outbound Megolm session's shared_with_set correctly listed the peer device with Shared status

But inbound_group_session count on the recipient never incremented — to-device m.room_key shares were silent-dropped.

The smoking gun: peer device record had olm_wedging_index: 1 while the outbound Megolm session's shared_with_set recorded olm_wedging_index: 0. The Olm channel had been flagged as wedged after the recipient set was selected. matrix-sdk-crypto bumped the wedging index but left the wedged session row in place, so subsequent encrypts continued routing through it.

Manual unblock (with explicit authorization):

DELETE FROM session WHERE sender_key = '<peer_curve25519>';
DELETE FROM outbound_group_session WHERE room_id = '<room>';

After respawn → /keys/claim for fresh OTK → fresh Olm session → fresh Megolm with rebuilt shared_with_set → m.room_key delivered → recipient inbound_group_session count went 22 → 25 (3 messages caught up).

Proposed change

Add a small apply_trust_policy-side (or session-manager-side) check: when iterating devices, if device.olm_wedging_index > current_olm_session.expected_wedging_index, drop the existing session row(s) for that device's sender_key so the next encrypt forces a fresh /keys/claim + Olm establishment.

Pseudo-Rust in trust.rs:

for device in devices_for_user {
    let stored_idx = device.olm_wedging_index;
    let session_idx = oldest_session_for(&device.curve25519).map(|s| s.expected_wedging_index).unwrap_or(0);
    if stored_idx > session_idx {
        crypto_store.delete_sessions(&device.curve25519).await?;
        warn!(
            user = %device.user_id,
            device = %device.device_id,
            "auto-cleared wedged Olm session(s) for device — fresh Olm will be claimed on next encrypt"
        );
    }
    identity.pin().await?;
    identity.verify().await?;
}

Why this matters

Zero manual intervention next time it happens. Today this took ~2 hours of cross-bot debugging across rpi5 + Tabby's Mac + NAS.
Wedge events are rare but recoverable; recovery should be automatic.
The same path runs in apply_trust_policy which is called from bootstrap + periodic-trust-refresh, so this gets invoked on every bridge startup (acceptable cadence).

Open questions

Should this also clear inbound_group_session rows for that sender_key, or only the Olm session rows? Probably only Olm — Megolm history is precious.
Whether to delete the existing outbound_group_session proactively (forces re-share to all recipients) or wait for natural rotation (next message after rotation_period or rotation_period_msgs).
Whether matrix-rust-sdk upstream has a hook for this we should be using instead of doing it ourselves. Worth checking before implementing.

Verification recipe

Identify a wedged session by device.olm_wedging_index > 0 against a known sender_key with no recent fresh creation_time.
Restart bridge with this patch.
On startup apply_trust_policy, observe the warn-level log + zero rows in session for that sender_key.
Send an encrypted room event.
Confirm /keys/claim HTTP call (via mitmproxy or bridge debug log), confirm new Olm session row, confirm m.room_key delivered to recipient (inbound_group_session count delta).

Wedge debugging thread: !qmEZYWbxkxTUFqNGQC:matrix.ziefle.org (2026-05-02 ~17:20Z–17:59Z)
Manual fix sequence (logged in PR #8 deploy comments)
Related issues: #5 (orphan device cleanup), #6 (recovery-procedure missing keys)

## Context Today's UTD asymmetry between rpi5 (`@claudius-maximus`) and Tabby's Mac (`@claudius-felinus`) was eventually root-caused to a wedged Olm session sitting under what looked like a complete server-side trust chain. After PR #3 (sync stability + auto-key-forward), PR #4 (bootstrap-cross-signing CLI), PR #7 (user-sign other users' master keys), and PR #8 (`pin()` before `verify()` so `pinned_master_key` tracks current state), all server-side trust state was correct: - master keys cross-signed in both directions on Synapse (verified via `e2e_cross_signing_signatures` admin dump) - local `pinned_master_key.signatures` post-PR-#8 contained the user-signing key - outbound Megolm session's `shared_with_set` correctly listed the peer device with `Shared` status But `inbound_group_session` count on the recipient never incremented — `to-device` `m.room_key` shares were silent-dropped. The smoking gun: peer device record had `olm_wedging_index: 1` while the outbound Megolm session's `shared_with_set` recorded `olm_wedging_index: 0`. The Olm channel had been flagged as wedged after the recipient set was selected. matrix-sdk-crypto bumped the wedging index but **left the wedged session row in place**, so subsequent encrypts continued routing through it. Manual unblock (with explicit authorization): ```sql DELETE FROM session WHERE sender_key = '<peer_curve25519>'; DELETE FROM outbound_group_session WHERE room_id = '<room>'; ``` After respawn → `/keys/claim` for fresh OTK → fresh Olm session → fresh Megolm with rebuilt `shared_with_set` → `m.room_key` delivered → recipient `inbound_group_session` count went `22 → 25` (3 messages caught up). ## Proposed change Add a small `apply_trust_policy`-side (or session-manager-side) check: when iterating devices, if `device.olm_wedging_index > current_olm_session.expected_wedging_index`, drop the existing `session` row(s) for that device's `sender_key` so the next encrypt forces a fresh `/keys/claim` + Olm establishment. Pseudo-Rust in `trust.rs`: ```rust for device in devices_for_user { let stored_idx = device.olm_wedging_index; let session_idx = oldest_session_for(&device.curve25519).map(|s| s.expected_wedging_index).unwrap_or(0); if stored_idx > session_idx { crypto_store.delete_sessions(&device.curve25519).await?; warn!( user = %device.user_id, device = %device.device_id, "auto-cleared wedged Olm session(s) for device — fresh Olm will be claimed on next encrypt" ); } identity.pin().await?; identity.verify().await?; } ``` ## Why this matters - Zero manual intervention next time it happens. Today this took ~2 hours of cross-bot debugging across rpi5 + Tabby's Mac + NAS. - Wedge events are rare but recoverable; recovery should be automatic. - The same path runs in `apply_trust_policy` which is called from bootstrap + periodic-trust-refresh, so this gets invoked on every bridge startup (acceptable cadence). ## Open questions - Should this also clear `inbound_group_session` rows for that sender_key, or only the Olm `session` rows? Probably only Olm — Megolm history is precious. - Whether to delete the existing `outbound_group_session` proactively (forces re-share to all recipients) or wait for natural rotation (next message after `rotation_period` or `rotation_period_msgs`). - Whether matrix-rust-sdk upstream has a hook for this we should be using instead of doing it ourselves. Worth checking before implementing. ## Verification recipe 1. Identify a wedged session by `device.olm_wedging_index > 0` against a known sender_key with no recent fresh `creation_time`. 2. Restart bridge with this patch. 3. On startup `apply_trust_policy`, observe the warn-level log + zero rows in `session` for that sender_key. 4. Send an encrypted room event. 5. Confirm `/keys/claim` HTTP call (via mitmproxy or bridge debug log), confirm new Olm session row, confirm `m.room_key` delivered to recipient (`inbound_group_session` count delta). ## Related - Wedge debugging thread: `!qmEZYWbxkxTUFqNGQC:matrix.ziefle.org` (2026-05-02 ~17:20Z–17:59Z) - Manual fix sequence (logged in PR #8 deploy comments) - Related issues: #5 (orphan device cleanup), #6 (recovery-procedure missing keys)

jmz referenced this issue

2026-05-02 18:18:35 +00:00

ops: stale forgotten bridge mba-726dadit on Tabby's MBA — encrypt-to-blackhole #5

jmz referenced this issue

2026-05-02 18:18:45 +00:00

fix: bridge device missing e2e_device_keys_json after recovery procedure (nas-pog5w1kc) #6

jmz commented

2026-05-28 00:28:08 +00:00

Author

Owner

Investigation update (2026-05-28 ~02:30 CET)

Started on this tonight, expecting to implement the apply_trust_policy-side check from the issue body. Found that the situation is more nuanced than the issue described, so writing it up here for whoever picks this up next:

What matrix-sdk-crypto already does (0.14.0, unchanged through 0.17.0)

The full wedge-recovery loop is implemented inside the library:

Wedged Olm session detected on decrypt → OlmError::SessionWedged propagated up.
machine.rs:1543 catches it → calls session_manager.mark_device_as_wedged(sender, curve_key).
That adds the device to users_for_key_claim → next outgoing-request cycle does /keys/claim → fresh Olm session.
check_if_unwedged() sends an m.dummy to-device event via the fresh session.
On the peer side, the dummy decrypts cleanly, creates a fresh session locally, and increments device.olm_wedging_index (olm/account.rs:1393).
Subsequent outbound Megolm encrypts walk ShareState::Shared { message_index, olm_wedging_index }, see olm_wedging_index < d.olm_wedging_index, and re-share the room_key (session_manager/group_sessions/mod.rs:751-758).

So in steady state the symptom this issue describes — wedge → no recovery — shouldn't happen. The library handles it.

The actual gap that bit us on 2026-05-02

session_manager/sessions.rs:80,118:

const UNWEDGING_INTERVAL: Duration = Duration::from_secs(60 * 60);
...
let should_unwedge = now.checked_sub(creation_time)
    .map(|elapsed| elapsed > Self::UNWEDGING_INTERVAL)
    .unwrap_or(true);

If the wedged session was created less than 1 hour ago, mark_device_as_wedged is a no-op — recovery is BLOCKED for up to an hour. Almost certainly what bit us: rapid trust-policy churn during the 2026-05-02 debugging session put us inside that 1-hour window, and the library refused to unwedge until the interval elapsed.

This is a deliberate throttle (avoids hammering /keys/claim for transient network blips) — sensible for normal Element clients, painful for bridges where wedges are rare + observability is poor.

Why the issue body's pseudo-Rust can't be implemented as-is

The proposed apply_trust_policy-side fix assumes device.olm_wedging_index and a way to delete sessions for a sender_key are accessible. Both are out of reach from outside the crate:

DeviceData.olm_wedging_index is pub(crate) (identities/device.rs:103) — not readable from apply_trust_policy's perspective.
SessionManager::mark_device_as_wedged is pub but SessionManager itself isn't reached via Client::encryption() — effectively crate-private to downstream users.
There's no delete_sessions(&curve25519) on any public API.

Bumping doesn't help

Checked the source of matrix-sdk-crypto-0.17.0 against our pinned =0.14.0:

	0.14.0	0.17.0
`UNWEDGING_INTERVAL`	1 hour	1 hour (unchanged)
`mark_device_as_wedged` visibility	crate-private	crate-private (unchanged)
`olm_wedging_index` visibility	`pub(crate)`	`pub(crate)` (unchanged)
Public force-unwedge API	none	none

Changelog entries for 0.15 / 0.16 / 0.16.1 / 0.17 don't mention wedge handling.

Where this leaves us

The structural fix lives upstream — matrix-org/matrix-rust-sdk needs to either expose force_unwedge_device() (public method on Client::encryption() or OlmMachine) OR make UNWEDGING_INTERVAL configurable per-client.

Until that lands, operators have two paths when wedge-within-1-hour hits:

Wait it out. Once the interval elapses, the library's recovery loop runs on the next sync cycle. No action needed.
Manual SQL DELETE (the recipe in the original issue body): direct DELETE on the session row for the wedged sender_key. Risky — bypasses matrix-sdk's in-memory caches; subsequent encrypts may produce inconsistent state until the next process restart. Use only when (1) isn't acceptable + operator can immediately restart the bridge afterward.

Leaving this issue open

Not closing — the upstream PR is still wanted long-term. When someone has bandwidth to draft the upstream patch:

File issue at matrix-org/matrix-rust-sdk asking for either approach (public force-unwedge OR configurable interval). Reference this issue for the bridge-side motivation.
After upstream merges + ships, bump our pin and implement the auto-clear in apply_trust_policy (per the issue body's original design, but using the new public API).

Re-titling this issue would be reasonable — something like feat(trust): auto-clear wedged Olm sessions inside the matrix-sdk UNWEDGING_INTERVAL window — makes the actual ask clearer.

## Investigation update (2026-05-28 ~02:30 CET) Started on this tonight, expecting to implement the `apply_trust_policy`-side check from the issue body. Found that the situation is more nuanced than the issue described, so writing it up here for whoever picks this up next: ### What matrix-sdk-crypto already does (0.14.0, unchanged through 0.17.0) The full wedge-recovery loop is implemented inside the library: 1. Wedged Olm session detected on decrypt → `OlmError::SessionWedged` propagated up. 2. `machine.rs:1543` catches it → calls `session_manager.mark_device_as_wedged(sender, curve_key)`. 3. That adds the device to `users_for_key_claim` → next outgoing-request cycle does `/keys/claim` → fresh Olm session. 4. `check_if_unwedged()` sends an `m.dummy` to-device event via the fresh session. 5. On the peer side, the dummy decrypts cleanly, creates a fresh session locally, and increments `device.olm_wedging_index` (`olm/account.rs:1393`). 6. Subsequent outbound Megolm encrypts walk `ShareState::Shared { message_index, olm_wedging_index }`, see `olm_wedging_index < d.olm_wedging_index`, and re-share the room_key (`session_manager/group_sessions/mod.rs:751-758`). So in steady state the symptom this issue describes — wedge → no recovery — shouldn't happen. The library handles it. ### The actual gap that bit us on 2026-05-02 `session_manager/sessions.rs:80,118`: ```rust const UNWEDGING_INTERVAL: Duration = Duration::from_secs(60 * 60); ... let should_unwedge = now.checked_sub(creation_time) .map(|elapsed| elapsed > Self::UNWEDGING_INTERVAL) .unwrap_or(true); ``` If the wedged session was created **less than 1 hour ago**, `mark_device_as_wedged` is a no-op — recovery is BLOCKED for up to an hour. Almost certainly what bit us: rapid trust-policy churn during the 2026-05-02 debugging session put us inside that 1-hour window, and the library refused to unwedge until the interval elapsed. This is a deliberate throttle (avoids hammering `/keys/claim` for transient network blips) — sensible for normal Element clients, painful for bridges where wedges are rare + observability is poor. ### Why the issue body's pseudo-Rust can't be implemented as-is The proposed `apply_trust_policy`-side fix assumes `device.olm_wedging_index` and a way to delete sessions for a `sender_key` are accessible. Both are out of reach from outside the crate: - `DeviceData.olm_wedging_index` is `pub(crate)` (`identities/device.rs:103`) — not readable from `apply_trust_policy`'s perspective. - `SessionManager::mark_device_as_wedged` is `pub` but `SessionManager` itself isn't reached via `Client::encryption()` — effectively crate-private to downstream users. - There's no `delete_sessions(&curve25519)` on any public API. ### Bumping doesn't help Checked the source of `matrix-sdk-crypto-0.17.0` against our pinned `=0.14.0`: | | 0.14.0 | 0.17.0 | |---|---|---| | `UNWEDGING_INTERVAL` | 1 hour | 1 hour (unchanged) | | `mark_device_as_wedged` visibility | crate-private | crate-private (unchanged) | | `olm_wedging_index` visibility | `pub(crate)` | `pub(crate)` (unchanged) | | Public force-unwedge API | none | none | Changelog entries for 0.15 / 0.16 / 0.16.1 / 0.17 don't mention wedge handling. ### Where this leaves us The structural fix lives upstream — `matrix-org/matrix-rust-sdk` needs to either expose `force_unwedge_device()` (public method on `Client::encryption()` or `OlmMachine`) OR make `UNWEDGING_INTERVAL` configurable per-client. Until that lands, operators have two paths when wedge-within-1-hour hits: 1. **Wait it out.** Once the interval elapses, the library's recovery loop runs on the next sync cycle. No action needed. 2. **Manual SQL DELETE** (the recipe in the original issue body): direct DELETE on the `session` row for the wedged `sender_key`. Risky — bypasses matrix-sdk's in-memory caches; subsequent encrypts may produce inconsistent state until the next process restart. Use only when (1) isn't acceptable + operator can immediately restart the bridge afterward. ### Leaving this issue open Not closing — the upstream PR is still wanted long-term. When someone has bandwidth to draft the upstream patch: - File issue at `matrix-org/matrix-rust-sdk` asking for either approach (public force-unwedge OR configurable interval). Reference this issue for the bridge-side motivation. - After upstream merges + ships, bump our pin and implement the auto-clear in `apply_trust_policy` (per the issue body's original design, but using the new public API). Re-titling this issue would be reasonable — something like `feat(trust): auto-clear wedged Olm sessions inside the matrix-sdk UNWEDGING_INTERVAL window` — makes the actual ask clearer.