Matrix reaction counts inflate on every reconnect (one message reached 33,224) #213

Closed
opened 2026-09-12 20:43:25 -04:00 by Salastil · 1 comment
Owner

Reported live: every emoji on messages in a bridged Matrix room was climbing into the tens of thousands.

Cause. A reaction was a stored number that incoming events nudged up and down (Store::update_reaction). That is only correct if every event arrives exactly once, and Matrix promises no such thing — an initial sync replays the recent timeline, so every reconnect delivered the same m.reaction events again and each one was added again.

backend/matrix/timeline.rs called record_matrix_reaction_event (which only files the event id for later redaction lookup) and then update_reaction(…, add: true) unconditionally on the next line. Nothing ever asked whether that event had already been counted.

Against a homeserver that was flapping this ran away. One message reached 33,224 on a single emoji.

What made it confusing to diagnose. The counts sit perfectly still while connected and idle — two separate 30-second samples showed zero change — and step up in bursts on reconnect:

🙏  32,464 -> 33,224   +760     across one reconnect window
❤️  13,294 -> 13,598   +304
😂  12,986 -> 13,289   +303
🔥   6,230 ->  6,382   +152

My first attempt to prove it restarted the daemon and saw no change at all, which looked like a refutation. It was not: the homeserver happened to be offline at that moment, so nothing synced and nothing could duplicate. The user pointing out that it had come back online is what made the burst pattern legible.

Fix. https://git.salastil.com/Salastil/nobilis/commit/8ea9a70

A reaction is counted by its own event id, in a matrix_reactions table: the count is how many distinct reaction events are on record, so seeing one twice changes nothing and the client is told nothing rather than told a larger number. On disk rather than in memory, because the duplicates arrive across restarts — exactly when an in-memory guard is empty.

Same change fixes a second bug found on the way: redactions resolved through a map built as reactions arrived, so a reaction removed after a restart fell through to the message-delete path and its count never came back down.

Repair. The old counts cannot be corrected, only cleared — a running total keeps no record of what it was counting. Matrix messages are reset once, guarded by PRAGMA user_version, and refill honestly as reactions are seen again. Only Matrix is touched; Discord counts come from its own message snapshots and were never inflated.

Verified live:

before repair:  🙏 33,224   ❤️ 13,598   😂 13,289   🔥 6,382   👍 5,997
after repair:   🙏      5   ❤️      2   😂      2   🔥     1   👍     1
after two further reconnects: 0 counts changed

4 new tests, 573 passing.

Reported live: every emoji on messages in a bridged Matrix room was climbing into the tens of thousands. **Cause.** A reaction was a stored number that incoming events nudged up and down (`Store::update_reaction`). That is only correct if every event arrives exactly once, and Matrix promises no such thing — an initial sync replays the recent timeline, so every reconnect delivered the same `m.reaction` events again and each one was added again. `backend/matrix/timeline.rs` called `record_matrix_reaction_event` (which only files the event id for later redaction lookup) and then `update_reaction(…, add: true)` unconditionally on the next line. Nothing ever asked whether that event had already been counted. Against a homeserver that was flapping this ran away. One message reached **33,224** on a single emoji. **What made it confusing to diagnose.** The counts sit perfectly still while connected and idle — two separate 30-second samples showed zero change — and step up in bursts on reconnect: ``` 🙏 32,464 -> 33,224 +760 across one reconnect window ❤️ 13,294 -> 13,598 +304 😂 12,986 -> 13,289 +303 🔥 6,230 -> 6,382 +152 ``` My first attempt to prove it restarted the daemon and saw no change at all, which looked like a refutation. It was not: the homeserver happened to be offline at that moment, so nothing synced and nothing could duplicate. The user pointing out that it had come back online is what made the burst pattern legible. **Fix.** https://git.salastil.com/Salastil/nobilis/commit/8ea9a70 A reaction is counted by its own event id, in a `matrix_reactions` table: the count is how many distinct reaction events are on record, so seeing one twice changes nothing and the client is told nothing rather than told a larger number. On disk rather than in memory, because the duplicates arrive *across* restarts — exactly when an in-memory guard is empty. Same change fixes a second bug found on the way: redactions resolved through a map built as reactions arrived, so a reaction removed after a restart fell through to the message-delete path and its count never came back down. **Repair.** The old counts cannot be corrected, only cleared — a running total keeps no record of what it was counting. Matrix messages are reset once, guarded by `PRAGMA user_version`, and refill honestly as reactions are seen again. Only Matrix is touched; Discord counts come from its own message snapshots and were never inflated. **Verified live:** ``` before repair: 🙏 33,224 ❤️ 13,598 😂 13,289 🔥 6,382 👍 5,997 after repair: 🙏 5 ❤️ 2 😂 2 🔥 1 👍 1 after two further reconnects: 0 counts changed ``` 4 new tests, 573 passing.
Salastil added the bugcritical labels 2026-09-12 20:43:25 -04:00
Author
Owner
Finished https://git.salastil.com/Salastil/nobilis/commit/8ea9a70
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Salastil/moho#213