The scrollback store runs SQLite in rollback-journal mode at the default synchronous=FULL, and every message is its own implicit transaction — there is no BEGIN anywhere in store.rs. Every incoming message on every protocol therefore costs a journal write and an fsync.
Measured on an ext4 SSD, replaying the daemons exact one-insert-per-statement pattern (300 inserts):
config
per insert
delete + FULL (today)
26.19 ms
wal + FULL
13.31 ms
wal + NORMAL
0.01 ms
It compounds, because store.rs:11 guards the connection with a single Mutex<Connection>: those 26 ms block every other read and write, including get_backlog and search_messages — and the latter is a LIKE %…% scan holding the same lock.
Consequences:
a 500-message history backfill is roughly 13 seconds of held mutex
a channel at 20 msg/s needs 520 ms of fsync per second of traffic, so it cannot keep up
Fix is PRAGMA journal_mode=WAL + PRAGMA synchronous=NORMAL at Store::open. The trade is losing the last transaction on power loss — never corruption — which for scrollback is worth it. WAL also stops writers blocking readers, which is the second half of the problem above.
Note journal_mode is persistent in the database file, so this applies to the existing db on first open, not only to new ones.
Found during a valgrind/DHAT pass. Memory itself came back clean; this was the performance finding.
The scrollback store runs SQLite in rollback-journal mode at the default `synchronous=FULL`, and every message is its own implicit transaction — there is no `BEGIN` anywhere in `store.rs`. Every incoming message on every protocol therefore costs a journal write and an fsync.
Measured on an ext4 SSD, replaying the daemons exact one-insert-per-statement pattern (300 inserts):
| config | per insert |
| --- | --- |
| `delete` + `FULL` (today) | **26.19 ms** |
| `wal` + `FULL` | 13.31 ms |
| `wal` + `NORMAL` | **0.01 ms** |
It compounds, because `store.rs:11` guards the connection with a single `Mutex<Connection>`: those 26 ms block every other read and write, including `get_backlog` and `search_messages` — and the latter is a `LIKE %…%` scan holding the same lock.
Consequences:
- a 500-message history backfill is roughly 13 seconds of held mutex
- a channel at 20 msg/s needs 520 ms of fsync per second of traffic, so it cannot keep up
Fix is `PRAGMA journal_mode=WAL` + `PRAGMA synchronous=NORMAL` at `Store::open`. The trade is losing the last transaction on power loss — never corruption — which for scrollback is worth it. WAL also stops writers blocking readers, which is the second half of the problem above.
Note `journal_mode` is persistent in the database file, so this applies to the existing db on first open, not only to new ones.
Found during a valgrind/DHAT pass. Memory itself came back clean; this was the performance finding.
Salastil
added the bug label 2026-09-12 17:03:41 -04:00
Measured the live daemon to size this, over a 6,439 s window (one full uptime):
6.97 messages/s sustained across all buffers — 44,857 rows written
busiest single buffer: a bridged Matrix room at 2.99 msg/s (19,262 rows)
At the measured 26.19 ms per insert that is 182 ms of fsync per second of wall clock — about 18% of one core, with the Mutex<Connection> held the whole time. Every get_backlog, search_messages and getMentions queues behind it.
Under wal + NORMAL the same load costs roughly 0.07 ms/s.
So this is not a theoretical improvement at this traffic level; the store lock is genuinely unavailable ~18% of the time today.
Measured the live daemon to size this, over a 6,439 s window (one full uptime):
- **6.97 messages/s sustained** across all buffers — 44,857 rows written
- busiest single buffer: a bridged Matrix room at **2.99 msg/s** (19,262 rows)
At the measured 26.19 ms per insert that is **182 ms of fsync per second of wall clock — about 18% of one core, with the `Mutex<Connection>` held the whole time**. Every `get_backlog`, `search_messages` and `getMentions` queues behind it.
Under `wal` + `NORMAL` the same load costs roughly 0.07 ms/s.
So this is not a theoretical improvement at this traffic level; the store lock is genuinely unavailable ~18% of the time today.
PRAGMA journal_mode = WAL and PRAGMA synchronous = NORMAL at Store::open, as specified.
Re-measured through the store's own write path rather than a standalone replay, and one detail is worth recording for anyone repeating it: the first run showed 0.02ms against 0.01ms, because std::env::temp_dir() is tmpfs here and fsync on tmpfs costs nothing. On this machine's ext4 home, where the database actually lives:
config
per insert
delete + FULL
5.72 ms
wal + NORMAL
0.06 ms
Lower than the 26.19ms in the report but the same conclusion, and the benchmark that ships with the fix (wal_is_worth_having, #[ignore]d) deliberately writes to a real filesystem for that reason.
Two things beyond the one-line change:
The pragma is asked rather than told. WAL needs shared memory and a filesystem that provides it, so a database on NFS would quietly stay in rollback mode; it now logs when the mode it settled on is not the one requested.
A test covers the case the report calls out - that journal_mode is persistent, so the 225MB scrollback.db already on disk converts on first open - by writing one in DELETE mode deliberately and reopening it.
Finished in [`bd33f88`](https://git.salastil.com/Salastil/nobilis/commit/bd33f88), carried into moho by [`3e746cf`](https://git.salastil.com/Salastil/moho/commit/3e746cf).
`PRAGMA journal_mode = WAL` and `PRAGMA synchronous = NORMAL` at `Store::open`, as specified.
Re-measured through the store's own write path rather than a standalone replay, and one detail is worth recording for anyone repeating it: the first run showed 0.02ms against 0.01ms, because `std::env::temp_dir()` is tmpfs here and fsync on tmpfs costs nothing. On this machine's ext4 home, where the database actually lives:
| config | per insert |
| --- | --- |
| `delete` + `FULL` | 5.72 ms |
| `wal` + `NORMAL` | 0.06 ms |
Lower than the 26.19ms in the report but the same conclusion, and the benchmark that ships with the fix (`wal_is_worth_having`, `#[ignore]`d) deliberately writes to a real filesystem for that reason.
Two things beyond the one-line change:
- The pragma is asked rather than told. WAL needs shared memory and a filesystem that provides it, so a database on NFS would quietly stay in rollback mode; it now logs when the mode it settled on is not the one requested.
- A test covers the case the report calls out - that `journal_mode` is persistent, so the 225MB scrollback.db already on disk converts on first open - by writing one in `DELETE` mode deliberately and reopening it.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The scrollback store runs SQLite in rollback-journal mode at the default
synchronous=FULL, and every message is its own implicit transaction — there is noBEGINanywhere instore.rs. Every incoming message on every protocol therefore costs a journal write and an fsync.Measured on an ext4 SSD, replaying the daemons exact one-insert-per-statement pattern (300 inserts):
delete+FULL(today)wal+FULLwal+NORMALIt compounds, because
store.rs:11guards the connection with a singleMutex<Connection>: those 26 ms block every other read and write, includingget_backlogandsearch_messages— and the latter is aLIKE %…%scan holding the same lock.Consequences:
Fix is
PRAGMA journal_mode=WAL+PRAGMA synchronous=NORMALatStore::open. The trade is losing the last transaction on power loss — never corruption — which for scrollback is worth it. WAL also stops writers blocking readers, which is the second half of the problem above.Note
journal_modeis persistent in the database file, so this applies to the existing db on first open, not only to new ones.Found during a valgrind/DHAT pass. Memory itself came back clean; this was the performance finding.
Measured the live daemon to size this, over a 6,439 s window (one full uptime):
At the measured 26.19 ms per insert that is 182 ms of fsync per second of wall clock — about 18% of one core, with the
Mutex<Connection>held the whole time. Everyget_backlog,search_messagesandgetMentionsqueues behind it.Under
wal+NORMALthe same load costs roughly 0.07 ms/s.So this is not a theoretical improvement at this traffic level; the store lock is genuinely unavailable ~18% of the time today.
Finished in
bd33f88, carried into moho by3e746cf.PRAGMA journal_mode = WALandPRAGMA synchronous = NORMALatStore::open, as specified.Re-measured through the store's own write path rather than a standalone replay, and one detail is worth recording for anyone repeating it: the first run showed 0.02ms against 0.01ms, because
std::env::temp_dir()is tmpfs here and fsync on tmpfs costs nothing. On this machine's ext4 home, where the database actually lives:delete+FULLwal+NORMALLower than the 26.19ms in the report but the same conclusion, and the benchmark that ships with the fix (
wal_is_worth_having,#[ignore]d) deliberately writes to a real filesystem for that reason.Two things beyond the one-line change:
journal_modeis persistent, so the 225MB scrollback.db already on disk converts on first open - by writing one inDELETEmode deliberately and reopening it.