Salastil/nobilis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Housekeeping that keeps up, and a vacuum that vacuums

Browse Source 
Two problems, and a third found while measuring the second.

**The prune ran on the wrong clock.** It shared the six-hourly interval the
media caches use, and the two are not the same kind of growth: a cache grows
with how many people you have seen, scrollback with how fast they are
talking. The busiest buffer here takes three messages a second, so six hours
after a prune that correctly cut it to 5000 it held 24,000 rows, and a full
interval reaches about 65,000 - thirteen times the cap the setting names. It
now has its own loop on a fifteen-minute interval, which keeps the worst case
a couple of thousand rows over and makes each pass small: the six-hourly one
had 44,131 rows to delete and took 2.1 seconds.

**The vacuum gave back almost nothing.** It asked for 200 pages once per pass
and only after a delete. Against a live freelist of 10,196 pages that is
twelve days of catching up while more arrives. It now asks how much is
actually free and keeps going until little is, in chunks, taking the lock per
chunk so one long vacuum is not a stall for every reader - and it runs every
pass, since a message deleted in a room frees pages too.

**And the part the ticket could not have known:** `PRAGMA
incremental_vacuum(200)` through `execute_batch` was handing back *one* page,
not 200. SQLite frees one page per step of that statement, and execute_batch
steps once. Traced directly - a freelist of 534 falling by exactly one per
call - and fixed by stepping to completion. So the real rate was 4KB per six
hours, two hundred times worse than the ticket's arithmetic.

One regression of my own, caught by the same test: setting journal_mode=WAL
before auto_vacuum=INCREMENTAL leaves auto_vacuum at NONE, silently, because
SQLite will not change it on a database already in WAL. Every scrollback.db
created since the WAL change would never have handed back a page. The pragmas
are now in the order that works and a test asserts both.

Measured against a copy of the live 225MB database, through this code:
8,447 rows pruned in 0.1s, 10,428 pages handed back, 225MB -> 184MB, the
whole pass under a second.

Closes #209.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Salastil
2026-09-12 21:19:22 -04:00
co-authored by Claude Opus 5
parent 9e885a40b1
commit be400666f4
2 changed files with 178 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/main.rs
+63 -26
View File
@@ -149,6 +149,42 @@ fn moho_dir_migrations(
vec![("cache", cache_root.join("moho"), cache_root.join("nobilis"))]
}
/// Cuts every buffer back to the cap, and gives the space back.
///
/// Split from the cache sweeps above so it can run on its own cadence. The
/// vacuum runs on every pass rather than only after a delete: pages are freed
/// by a message being deleted from a room as well, and the backlog they leave
/// is what kept the file growing when nothing but a prune could reclaim it.
async fn prune_scrollback(state: AppState) {
// The same settling time the sweeps take: a daemon that has just started
// is reconnecting and backfilling, and pruning into that is work done
// twice.
tokio::time::sleep(std::time::Duration::from_secs(60)).await;
loop {
match state.store.prune_old_messages(SCROLLBACK_KEEP_PER_BUFFER) {
Ok(0) => {}
Ok(n) => tracing::info!(
"scrollback: pruned {n} row(s) beyond {SCROLLBACK_KEEP_PER_BUFFER} kept per buffer"
),
Err(e) => tracing::warn!("scrollback: pruning failed: {e}"),
}
match state.store.incremental_vacuum() {
Ok(0) => {}
Ok(pages) => tracing::info!("scrollback: handed back {pages} page(s) to the filesystem"),
Err(e) => tracing::debug!("scrollback: incremental_vacuum failed: {e}"),
}
// Kept to the same number the runtime holds, so the list does not
// grow without bound across restarts while showing only the newest.
match state.store.prune_transfers(crate::runtime::DCC_KEEP as i64) {
Ok(0) | Err(_) => {}
Ok(n) => tracing::info!("transfers: forgot {n} old record(s)"),
}
tokio::time::sleep(SCROLLBACK_SWEEP_INTERVAL).await;
}
}
/// flock()-based singleton lock, same purpose as
/// daemon/nobilis/nobilis.c's acquire_singleton_lock(): prevent two nobilis
/// processes racing the same socket/account-store. The lock file's fd is
@@ -373,15 +409,34 @@ async fn run() -> Result<()> {
}
}
/// Keeps two genuinely unbounded-over-time growth vectors in check for as
/// long as this daemon process stays up: scrollback (a busy buffer left
/// open for months of uptime never stops growing on its own) and the
/// Sneedchat avatar cache (every distinct poster ever seen gets a
/// permanently-cached file - see backend/sneedchat/mod.rs's
/// cached_avatar_path). Neither is a one-time startup cost, so this
/// re-runs periodically rather than once.
/// How often the scrollback is cut back to its cap.
///
/// Its own interval, because it used to share the six-hourly one the media
/// caches use and they are not the same kind of growth. A cache grows with how
/// many distinct people you have seen; scrollback grows with how fast they are
/// talking. Measured on a live daemon, the busiest buffer takes three messages
/// a second - so six hours after a prune that correctly cut it to 5000 it held
/// 24,000 rows, and a full interval reaches something like 65,000. Thirteen
/// times the cap the setting names.
///
/// Fifteen minutes keeps the worst case near 2,700 rows over. It also keeps
/// each pass small, which matters because the prune holds the database while
/// it runs: the once-per-six-hours pass had 44,131 rows to delete and took
/// 2.1 seconds, where a quarter-hourly one has a few thousand.
const SCROLLBACK_SWEEP_INTERVAL: std::time::Duration = std::time::Duration::from_secs(15 * 60);
/// The number of messages kept per buffer. What the pruning is for.
const SCROLLBACK_KEEP_PER_BUFFER: i64 = 5000;
/// Keeps the stores that grow with use from growing forever: the scrollback
/// database, and the media caches (every distinct poster ever seen leaves a
/// cached avatar - see backend/sneedchat/mod.rs's cached_avatar_path).
///
/// Two loops rather than one, on two intervals, because the two answer to
/// different things - see SCROLLBACK_SWEEP_INTERVAL. Neither is a one-time
/// startup cost, so both re-run for as long as the daemon is up.
async fn run_housekeeping(state: AppState) {
const SCROLLBACK_KEEP_PER_BUFFER: i64 = 5000;
tokio::spawn(prune_scrollback(state.clone()));
// Ahead of the wait: this is a one-time local correction, and holding it
// back a minute would only mean a minute of pictures that should move
@@ -391,24 +446,6 @@ async fn run_housekeeping(state: AppState) {
// Let the initial reconnect burst above settle before the first pass.
tokio::time::sleep(std::time::Duration::from_secs(60)).await;
loop {
match state.store.prune_old_messages(SCROLLBACK_KEEP_PER_BUFFER) {
Ok(0) => {}
Ok(n) => {
tracing::info!("scrollback: pruned {n} row(s) beyond {SCROLLBACK_KEEP_PER_BUFFER} kept per buffer");
if let Err(e) = state.store.incremental_vacuum() {
tracing::debug!("scrollback: incremental_vacuum failed: {e}");
}
}
Err(e) => tracing::warn!("scrollback: pruning failed: {e}"),
}
// Kept to the same number the runtime holds, so the list does not
// grow without bound across restarts while showing only the newest.
match state.store.prune_transfers(crate::runtime::DCC_KEEP as i64) {
Ok(0) | Err(_) => {}
Ok(n) => tracing::info!("transfers: forgot {n} old record(s)"),
}
backend::sneedchat::sweep_avatar_cache().await;
backend::sneedchat::sweep_attachment_cache().await;
backend::matrix::sweep_media_cache().await;
src/store.rs
+115 -14
View File
@@ -40,6 +40,21 @@ impl Store {
let conn = Connection::open(db_path)
.with_context(|| format!("opening scrollback db at {}", db_path.display()))?;
// Only takes effect for a brand-new file (or after a full VACUUM,
// which isn't done automatically here since that could be slow on
// an existing multi-month scrollback.db) - lets incremental_vacuum
// actually hand freed pages back to the OS after prune_old_messages/
// delete_message, instead of SQLite just quietly reusing them
// in-place forever without the file ever shrinking on disk.
let _ = conn.execute_batch("PRAGMA auto_vacuum = INCREMENTAL;");
// Strictly after auto_vacuum, and that is not a style choice: SQLite
// will not change auto_vacuum on a database that is already in WAL
// mode, and says nothing when it refuses. Setting WAL first left every
// newly created scrollback.db at auto_vacuum=NONE, which makes the
// incremental_vacuum below a permanent no-op - the file would free
// pages internally and never hand one back. Found by a vacuum that
// reclaimed nothing while reporting that it had.
// Write-ahead logging, and don't fsync on every one.
//
// Every message written here is its own implicit transaction - there
@@ -77,13 +92,6 @@ impl Store {
// Persistent in the database file for journal_mode, per-connection for
// synchronous, which is why this one is set every time.
let _ = conn.execute_batch("PRAGMA synchronous = NORMAL;");
// Only takes effect for a brand-new file (or after a full VACUUM,
// which isn't done automatically here since that could be slow on
// an existing multi-month scrollback.db) - lets incremental_vacuum
// actually hand freed pages back to the OS after prune_old_messages/
// delete_message, instead of SQLite just quietly reusing them
// in-place forever without the file ever shrinking on disk.
let _ = conn.execute_batch("PRAGMA auto_vacuum = INCREMENTAL;");
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -1109,13 +1117,62 @@ impl Store {
Ok(deleted)
}
/// Hands freed pages back to the OS a couple hundred at a time, rather
/// than a single large blocking VACUUM - a no-op on a database that
/// predates `auto_vacuum = INCREMENTAL` (see `open`'s doc comment).
pub fn incremental_vacuum(&self) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute_batch("PRAGMA incremental_vacuum(200);")?;
Ok(())
/// Hands freed pages back to the OS until there are few enough left to
/// stop caring - a no-op on a database that predates
/// `auto_vacuum = INCREMENTAL` (see `open`'s doc comment).
///
/// This used to reclaim a flat 200 pages, once, on a pass that had deleted
/// something. At 4KB a page that is 800KB per housekeeping pass, against a
/// live scrollback that was holding 10,196 free pages - 40MB - and gaining
/// on every prune. Twelve days to give back what is already free, while
/// more arrives, is a file that only grows.
///
/// So it asks how much is free and keeps going until the answer is small,
/// in chunks, taking the lock per chunk rather than for the whole run: the
/// connection is behind one mutex, so a single long vacuum is a stall for
/// every reader as well. Bounded per call, because giving back eighty
/// megabytes at once is not worth blocking a housekeeping pass for when
/// the next one is minutes away.
///
/// Returns how many pages it handed back.
pub fn incremental_vacuum(&self) -> Result<usize> {
// ~2MB of work between chances for anybody else to use the database.
const CHUNK_PAGES: i64 = 512;
// A little slack left free on purpose: a database in steady use is
// always freeing and reusing a few pages, and chasing the last of them
// would mean vacuuming on every pass forever to no purpose.
const KEEP_FREE: i64 = 256;
// ~80MB, so one pass cannot run away on a database that has been
// growing untended for months.
const MAX_PAGES_PER_CALL: i64 = 20_000;
let mut reclaimed: i64 = 0;
while reclaimed < MAX_PAGES_PER_CALL {
let conn = self.conn.lock().unwrap();
let free: i64 = conn.query_row("PRAGMA freelist_count", [], |row| row.get(0))?;
if free <= KEEP_FREE {
break;
}
let take = CHUNK_PAGES.min(free - KEEP_FREE).min(MAX_PAGES_PER_CALL - reclaimed);
// Stepped to completion rather than run with `execute_batch`,
// which is how this quietly did almost nothing before. SQLite
// implements this pragma as a statement that frees *one page per
// step*; execute_batch steps once, so the old
// `incremental_vacuum(200)` handed back a single 4KB page per
// housekeeping pass, not the 200 it reads as. Measured: a freelist
// of 534 fell by exactly one per call.
let mut stmt = conn.prepare(&format!("PRAGMA incremental_vacuum({take})"))?;
let mut rows = stmt.query([])?;
while rows.next()?.is_some() {}
drop(rows);
drop(stmt);
// Dropped explicitly to say what the chunking is for: the next
// iteration takes the lock again, and between the two anything
// waiting on it gets a turn.
drop(conn);
reclaimed += take;
}
Ok(reclaimed as usize)
}
}
@@ -1175,6 +1232,18 @@ mod tests {
assert_eq!(journal_mode(&s), "wal", "a fresh db was not created in WAL mode");
// Both, and in that order. SQLite refuses to set auto_vacuum on a
// database already in WAL mode and does not say so, so getting these
// two the wrong way round silently costs the ability to hand pages
// back at all - which is what incremental_vacuum exists to do.
let auto_vacuum: i64 = s
.conn
.lock()
.unwrap()
.query_row("PRAGMA auto_vacuum", [], |row| row.get(0))
.expect("asking auto_vacuum");
assert_eq!(auto_vacuum, 2, "2 is INCREMENTAL; 0 means the vacuum can never reclaim anything");
// And the fsync-per-message setting that is the other half of it.
let sync: i64 = s
.conn
@@ -1228,6 +1297,38 @@ mod tests {
let _ = std::fs::remove_dir_all(dir);
}
/// The vacuum used to hand back a flat 200 pages a pass, which against a
/// real scrollback's ten thousand free ones was twelve days of catching up
/// while more arrived. It has to clear what is actually free.
#[test]
fn the_vacuum_clears_the_freelist_rather_than_a_fixed_slice() {
let (s, dir) = store();
// Enough rows that deleting them frees well over the old 200 pages.
for i in 0..4000 {
append_saying(&s, "#chan", &format!("m{i}"), &"x".repeat(400));
}
s.prune_old_messages(10).expect("pruning");
let free = |s: &Store| -> i64 {
s.conn
.lock()
.unwrap()
.query_row("PRAGMA freelist_count", [], |row| row.get(0))
.expect("asking the freelist")
};
let before = free(&s);
assert!(before > 200, "the fixture freed only {before} pages - it cannot show the difference");
let reclaimed = s.incremental_vacuum().expect("vacuuming");
let after = free(&s);
assert!(
reclaimed as i64 >= before - 256,
"handed back {reclaimed} of {before} free pages"
);
assert!(after <= 256, "{after} pages still free after a vacuum");
let _ = std::fs::remove_dir_all(dir);
}
/// Like `append`, but with text worth searching for.
fn append_saying(s: &Store, buffer: &str, id: &str, body: &str) {
s.append_message(buffer, id, "someone", body, 1, false, false, "chat", None, &[], false, None, &[], &[], None, None, "channel", None, &[])