A daemon that had been up half a day was holding 334MB resident, of which
fourteen megabytes were in the main heap. The rest was spread across 147
anonymous mappings, 32 of them full-size (64MB) glibc arenas.
glibc hands each allocating thread its own arena, up to eight per core, and
never returns an arena's free pages to the system on its own. #[tokio::main]
builds a worker per core - 33 on the machine this was measured on - so the
daemon's allocation spread across dozens of arenas and its resident size
became the high-water mark of everything it had ever done at once rather than
what it was actually holding.
Nothing here is a leak in the ordinary sense. The maps are bounded, the
scrollback lives on disk, and the growth plateaus. It is a footprint, and it
is one the allocator will hold onto forever.
The solution
Two settings, both glibc's own:
mallopt(M_ARENA_MAX, 4), which stops the spread. Fewer arenas mean more
threads sharing one, which a daemon that spends its life waiting on sockets
can afford - and glibc's per-thread cache still absorbs the small
allocations without reaching an arena at all.
malloc_trim(0) every five minutes, which is the part that actually hands
the free pages back. A few milliseconds, and nothing else in the process
notices.
Both are Linux-and-glibc only, behind cfg. A musl build has neither symbol
and needs neither.
The cap has to be set before there is a second thread to have an arena,
which is why main builds its own runtime now instead of wearing #[tokio::main]: the macro creates every worker before a line of the body
runs, so setting the cap in there set it after the arenas it was meant to
prevent had already been handed out. Measured that way round too - still 32.
Result
32 arenas before, 4 after, on a daemon carrying the same accounts.
Fixed in nobilis dbbc483 "Hold less of what a long day leaves behind".
## The problem
A daemon that had been up half a day was holding 334MB resident, of which
fourteen megabytes were in the main heap. The rest was spread across 147
anonymous mappings, **32 of them full-size (64MB) glibc arenas**.
glibc hands each allocating thread its own arena, up to eight per core, and
never returns an arena's free pages to the system on its own. `#[tokio::main]`
builds a worker per core - 33 on the machine this was measured on - so the
daemon's allocation spread across dozens of arenas and its resident size
became the high-water mark of everything it had ever done at once rather than
what it was actually holding.
Nothing here is a leak in the ordinary sense. The maps are bounded, the
scrollback lives on disk, and the growth plateaus. It is a footprint, and it
is one the allocator will hold onto forever.
## The solution
Two settings, both glibc's own:
- `mallopt(M_ARENA_MAX, 4)`, which stops the spread. Fewer arenas mean more
threads sharing one, which a daemon that spends its life waiting on sockets
can afford - and glibc's per-thread cache still absorbs the small
allocations without reaching an arena at all.
- `malloc_trim(0)` every five minutes, which is the part that actually hands
the free pages back. A few milliseconds, and nothing else in the process
notices.
Both are Linux-and-glibc only, behind `cfg`. A musl build has neither symbol
and needs neither.
**The cap has to be set before there is a second thread to have an arena**,
which is why `main` builds its own runtime now instead of wearing
`#[tokio::main]`: the macro creates every worker before a line of the body
runs, so setting the cap in there set it after the arenas it was meant to
prevent had already been handed out. Measured that way round too - still 32.
## Result
**32 arenas before, 4 after**, on a daemon carrying the same accounts.
Fixed in nobilis dbbc483 "Hold less of what a long day leaves behind".
Salastil
added the bug label 2026-09-07 17:52:35 -04:00
Verified on a daemon carrying the same accounts: 32 full-size arenas before, 4 after.
Finished - https://git.salastil.com/Salastil/nobilis/commit/dbbc483a32607bf036670b3ed2f0b8c515a0300b
Verified on a daemon carrying the same accounts: 32 full-size arenas before, 4 after.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The problem
A daemon that had been up half a day was holding 334MB resident, of which
fourteen megabytes were in the main heap. The rest was spread across 147
anonymous mappings, 32 of them full-size (64MB) glibc arenas.
glibc hands each allocating thread its own arena, up to eight per core, and
never returns an arena's free pages to the system on its own.
#[tokio::main]builds a worker per core - 33 on the machine this was measured on - so the
daemon's allocation spread across dozens of arenas and its resident size
became the high-water mark of everything it had ever done at once rather than
what it was actually holding.
Nothing here is a leak in the ordinary sense. The maps are bounded, the
scrollback lives on disk, and the growth plateaus. It is a footprint, and it
is one the allocator will hold onto forever.
The solution
Two settings, both glibc's own:
mallopt(M_ARENA_MAX, 4), which stops the spread. Fewer arenas mean morethreads sharing one, which a daemon that spends its life waiting on sockets
can afford - and glibc's per-thread cache still absorbs the small
allocations without reaching an arena at all.
malloc_trim(0)every five minutes, which is the part that actually handsthe free pages back. A few milliseconds, and nothing else in the process
notices.
Both are Linux-and-glibc only, behind
cfg. A musl build has neither symboland needs neither.
The cap has to be set before there is a second thread to have an arena,
which is why
mainbuilds its own runtime now instead of wearing#[tokio::main]: the macro creates every worker before a line of the bodyruns, so setting the cap in there set it after the arenas it was meant to
prevent had already been handed out. Measured that way round too - still 32.
Result
32 arenas before, 4 after, on a daemon carrying the same accounts.
Fixed in nobilis dbbc483 "Hold less of what a long day leaves behind".
Finished -
dbbc483a32Verified on a daemon carrying the same accounts: 32 full-size arenas before, 4 after.