trufflepig/evaluation/performance

CLI performance smoke

python3 evaluation/performance/run_smoke.py records bounded local measurements in cli_smoke.json. It reads the current lunatic and tgstation working trees and removes its dedicated index caches afterwards. Set TRUFFLEPIG_SMOKE_BINARY to use a frozen copy of the executable. Each command has a 120-second deadline; an unsuccessful initial index skips dependent runs.

The binary is a debug build. A cold index means an empty Trufflepig cache, with the operating-system page cache uncontrolled. The warm index uses the preceding index cache in a fresh process. Ordinary search is lexical, runs in a fresh process with --no-daemon, and includes repository reconciliation. These timings therefore do not measure resident-daemon query latency. Avoid replacing the binary or modifying either corpus during the script.

Elapsed time comes from Python's monotonic clock. Linux wait4 supplies each child's CPU times and peak resident memory in KiB; /usr/bin/time is unavailable on this host. Cache measurements sum apparent file sizes and allocated blocks after each child exits, excluding captured command output. They are persistent disk measurements, not peak temporary disk usage.

Commit IDs and dirty flags describe provenance. Inputs are current working-tree files, not frozen evaluation snapshots. Each operation has one sample, and other development processes may compete for resources. No latency confidence interval, retrieval recall, task success, or performance guarantee follows from this smoke. Responses preserve coverage, errors, and timeout outcomes so incomplete work remains visible. The corpora are development inputs from the evaluation manifest.

The recorded debug run completes all six commands within their deadlines:

Corpus	Cold index	Warm index	Search with reconciliation	Cold peak RSS	Index disk
lunatic	112.48 s	21.92 s	21.76 s	88.43 MiB	450.32 MiB
tgstation	102.29 s	52.83 s	52.58 s	98.04 MiB	1261.34 MiB

Lunatic reports 2,480 indexed files, 9 excluded files, and 14 parsing failures. Tgstation reports 11,620 indexed files, 6,573 excluded files, and 849 parsing failures. Both report zero walk failures and zero truncated files. Parsing failures prevent treating structural coverage as complete. Persistent index storage is approximately 21.5 and 17.1 times indexed source bytes respectively; these measurements expose costs requiring optimization. The sampled exact-name queries return a source definition first, which establishes neither recall nor general ranking quality.

Background history smoke

Run python3 evaluation/performance/run_history_smoke.py --trufflepig target/debug/trufflepig. The fixed fixture creates 270 first-parent commits in one Rust file, then runs 16 exact-symbol queries across five client slots while history indexing proceeds. It checks query results, captured-view completion, and daemon shutdown. Scratch uses TMPDIR or a cache directory beneath the user's home, and is removed.

Recorded observations include complete client elapsed times. This small synthetic run establishes concurrent progress only; it is not a throughput target, large-repository latency result, or agent effectiveness trial.

Workspace lifecycle smoke

Run python3 evaluation/performance/run_workspace_smoke.py --binary target/debug/trufflepig. The synthetic workspace contains two Git repositories with two commits each. It checks cold publication warming, foreign source reads from member subdirectories, eight queries while source edits continue, reconciliation of the final edit, handle persistence across coordinator restart, and index reuse by an overlapping workspace. History checks wait for indexing, compare a scoped commit, and read its exact preimage through a handle from the other member. An isolated foreground query checks that --no-daemon starts no fixture background processes.

Recorded observations retain each response, exit status, complete client elapsed time, and observed edit counts. RSS samples read Linux /proc for fixture daemons and history workers after client completion; they are neither peak memory nor client memory measurements. Cold startup may report unavailable until an initial member publication completes. Every command has a 35-second deadline; the smoke does not establish a latency target.

Scratch uses TMPDIR beneath the user's cache and rejects /tmp. Cleanup stops the fixture coordinators and member daemons, waits for history registrations to expire, and records any forced termination of remaining fixture processes. The fixture directories are removed. The recorded run verifies navigation and lifecycle mechanisms, not feature placement, model residency, or agent effectiveness.