Salastil/odysseus
1
0
Fork 0
mirror of https://github.com/pewdiepie-archdaemon/odysseus.git synced 2026-08-02 03:28:36 -04:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: Salastil/odysseus:discovery
Branches Tags
Salastil/odysseus:dev Salastil/odysseus:dependabot/pip/python-b2cbfa3651 Salastil/odysseus:dependabot/github_actions/actions-d3190f52d4 Salastil/odysseus:discovery Salastil/odysseus:fix/add-directory-event-loop Salastil/odysseus:main Salastil/odysseus:feature/tbench-bounded-read-guard
..
compare: Salastil/odysseus:fix/add-directory-event-loop
Branches Tags
Salastil/odysseus:dependabot/pip/python-b2cbfa3651 Salastil/odysseus:dependabot/github_actions/actions-d3190f52d4 Salastil/odysseus:dev Salastil/odysseus:discovery Salastil/odysseus:fix/add-directory-event-loop Salastil/odysseus:main Salastil/odysseus:feature/tbench-bounded-read-guard

2 Commits
discovery .. fix/add-directory-event-loop

Author SHA1 Message Date
StressTestor e222e92153 fix(personal): serialize add/remove/reload on an async job lock 
The #5558 fix took the job lock INSIDE the threadpool worker and only on the
add path, so (1) remove_directory and /reload mutated PersonalDocsManager's
unsynchronized list/index concurrently with an in-flight add — the inconsistent
state the PR claimed to prevent — and (2) a queued add blocked on the lock while
holding an AnyIO threadpool token, starving the shared pool.

Move the lock to an asyncio.Lock acquired in the async handler BEFORE offloading,
and route add, remove and reload through it. A waiting request now parks on the
event loop instead of pinning a worker, and all three mutators are serialized so
the 'add/remove are serialized and cannot leave inconsistent state' guarantee
holds. remove and reload also run their blocking work off the event loop. The
lock is per-router so each app binds it to its own loop; single-process scope.

Tests: add-vs-remove and add-vs-reload serialization regressions (async via
ASGITransport, since asyncio.Lock deadlocks starlette TestClient's portal); the
existing add-vs-add test converted to the same driver.
 2026-07-27 20:43:04 +00:00
StressTestor b91f48f50a fix(personal): run directory indexing off the event loop (#5558) 
POST /api/personal/add_directory called rag.index_personal_documents
inline from an async handler, so the whole indexing job (os.walk, file
reads, per-chunk embedding, Chroma inserts) ran on the event loop and
every other request queued behind it. Indexing a real directory froze
the UI and API for 25+ minutes with no sign of life.

Move the blocking section into the threadpool via run_in_threadpool.
personal_docs_manager.add_directory stays inside it because its
refresh_index() re-extracts text across tracked directories, which is
also blocking work. A module-level lock serializes index jobs so the
threadpool move does not introduce parallel jobs racing
PersonalDocsManager's unsynchronized list mutations and file writes;
they previously serialized on the blocked loop, so one-at-a-time is
behavior parity.
 2026-07-27 20:43:04 +00:00
21 changed files with 1155 additions and 1316 deletions
Expand all files Collapse all files
.github/ISSUE_TEMPLATE/feature_request.yml
+2 -2
View File
@@ -8,8 +8,8 @@ body:
value: |
**Before submitting:** search [open issues](https://github.com/odysseus-dev/odysseus/issues)
and [discussions](https://github.com/odysseus-dev/odysseus/discussions) first.
The [roadmap](https://github.com/odysseus-dev/odysseus/blob/main/ROADMAP.md) is directional rather than a complete backlog.
Feature requests that duplicate an existing issue or accepted proposal may be closed as duplicates.
Feature requests that duplicate [ROADMAP.md](https://github.com/odysseus-dev/odysseus/blob/main/ROADMAP.md)
or an existing open issue will be closed as duplicates.
If your idea needs community input before it becomes a concrete proposal,
start a [discussion](https://github.com/odysseus-dev/odysseus/discussions/categories/ideas) instead.
README.md
+4 -8
View File
@@ -1,5 +1,3 @@
# Odysseus
<p align="center">
<img src="docs/odysseus-wordmark.png" alt="Odysseus" width="238">
</p>
@@ -11,8 +9,6 @@
<p align="center">
<a href="#quick-start">Quick Start</a> ·
<a href="docs/setup.md">Setup Guide</a> ·
<a href="docs/ARCHITECTURE.md">Architecture</a> ·
<a href="SECURITY.md">Security</a> ·
<a href="CONTRIBUTING.md">Contributing</a> ·
<a href="ROADMAP.md">Roadmap</a>
</p>
@@ -55,15 +51,15 @@ Native installs, GPU notes, Windows/macOS instructions, HTTPS, and configuration
## Demo
Explore the interface through the [interactive product tour](docs/index.html).
A full hover-to-play tour lives on the landing page: [`docs/index.html`](docs/index.html).
## Contributing
Help is welcome. The best entry points are fresh-install testing, provider setup bugs, mobile/editor polish, documentation, and small focused refactors. Read the [contributing guide](CONTRIBUTING.md), review the [public roadmap](ROADMAP.md), and browse the open [GitHub issues](https://github.com/odysseus-dev/odysseus/issues).
Help is welcome. The best entry points are fresh-install testing, provider setup bugs, mobile/editor polish, docs, and small focused refactors. See [CONTRIBUTING.md](CONTRIBUTING.md) and [ROADMAP.md](ROADMAP.md).
## Security
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model or service ports publicly. Read the [security policy](SECURITY.md) and the [deployment security guidance](docs/setup.md#security-notes).
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly. Deployment details are in the [setup guide](docs/setup.md#security-notes).
## Star History
@@ -77,4 +73,4 @@ Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled
## License
Licensed under AGPL-3.0-or-later. See the [license](LICENSE) and [acknowledgments](ACKNOWLEDGMENTS.md).
AGPL-3.0-or-later -- see [LICENSE](LICENSE) and [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md).
ROADMAP.md
+75 -43
View File
@@ -1,55 +1,87 @@
# Roadmap
# Roadmap / Help Wanted
This document provides a high-level view of the areas Odysseus is currently improving.
Odysseus is on a voyage, but not home yet. It works great for me (lol), but this ship is moving fast and feedback/help would be appreciated! (I don't know what I'm doing, help).
It is directional rather than exhaustive. Priorities may change as the project evolves, defects are discovered, and maintainers learn more from implementation work and user feedback.
If you see weird CSS, strange layout behavior, or a suspiciously murky corner of
the codebase, you are probably right to stay away.
For current implementation work, see the open [GitHub issues](https://github.com/odysseus-dev/odysseus/issues). Accepted behaviour should be documented in the repository alongside the code.
## High Priority
## Current priorities
- SQUASH BUGS
- Fresh install smoke tests on Linux, macOS, and Windows. Docker, native Python,
and WSL all need coverage.
### Reliability and setup
- Integration audit: do integrations even work? Confirm what works, what needs setup docs, and what should be removed or hidden.
- Cookbook reliability on other computers. This is probably the area most likely to need work across different machines, GPUs, drivers, shells, and Python environments.
- Cookbook SGLang support across platforms. Make sure SGLang setup/serve works
predictably on Linux, Windows/WSL, macOS where possible, Docker, and common
NVIDIA/AMD hardware paths.
- Deep Research model presets by hardware. Recommend approved model/parameter
profiles for small, medium, and large local setups so people with different
hardware can use Deep Research without guessing. Surface this either in Deep
Research settings or as a Cookbook scan/dropdown suggestion.
- Cookbook model scan/download ranking. Prioritize newer architectures and
better hardware-fit models instead of scoring everything almost the same.
Ranking should account for architecture age, quant format, VRAM/RAM fit,
backend support, vision/mmproj requirements, and likely serve reliability.
- Cookbook error feedback and logging. Failed downloads, dependency installs,
preflights, and serve jobs should show the actual command/output/error in the
UI, with copyable logs and clear next steps instead of just "crashed".
- Agent prompt/context bloat. Agent mode is too heavy for smaller local models:
tool schemas, skills, memory, documents, and instructions can eat the context
before the user request really starts. We need slimmer prompts, better tool
selection, smaller default tool sets, and clearer guidance for models with
4k/8k/16k context windows.
- Local model speculative decoding support. For Odysseus-tuned local models,
plan to ship or recommend a small same-tokenizer draft model when the serving
backend supports it. Early vLLM testing showed a generic `Qwen3-0.6B` draft
beside `Qwen3-8B` can materially reduce wall time, while an unsupported
DSpark conversion performed poorly. Treat this as a supported draft-model lane
first; keep MTP-specific packaging as future work only when the architecture
and runtime support are real. Judge this by time-to-success, tool correctness,
grammar, and unchanged target output, not tokens/sec alone.
- Skill/tool prompt-injection audit. User-editable skills, notes, documents,
fetched pages, and memories should be treated as untrusted data. Keep testing
whether models follow malicious instructions from those surfaces.
- Better degraded-state reporting for ChromaDB, SearXNG, email, ntfy, and provider probes.
- Email performance audit. Fetching, searching, opening, deleting, and sending
email can feel slow, especially over IMAP/SMTP providers with high latency.
Need someone who knows mail performance to profile the current flow, identify
whether the bottleneck is IMAP folder select/fetch, cache invalidation,
attachment/body loading, SMTP handshakes, or frontend refresh behavior, then
propose safer caching/prefetch/batching without breaking multi-account state.
- Provider setup/probing audit for Anthropic, Gemini, Groq, xAI, OpenRouter, OpenAI, and DeepSeek.
- Improve fresh-install and smoke-test coverage across supported environments.
- Make provider setup, probing, and failure states more predictable.
- Improve Cookbook reliability across hardware, operating systems, drivers, shells, and serving backends.
- Improve degraded-state reporting and recovery guidance when optional services are unavailable.
## Refactor Targets
- CSS cleanup. `static/style.css` basically Calypso's island atm.
- Tour core helper. The onboarding tours have too much copy-pasted scaffolding; promote a shared `tour-core.js` helper before adding more tours.
- Modal/window positioning cleanup. Some window controls have improved, but the
underlying popup/dropdown/fixed-position behavior is still too fragile.
- Mobile media override discoverability. A lot of "CSS did not move" bugs are mobile `@media` overrides of the same selector; comments or linting around desktop/mobile paired rules would help.
- Dead code pass for old routes, stale feature flags, and unused UI states.
### Local model workflows
## Frontend
- Improve hardware-aware model recommendations and compatibility guidance.
- Evaluate serving optimizations, including speculative decoding, through reproducible benchmarks.
- Improve installation, preflight checks, logging, and error reporting for local model serving.
- Reduce prompt and context overhead for smaller local models.
- Expand the Editor for quicker, more robust everyday use. Better file/document
handling, smoother window behavior, clearer save/export flows, stronger image
editing affordances, and fewer brittle edge cases.
- Better AI integration for Notes and Todos. Notes should be easier for the
agent to read, update, summarize, and turn into actions. Todos should be
assignable to an agent from the UI, possibly through a button, task action,
or dedicated skill/tool flow.
- Mobile gallery/editor polish. Easier to launch/download inpaint model or any missing pieces.
- Accessibility pass: keyboard navigation, focus states, contrast, reduced motion.
- Improve empty states and error messages on fresh installs.
- Tighten first-run setup, hints, and tours so they do not repeat or fight each other.
- Vendor CDN assets eventually for a more fully self-hosted/offline mode.
### Safety and resilience
## Backend
- Continue hardening tool execution, filesystem access, credentials, networking, and destructive operations.
- Treat content from documents, notes, memories, skills, and fetched pages as potentially untrusted.
- Improve security-focused regression coverage and operational guidance.
- Review integrations that expand access to sensitive data or privileged operations.
- More tests around endpoint probing and provider setup.
- Better task scheduler defaults and visibility.
- Backup/restore guide and helper flow for `data/`.
- Security hardening around admin-only tools and clear docs for their risk.
### Product usability
## Not The Focus Right Now
- Improve first-run setup, onboarding, hints, and tours.
- Improve accessibility, keyboard navigation, focus behaviour, contrast, and reduced-motion support.
- Improve empty states, error messages, and recovery paths.
- Strengthen Notes, Todos, Editor, mobile, and everyday workspace flows.
### Architecture and maintainability
- Reduce duplication and technical debt through focused, reviewable refactors.
- Improve subsystem documentation as behaviour and architecture become stable.
- Remove stale code, obsolete feature flags, and unsupported integrations.
- Keep implementation decisions grounded in current code and verified behaviour.
## Tracking work
Concrete implementation tasks, defects, proposals, and technical investigations are tracked in:
- [GitHub Issues](https://github.com/odysseus-dev/odysseus/issues)
- [Contributing Guide](CONTRIBUTING.md)
Maintainers may use additional private coordination tools for ownership, planning, and unresolved decisions.
This roadmap is not a complete backlog or a guarantee that a particular item will be delivered.
I prob shouldnt add more themes.
SECURITY.md
+1 -3
View File
@@ -37,6 +37,4 @@ Only `.env.example`, docs, source, tests, and static assets should be committed.
## Reporting
Report security vulnerabilities privately through [GitHub Security Advisories](https://github.com/odysseus-dev/odysseus/security/advisories/new).
Do not open a public issue or discussion, and do not disclose exploit details publicly.
Please report vulnerabilities privately via GitHub security advisories if available, or by opening a minimal issue that does not disclose exploit details.
THREAT_MODEL.md
+5 -5
View File
@@ -68,14 +68,14 @@ External content that reaches the LLM is treated as untrusted via `src/prompt_se
- `X-Content-Type-Options: nosniff` and `Referrer-Policy: no-referrer` everywhere.
- **CSP:** nonce-based `script-src 'self' 'nonce-{nonce}' https://cdn.jsdelivr.net`. `style-src 'unsafe-inline'` is intentionally kept — `static/index.html` ships inline `<style>` blocks and JS modules set `style=""` attributes at runtime. Inline styles do not execute script so the risk is visual-only. Removing this requires templating the HTML files and auditing all JS-set style attributes.
## Token-Supplied Model Endpoints
Direct `/api/v1/chat` requests with a token-supplied `base_url` must use a public HTTP(S) endpoint. This restriction applies only to untrusted direct values; administrator-configured endpoints may intentionally use local or LAN URLs for private model providers.
## Known Gaps
These are open, acknowledged, and contributor help is welcome:
1. **No shell/filesystem sandbox.** The agent `bash` and `read_file`/`write_file` tools run as the app process user with no network egress filtering or filesystem confinement. A successful prompt-injection reaching a shell-enabled admin session can make outbound requests to internal services. See #1058 for the sandbox proposal.
2. **Token scopes are coarse.** There is no way to grant a session a subset of the owning user's privileges. Companion/mobile tokens carry either `chat` or `admin` scope with no per-capability granularity.
2. **SSRF via `/api/v1/chat` `base_url` parameter.** A chat-scoped API token can supply an arbitrary `base_url`; the server forwards the LLM request to that host without validating the scheme or address. PR #1039 fixes this.
3. **`src/search/` partial consolidation.** `src.search.core` and `src.search.providers` correctly alias `services.search` via `sys.modules` replacement. `analytics`, `cache`, `content`, `query`, and `ranking` are still independent copies that can drift. The SSRF regression tests in `tests/test_webhook_ssrf_resilience.py` test `src.webhook_manager` directly (separate from search), so the safety net there is intact. See #1058.
4. **Token scopes are coarse.** There is no way to grant a session a subset of the owning user's privileges. Companion/mobile tokens carry either `chat` or `admin` scope with no per-capability granularity.
discovery/README.md
-34
View File
@@ -1,34 +0,0 @@
# Odysseus discovery maps
Compact, code-grounded discovery maps of cross-cutting systems in the checked-in Odysseus codebase. They preserve investigation context and open factual questions; they are not canonical subsystem specifications, a feature certification, or a substitute for normal testing.
> [!IMPORTANT]
> Checked-in code is the source of truth for current behaviour. Mature subsystem specifications, where they exist, are the canonical documentation of accepted subsystem behaviour. Check code, tests, and configuration before reconciling a discovery finding. Discovery remains non-canonical.
## Explore the maps
| Document | Purpose |
|---|---|
| [Current system map](system-map.md) | Records evidence locations, confirmed local observations, and factual open questions about subsystem boundaries. |
| [Safety boundaries](safety-boundaries.md) | Records evidence about broad authority, safeguards, confirmed risks or gaps, and unverified behaviour. |
## Working rules
- **Trace the code first.** Confirm the current path in source before recording a claim.
- **Promote selectively.** When an owning mature specification exists, add a fact only when it is verified, useful, and not already represented there.
- **Record missing ownership.** When no owning specification exists, retain the verified finding in discovery and record missing documentation ownership as a follow-up.
- **Retain uncertainty here.** Keep unresolved questions and useful investigation context in discovery rather than treating them as canonical truth.
- **Keep specifications current-state only.** Do not record intentions, design direction, refactor plans, decision history, priority, ownership, or sequencing here.
- **Investigate with cause.** Do not exhaustively revalidate existing functionality without a report, visible failure, relevant change, or high-authority review need.
- **Review authority carefully.** Give execution, data access, external tools, credentials, destructive operations, and unattended work focused review.
- **Use stable locations.** Cite modules, routes, classes, and functions instead of fragile line ranges or generated evidence tables.
## Reconciliation flow
1. Start with the relevant map and trace the cited code.
2. Classify the finding against current source evidence and an owning mature specification where one exists.
3. Promote only verified, useful facts that are missing from an existing owning specification.
4. When no owning specification exists, retain the verified finding here and record missing documentation ownership as a follow-up; otherwise retain unresolved context here and correct stale wording.
> [!NOTE]
> This package intentionally contains no generator, validator, maturity scale, feature database, or parallel work tracker. The [architecture runtime inventory](./architecture-runtime-inventory.md) preserves dated structural metrics, investigation context, and historical planning as an explicitly non-canonical snapshot.
discovery/architecture-runtime-inventory.md
-331
View File
@@ -1,331 +0,0 @@
# Architecture runtime inventory
> [!WARNING]
> This document is a dated structural snapshot, not a canonical runtime specification.
> Counts, paths, and implementation details may drift as the repository changes.
> Verify implementation-sensitive claims against the current code and tests.
- **Branch:** `discovery`
- **Commit:** `c762efe1c97a`
- **Generated:** `2026-07-28T05:38:14+01:00`
- **Historical context:** readability and refactor planning in [#4071](https://github.com/odysseus-dev/odysseus/issues/4071) and [#4082](https://github.com/odysseus-dev/odysseus/issues/4082)
## Disposition
> [!NOTE]
> Reviewed for documentation classification. Stable runtime structure and
> subsystem ownership have been transferred to the proposed canonical
> destination, [`docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md), for
> maintainer review.
>
> This document retains dated metrics, rankings, investigation context,
> refactor-sensitive observations, and historical planning. Those contents are
> non-canonical and belong under `discovery/`.
| Content | Authority and destination |
|---|---|
| Stable runtime structure | Proposed canonical destination: `docs/ARCHITECTURE.md` |
| Stable subsystem boundaries | Proposed canonical destination: `docs/ARCHITECTURE.md` |
| Frontend module organization | `static/js/MODULE_SUMMARY.md` |
| Counts, line totals, and rankings | This non-canonical inventory |
| Investigation context and open questions | `discovery/` |
| Refactor options and prioritization | Issues, Plane, or non-canonical discovery material |
The transfer preserves the stable facts without promoting generated metrics or
historical prioritization into canonical documentation.
## Purpose
This inventory provides a reviewable map of the current repository structure,
large runtime modules, major subsystem boundaries, and refactor-sensitive areas.
It does not:
- define accepted subsystem behaviour;
- certify runtime correctness;
- prescribe a committed refactor sequence;
- replace focused specifications, tests, or source review.
For cross-cutting implementation evidence, see the [discovery maps](./README.md).
## Top-level runtime structure
| Area | Role |
|---|---|
| `app.py` | FastAPI application composition and entry point |
| `launcher.py` | Application launch support |
| `setup.py` | Native setup workflow |
| `core/` | Authentication, middleware, persistence, sessions, and platform primitives |
| `routes/` | HTTP and API route handlers |
| `src/` | Application services, orchestration, tools, providers, and runtime helpers |
| `services/` | Domain-oriented service packages |
| `mcp_servers/` | Built-in MCP server implementations |
| `scripts/` | CLI tools, diagnostics, maintenance, and migration helpers |
| `static/` | No-build browser frontend and bundled assets |
| `tests/` | Automated test suite and supporting test infrastructure |
## Directory snapshot
| Directory | Tracked files | Tracked Python files | Direct subdirectories |
|---|---:|---:|---|
| `src/` | 143 | 143 | `agent_tools/`, `model_capability_readers/`, `search/`, `tools/` |
| `routes/` | 73 | 73 | `admin_wipe/`, `cleanup/`, `compare/`, `contacts/`, `gallery/`, `history/`, `memory/`, `note/`, `research/` |
| `core/` | 11 | 11 | None |
| `services/` | 42 | 40 | `docs/`, `faces/`, `hwfit/`, `memory/`, `research/`, `search/`, `shell/`, `stt/`, `tts/`, `youtube/` |
| `mcp_servers/` | 5 | 5 | None |
| `scripts/` | 44 | 17 | `_completion/`, `_lib/`, `demo_email/` |
| `static/js/` | 154 | 0 | `calendar/`, `color/`, `compare/`, `editor/`, `emailLibrary/`, `markdown/`, `model/`, `research/`, `util/` |
| `tests/` | 768 | 758 | `cli/`, `helpers/`, `streaming/`, `tools/` |
> [!NOTE]
> Counts in this table use `git ls-files`, so generated caches, virtual
> environments, and other untracked local files are excluded.
## Largest backend modules
Large files are review signals, not proof that a module should be split.
Coupling, ownership, import compatibility, tests, and runtime authority matter more
than line count alone.
| Rank | File | Lines | Classes | Top-level functions | Review signal |
|---:|---|---:|---:|---:|---|
| 1 | `routes/email_routes.py` | 6032 | 1 | 58 | High |
| 2 | `src/agent_loop.py` | 5248 | 0 | 63 | High |
| 3 | `routes/cookbook_routes.py` | 4545 | 0 | 16 | High |
| 4 | `mcp_servers/email_server.py` | 2920 | 0 | 77 | High |
| 5 | `src/llm_core.py` | 2895 | 3 | 85 | High |
| 6 | `src/builtin_actions.py` | 2845 | 2 | 27 | High |
| 7 | `routes/model_routes.py` | 2743 | 0 | 65 | High |
| 8 | `src/task_scheduler.py` | 2627 | 1 | 8 | Medium |
| 9 | `core/database.py` | 2562 | 28 | 67 | High |
| 10 | `routes/gallery/gallery_routes.py` | 2325 | 0 | 16 | Medium |
| 11 | `routes/chat_routes.py` | 2063 | 0 | 18 | Medium |
| 12 | `routes/shell_routes.py` | 1971 | 1 | 21 | Medium |
| 13 | `src/visual_report.py` | 1933 | 0 | 11 | Medium |
| 14 | `routes/email_helpers.py` | 1888 | 3 | 48 | Medium |
| 15 | `routes/document_routes.py` | 1810 | 0 | 5 | Medium |
| 16 | `src/tools/cookbook.py` | 1705 | 0 | 34 | Medium |
| 17 | `routes/calendar_routes.py` | 1667 | 2 | 19 | Medium |
| 18 | `routes/skills_routes.py` | 1662 | 3 | 19 | Medium |
| 19 | `src/tool_schemas.py` | 1595 | 0 | 3 | Medium |
| 20 | `routes/email_pollers.py` | 1551 | 0 | 23 | Medium |
The largest current backend concentrations include:
- email routing and helper logic;
- agent-loop orchestration;
- Cookbook lifecycle and serving logic;
- provider and model routing;
- task scheduling;
- shared database models and persistence helpers.
These areas require focused ownership and compatibility analysis before structural
changes are attempted.
## Largest frontend modules
| Rank | File | Lines |
|---:|---|---:|
| 1 | `static/style.css` | 41132 |
| 2 | `static/js/document.js` | 11200 |
| 3 | `static/js/emailLibrary.js` | 8505 |
| 4 | `static/js/slashCommands.js` | 6520 |
| 5 | `static/js/chat.js` | 6001 |
| 6 | `static/js/settings.js` | 5819 |
| 7 | `static/js/notes.js` | 5365 |
| 8 | `static/app.js` | 4681 |
| 9 | `static/js/cookbookRunning.js` | 4433 |
| 10 | `static/js/galleryEditor.js` | 4386 |
| 11 | `static/js/cookbookServe.js` | 4305 |
| 12 | `static/js/calendar.js` | 3722 |
| 13 | `static/js/cookbook.js` | 3677 |
| 14 | `static/js/sessions.js` | 3665 |
| 15 | `static/js/documentLibrary.js` | 3422 |
| 16 | `static/js/tasks.js` | 3187 |
| 17 | `static/js/admin.js` | 3144 |
| 18 | `static/js/gallery.js` | 2958 |
| 19 | `static/js/cookbook-hwfit.js` | 2826 |
| 20 | `static/js/chatRenderer.js` | 2808 |
The browser frontend remains a no-build ES-module application. Its current source
tree is authoritative; the maintained structural summary is available in
[`static/js/MODULE_SUMMARY.md`](../static/js/MODULE_SUMMARY.md).
CSS modularization remains tracked separately in
[#2617](https://github.com/odysseus-dev/odysseus/issues/2617).
## Major subsystem boundaries
| Subsystem | Primary implementation locations |
|---|---|
| Application startup | `app.py`, `src/app_initializer.py`, `core/` |
| Authentication and sessions | `core/auth.py`, `core/middleware.py`, `core/session_manager.py`, `routes/auth_routes.py` |
| Chat and streaming | `routes/chat_routes.py`, `routes/chat_helpers.py`, `src/chat_handler.py`, `src/chat_processor.py`, `src/llm_core.py` |
| Agents and tools | `src/agent_loop.py`, `src/tool_execution.py`, `src/agent_tools/`, `src/tools/`, `src/tool_policy.py`, `src/tool_security.py` |
| Models and providers | `routes/model_routes.py`, `src/model_discovery.py`, `src/model_capabilities.py`, `src/endpoint_resolver.py`, `src/llm_core.py` |
| Cookbook and hardware fit | `routes/cookbook_routes.py`, `routes/cookbook_helpers.py`, `src/cookbook_serve_lifecycle.py`, `services/hwfit/` |
| Search and research | `routes/search_routes.py`, `services/search/`, `routes/research/`, `services/research/`, `src/deep_research.py` |
| Documents and retrieval | `routes/document_routes.py`, `src/document_processor.py`, `src/personal_docs.py`, `src/rag_manager.py`, `src/pdf_runtime.py` |
| Memory and skills | `routes/memory/`, `services/memory/`, `routes/skills_routes.py` |
| Email | `routes/email_routes.py`, `routes/email_helpers.py`, `routes/email_pollers.py`, `mcp_servers/email_server.py` |
| Calendar, contacts, notes, and tasks | `routes/calendar_routes.py`, `routes/contacts/`, `routes/note/`, `routes/task_routes.py`, `src/task_scheduler.py` |
| Media and speech | `routes/gallery/`, `routes/stt_routes.py`, `routes/tts_routes.py`, `services/stt/`, `services/tts/` |
| Persistence and operations | `core/database.py`, `src/runtime_paths.py`, `src/bg_jobs.py`, `routes/backup_routes.py`, `routes/cleanup/` |
For a broader evidence map, see
[`system-map.md`](./system-map.md).
## Refactor-sensitive areas
### Shared persistence
`core/database.py` is a central dependency containing models and shared persistence
helpers. Changes can affect routes, services, background work, tests, migrations,
and import compatibility.
A split should not begin from file size alone. It requires:
- an importer inventory;
- model and helper ownership decisions;
- migration compatibility checks;
- stable re-export or migration strategy;
- focused and full-suite validation.
### Agent orchestration
`src/agent_loop.py` coordinates model interaction, tool selection, policy decisions,
multi-round execution, and background behaviour. Extraction work must preserve tool
event semantics, policy enforcement, cancellation, and test patch points.
Historical agent-loop modularization discussion is tracked in
[#3266](https://github.com/odysseus-dev/odysseus/issues/3266).
### Tool implementation boundaries
Tool implementation is no longer represented by one proposed future package alone.
Current responsibilities are distributed across:
- `src/tool_execution.py`;
- `src/tool_schemas.py`;
- `src/tool_index.py`;
- `src/tool_policy.py`;
- `src/tool_security.py`;
- `src/agent_tools/`;
- `src/tools/`;
- remaining compatibility surfaces such as `src/tool_implementations.py`.
Historical tool modularization work is tracked in
[#3629](https://github.com/odysseus-dev/odysseus/issues/3629).
### Route ownership
`routes/` now contains both flat modules and domain packages. Existing package
boundaries should be extended only through focused changes. Broad mechanical route
movement would affect registration, imports, tests, monkeypatch targets, and
compatibility paths.
### Frontend concentration
The no-build frontend contains several large JavaScript modules and one central CSS
file. Refactors should preserve module load order, global compatibility exports,
DOM contracts, deep-link handling, and browser behaviour.
## Non-implemented architecture options
> [!NOTE]
> The paths below are historical or possible design directions. They do not describe
> the current repository and are not approved implementation plans.
Earlier planning discussed:
- renaming `app.py` to `main.py`;
- moving agent orchestration into a new `src/agent/` package;
- introducing broad `src/domain/`, `src/infra/`, `src/api/`, or `src/pkg/` layers;
- moving all routes into domain subpackages;
- splitting database models into a new infrastructure hierarchy.
These options should be reconsidered against the current tree rather than copied
forward as assumed targets.
## Refactor guardrails
- Keep structural changes behaviour-preserving.
- Change one ownership boundary at a time.
- Do not mix file movement with unrelated feature work.
- Preserve existing import and monkeypatch paths where compatibility is required.
- Identify focused tests before modifying high-authority modules.
- Validate startup, imports, and affected runtime paths.
- Avoid repository-wide package reorganizations without maintainer agreement.
- Treat generated metrics as snapshots, not architectural decisions.
## Reproduce the snapshot
Run these commands from the repository root.
```bash
# Tracked directory totals
for dir in src routes core services mcp_servers scripts static/js tests; do
files="$(git ls-files "$dir" | wc -l)"
python_files="$(git ls-files "$dir" '*.py' | wc -l)"
printf '%-14s tracked=%-5s python=%-5s\n' \
"$dir" \
"$files" \
"$python_files"
done
# Largest tracked backend files
git ls-files \
'app.py' \
'launcher.py' \
'setup.py' \
'core/*.py' \
'core/**/*.py' \
'routes/*.py' \
'routes/**/*.py' \
'services/*.py' \
'services/**/*.py' \
'src/*.py' \
'src/**/*.py' \
'mcp_servers/*.py' \
'scripts/*.py' \
'scripts/**/*.py' |
xargs wc -l |
sort -nr |
head -31
# Largest tracked frontend source files
git ls-files \
'static/*.js' \
'static/*.css' \
'static/*.html' \
'static/**/*.js' \
'static/**/*.css' \
'static/**/*.html' |
grep -vE '\.min\.js$' |
xargs wc -l |
sort -nr |
head -31
```
## Validation for architecture changes
Use the smallest relevant checks first, then expand according to risk:
```bash
python3 -m compileall -q app.py core routes services src
venv/bin/python -m pytest tests/<focused-test-file>.py -q
venv/bin/python -m pytest -q
```
Startup, browser, Docker, and integration checks may also be required depending on
the affected boundary.
## Related documentation
- [Documentation style](../docs/STYLE.md)
- [Discovery maps](../discovery/README.md)
- [Current system map](../discovery/system-map.md)
- [Safety boundaries](../discovery/safety-boundaries.md)
- [Frontend module summary](../static/js/MODULE_SUMMARY.md)
- [Testing standard](../tests/TESTING_STANDARD.md)
discovery/safety-boundaries.md
-142
View File
@@ -1,142 +0,0 @@
# Safety boundaries
> [!IMPORTANT]
> This non-canonical discovery map records code-grounded safeguards, confirmed risks or gaps, and unverified behaviour. Broad authority does not by itself establish a vulnerability. Verify the cited source before relying on a finding. No destructive test, external connection, or real credential was used for this map.
## Navigate the boundaries
- [Shell and subprocess execution](#shell-and-subprocess-execution)
- [Filesystem access and workspace confinement](#filesystem-access-and-workspace-confinement)
- [Agent-controlled tool dispatch](#agent-controlled-tool-dispatch)
- [MCP and external tool servers](#mcp-and-external-tool-servers)
- [Outbound network requests and URL validation](#outbound-network-requests-and-url-validation)
- [Secrets, credentials, and vault sessions](#secrets-credentials-and-vault-sessions)
- [Authentication and privileged administration](#authentication-and-privileged-administration)
- [Deletion, wipe, backup, and restore](#deletion-wipe-backup-and-restore)
- [Background jobs and unattended task execution](#background-jobs-and-unattended-task-execution)
## Shell and subprocess execution
- **Boundary:** Shell routes, agent `bash` and `python` tools, local model serving, and detached background jobs.
- **Available authority:** Commands run as the application process user and can create child processes.
- **User-controlled inputs:** Direct shell requests, model-produced tool arguments, scheduled-task prompts, and model-serving configuration.
- **Current safeguards:** Agent dispatch applies owner/admin checks and tool policy; process helpers use timeouts or bounded background-job lifecycle where implemented.
- **Confirmed risks or gaps:** Intentional authority with a confirmed gap: the agent shell starts in its workspace but is not sandboxed to it, and has no egress sandbox. This is documented in source and the threat model; it is not a newly demonstrated bypass.
- **Unverified behaviour:** Role-gate and disabled-tool outcomes, direct shell-route behaviour, and timeout, cancellation, and output handling for foreground and detached processes remain unverified.
## Filesystem access and workspace confinement
- **Boundary:** Agent read, write, patch, listing, glob, and grep tools.
- **Available authority:** Read and modify files within active workspace confinement or fallback allowlisted roots.
- **User-controlled inputs:** Tool paths, patches, file contents, search patterns, and workspace selection passed into the tool dispatcher.
- **Current safeguards:** [`src/tool_execution.py`](../src/tool_execution.py) resolves paths, blocks sensitive subpaths, applies allowlist containment, and tightens paths to the active workspace when one is bound. File tools use those resolvers.
- **Confirmed risks or gaps:** Intentional authority with safeguards. The file-tool policy does not sandbox the shell; treating a workspace as a whole-process containment boundary would be incorrect.
- **Unverified behaviour:** Traversal, symlink, sensitive-name, absolute-path, and workspace-switch behaviour remains unverified.
## Agent-controlled tool dispatch
- **Boundary:** Model output becomes native or parsed tool calls and is dispatched by the agent loop.
- **Available authority:** The authority of every enabled tool, including privileged built-ins and external tools.
- **User-controlled inputs:** Chat content, attached/retrieved content that may influence the model, tool arguments, per-request tool selection, and policy toggles.
- **Current safeguards:** [`src/tool_security.py`](../src/tool_security.py) blocks protected tools for non-admin users and fails closed for malformed tool names; [`src/tool_policy.py`](../src/tool_policy.py) supports disabled and guide-only policy; prompt-security helpers label untrusted context.
- **Confirmed risks or gaps:** Credible risk requiring verification: aliases, legacy text tools, native function calls, and MCP-qualified names must all reach the same policy outcome. The code has specific alias handling for email/MCP names, which makes this a sensitive compatibility seam.
- **Unverified behaviour:** The current policy outcomes for owner role, request mode, disabled state, native versus parsed invocation, qualified aliases, and external-content entry points remain unverified.
## MCP and external tool servers
- **Boundary:** Configured MCP servers and their tools are exposed to the agent through the MCP manager and routes.
- **Available authority:** Depends on the server: external network access, local process access, messaging, or data mutation may be delegated outside the application.
- **User-controlled inputs:** Server configuration, remote OAuth completion, tool arguments, and model-selected MCP calls.
- **Current safeguards:** MCP routes are registered through [`routes/mcp_routes.py`](../routes/mcp_routes.py); MCP-qualified tools are denied to non-admin users by [`src/tool_security.py`](../src/tool_security.py). OAuth state and token persistence are handled in [`src/mcp_oauth.py`](../src/mcp_oauth.py).
- **Confirmed risks or gaps:** Credible risk requiring verification: an MCP server authority is broader than the application can infer from its tool name. This map does not establish a trust or approval model for server installation and individual tool invocation.
- **Unverified behaviour:** Server onboarding, credential storage, server-origin trust, OAuth callback deployment, tool disablement, and invocation audit behaviour remain unverified.
## Outbound network requests and URL validation
- **Boundary:** Search/content fetch, research, webhooks, skill import, provider endpoints, and other HTTP clients.
- **Available authority:** The application can make outbound requests from its network position.
- **User-controlled inputs:** Search/fetch URLs, imported skill URLs, webhook configuration, and some endpoint settings.
- **Current safeguards:** [`src/url_security.py`](../src/url_security.py) validates untrusted public HTTP URLs and fails closed on unsuitable schemes or private addresses. [`services/search/content.py`](../services/search/content.py) resolves and rejects non-public hosts, pins resolved addresses for fetches, caps bodies, and limits redirects.
- **Confirmed risks or gaps:** Intentional split: administrator-created model endpoints may target private providers, while untrusted URLs use public-address checks. That distinction is required for self-hosted deployments but needs explicit call-site review.
- **Unverified behaviour:** The URL-source classification for outbound clients and the current handling of redirects and DNS changes remain unverified.
## Secrets, credentials, and vault sessions
- **Boundary:** Application-managed encrypted secrets, API keys, provider credentials, and Bitwarden/Vaultwarden CLI sessions.
- **Available authority:** Credentials unlock remote providers and connected personal services.
- **User-controlled inputs:** Administrative configuration, login/unlock requests, imported settings, and agent vault tool arguments.
- **Current safeguards:** [`src/secret_storage.py`](../src/secret_storage.py) uses a locally stored Fernet key with restrictive permissions for supported database secrets. Vault routes require an administrator, avoid passing master passwords in command arguments, and set restrictive permissions on the vault-session file.
- **Confirmed risks or gaps:** Confirmed current boundary: vault session data is persisted through the vault path, not through [`src/secret_storage.py`](../src/secret_storage.py). This is an unresolved question about current security semantics, not a confirmed exposure.
- **Unverified behaviour:** Current encryption-at-rest, owner scope, rotation, lock/logout, backup/restore, and log/tool-result exposure behaviour remains unverified.
## Authentication and privileged administration
- **Boundary:** Session authentication, API tokens, privileged routes, and internal tool loopback.
- **Available authority:** Administrative identity can access execution, settings, integrations, data deletion, and secrets.
- **User-controlled inputs:** Login/signup data, session cookies, API tokens, authentication configuration, and requests to privileged routes.
- **Current safeguards:** [`core/auth.py`](../core/auth.py), [`core/middleware.py`](../core/middleware.py), and route-level checks establish identity and administrator gates. [`app.py`](../app.py) warns when localhost bypass is configured; [`SECURITY.md`](../SECURITY.md) documents deployment requirements.
- **Confirmed risks or gaps:** Intentional authority with safeguards. Security depends on deployments keeping authentication enabled and internal services private; this map does not audit reverse-proxy or environment configuration.
- **Unverified behaviour:** Setup, anonymous, non-admin, admin, token, and internal-loopback behaviour, including privileged-route gate consistency, remains unverified.
## Deletion, wipe, backup, and restore
- **Boundary:** Administrative wipe, cleanup, backup import/export, and the backup restore command.
- **Available authority:** Delete or replace user data and credentials.
- **User-controlled inputs:** Administrative HTTP requests, cleanup choices, backup payloads, archive paths, and restore command options.
- **Current safeguards:** Administrative wipe routes use the administrative boundary. Cleanup exposes a preview route before mutation. The documented backup tool requires explicit restore confirmation, stages the old data directory, and validates archive members before extraction.
- **Confirmed risks or gaps:** Intentional destructive authority. Backup archives contain secrets by design, as documented in [`docs/backup-restore.md`](../docs/backup-restore.md); this is an operator confidentiality responsibility, not a code defect established here.
- **Unverified behaviour:** Role-gate, confirmation, archive-rejection, staged-recovery, and owner-isolation behaviour remains unverified. No destructive runtime test was performed.
## Background jobs and unattended task execution
- **Boundary:** Scheduled tasks, background-job monitor, startup tasks, and notification/delivery work that continue without an active browser request.
- **Available authority:** Scheduled agent work can obtain model access and, for eligible owners, shell and file tools; task output can interact with connected services.
- **User-controlled inputs:** Stored task prompt, schedule, model/crew selection, enabled-tool configuration, output target, and prior persisted state.
- **Current safeguards:** [`src/task_scheduler.py`](../src/task_scheduler.py) serializes execution, records task runs, associates work with an owner, and applies the agent owner-based tool gate. [`src/bg_jobs.py`](../src/bg_jobs.py) keeps bounded state and can terminate overlong subprocess jobs.
- **Confirmed risks or gaps:** Credible risk requiring verification: authority is inherited and exercised later, so changes to roles, task configuration, and disabled tools must be checked at execution time rather than assumed from task creation.
- **Unverified behaviour:** Creation, editing, role-change, scheduling, cancellation, restart-recovery, and execution behaviour remains unverified, including whether current policy is re-evaluated before privileged action.
discovery/system-map.md
-139
View File
@@ -1,139 +0,0 @@
# Current system map
> [!NOTE]
> This non-canonical discovery map is an evidence guide, not an exhaustive feature catalog or runtime certification. Verify the cited source before relying on a finding. Each section records local implementation observations, evidence locations, confirmed current problems, and unresolved factual questions.
## Navigate the system
- [Startup and application composition](#startup-and-application-composition)
- [Frontend shell and browser interaction](#frontend-shell-and-browser-interaction)
- [Chat, sessions, and streaming](#chat-sessions-and-streaming)
- [Agents, tools, and execution](#agents-tools-and-execution)
- [Models, providers, and local serving](#models-providers-and-local-serving)
- [Search and research](#search-and-research)
- [Documents, retrieval, and personal knowledge](#documents-retrieval-and-personal-knowledge)
- [Memory and skills](#memory-and-skills)
- [Email, calendar, contacts, notes, and tasks](#email-calendar-contacts-notes-and-tasks)
- [Media, speech, and image work](#media-speech-and-image-work)
- [Authentication, secrets, and privileged administration](#authentication-secrets-and-privileged-administration)
- [Persistence, background work, and operations](#persistence-background-work-and-operations)
## Startup and application composition
- **How it works:** [`app.py`](../app.py) creates the application, mounts static assets, constructs shared services, registers route factories, and owns lifespan startup and shutdown. [`src/app_initializer.py`](../src/app_initializer.py) prepares application state; [`core/`](../core/) provides persistence, authentication, middleware, sessions, and platform helpers.
- **Evidence locations:** [`app.py`](../app.py); [`src/app_initializer.py`](../src/app_initializer.py); [`core/database.py`](../core/database.py); [`core/auth.py`](../core/auth.py); [`core/middleware.py`](../core/middleware.py); [`routes/`](../routes/).
- **Known problems:** None recorded by this mapping.
- **Open question:** Which component currently owns startup and shutdown for each long-lived service?
## Frontend shell and browser interaction
- **How it works:** [`static/index.html`](../static/index.html) is served by the root and SPA deep-link routes in [`app.py`](../app.py); [`static/app.js`](../static/app.js), [`static/style.css`](../static/style.css), and [`static/js/`](../static/js/) implement the client surface.
- **Evidence locations:** [`static/index.html`](../static/index.html); [`static/app.js`](../static/app.js); [`static/js/`](../static/js/); [`static/style.css`](../static/style.css); [`app.py`](../app.py) deep-link handlers.
- **Known problems:** The `/backgrounds` route in [`app.py`](../app.py) calls `serve_html_with_nonce` for `static/backgrounds.html`, but that file is absent from [`static/`](../static/). This is a confirmed broken prototype route, not evidence about the rest of the frontend.
- **Open question:** Is `/backgrounds` currently an intentionally supported route or an obsolete prototype?
## Chat, sessions, and streaming
- **How it works:** [`routes/chat_routes.py`](../routes/chat_routes.py) and [`routes/chat_helpers.py`](../routes/chat_helpers.py) coordinate requests, session state, and SSE delivery. [`src/chat_handler.py`](../src/chat_handler.py), [`src/chat_processor.py`](../src/chat_processor.py), [`src/llm_core.py`](../src/llm_core.py), and [`src/session_actions.py`](../src/session_actions.py) provide message preparation, provider interaction, and session operations.
- **Evidence locations:** [`routes/chat_routes.py`](../routes/chat_routes.py); [`routes/chat_helpers.py`](../routes/chat_helpers.py); [`routes/session_routes.py`](../routes/session_routes.py); [`src/chat_handler.py`](../src/chat_handler.py); [`src/chat_processor.py`](../src/chat_processor.py); [`src/llm_core.py`](../src/llm_core.py); [`core/session_manager.py`](../core/session_manager.py).
- **Known problems:** [`src/agent_loop.py`](../src/agent_loop.py) annotates `_resolved_tool_event_name` with `Any` but imports no `Any` and does not enable postponed annotation evaluation. Python evaluates that annotation while importing the module, so this is an import-time defect at the checked baseline.
- **Open question:** No end-to-end provider or browser streaming run was performed for this map.
## Agents, tools, and execution
- **How it works:** [`src/agent_loop.py`](../src/agent_loop.py) drives multi-round tool use. [`src/tool_execution.py`](../src/tool_execution.py) dispatches calls and binds workspace context. [`src/agent_tools/`](../src/agent_tools/) contains individual implementations; [`src/tool_security.py`](../src/tool_security.py) and [`src/tool_policy.py`](../src/tool_policy.py) apply role and request policies. Long-running command work is represented by [`src/bg_jobs.py`](../src/bg_jobs.py).
- **Evidence locations:** [`src/agent_loop.py`](../src/agent_loop.py); [`src/tool_execution.py`](../src/tool_execution.py); [`src/agent_tools/`](../src/agent_tools/); [`src/tool_security.py`](../src/tool_security.py); [`src/tool_policy.py`](../src/tool_policy.py); [`src/tool_schemas.py`](../src/tool_schemas.py); [`src/bg_jobs.py`](../src/bg_jobs.py).
- **Known problems:** The import-time annotation defect above blocks the main agent/tool path. The shell is intentionally not a filesystem or network sandbox; that is an authority boundary, not by itself a vulnerability claim.
- **Open question:** Which native, legacy, and MCP-qualified invocation paths reach each policy gate?
## Models, providers, and local serving
- **How it works:** Model routes delegate to discovery, capabilities, endpoint resolution, and LLM core modules. Cookbook routes and hardware-fit services handle model lifecycle and local-serving support.
- **Evidence locations:** [`routes/model_routes.py`](../routes/model_routes.py); [`src/model_discovery.py`](../src/model_discovery.py); [`src/model_capabilities.py`](../src/model_capabilities.py); [`src/endpoint_resolver.py`](../src/endpoint_resolver.py); [`src/llm_core.py`](../src/llm_core.py); [`routes/cookbook_routes.py`](../routes/cookbook_routes.py); [`src/cookbook_serve_lifecycle.py`](../src/cookbook_serve_lifecycle.py); [`services/hwfit/`](../services/hwfit/).
- **Known problems:** None recorded by this mapping.
- **Open question:** Which endpoint inputs are administrator-created and permitted to use private provider addresses?
## Search and research
- **How it works:** HTTP search routes use [`services/search/`](../services/search/); research is exposed through [`routes/research/`](../routes/research/) and implemented in [`services/research/`](../services/research/), [`src/deep_research.py`](../src/deep_research.py), and related helpers. [`src/search/`](../src/search/) remains an import-compatibility layer for callers not yet moved to `services.search`.
- **Evidence locations:** [`routes/search_routes.py`](../routes/search_routes.py); [`services/search/`](../services/search/); [`routes/research/research_routes.py`](../routes/research/research_routes.py); [`services/research/`](../services/research/); [`src/deep_research.py`](../src/deep_research.py); [`src/search/`](../src/search/).
- **Known problems:** None recorded by this mapping.
- **Open question:** No live provider request was made; provider configuration and network access remain unverified.
## Documents, retrieval, and personal knowledge
- **How it works:** Document routes coordinate upload handling, document processing, and editor actions. Personal-document and RAG modules use Chroma and embedding clients. PDF viewing uses the optional-dependency loader in [`src/pdf_runtime.py`](../src/pdf_runtime.py); form extraction and filling live separately in [`src/pdf_forms.py`](../src/pdf_forms.py) and [`src/pdf_form_doc.py`](../src/pdf_form_doc.py).
- **Evidence locations:** [`routes/document_routes.py`](../routes/document_routes.py); [`src/upload_handler.py`](../src/upload_handler.py); [`src/document_processor.py`](../src/document_processor.py); [`src/document_actions.py`](../src/document_actions.py); [`src/personal_docs.py`](../src/personal_docs.py); [`src/rag_manager.py`](../src/rag_manager.py); [`src/embeddings.py`](../src/embeddings.py); [`src/pdf_runtime.py`](../src/pdf_runtime.py); [`src/pdf_forms.py`](../src/pdf_forms.py); [`src/pdf_form_doc.py`](../src/pdf_form_doc.py).
- **Known problems:** PDF viewing/runtime loading and PDF form processing are separate implementations. That separation is confirmed and intentional in the source; it is not a defect without a reported behavioural failure.
- **Open question:** Optional PDF dependencies and representative uploaded documents were not exercised.
## Memory and skills
- **How it works:** Memory routes use [`services/memory/`](../services/memory/) and vector helpers. Skills are exposed through [`routes/skills_routes.py`](../routes/skills_routes.py), stored and managed in [`services/memory/skills.py`](../services/memory/skills.py), and may be imported through [`services/memory/skill_importer.py`](../services/memory/skill_importer.py).
- **Evidence locations:** [`routes/memory/memory_routes.py`](../routes/memory/memory_routes.py); [`services/memory/`](../services/memory/); [`src/memory.py`](../src/memory.py); [`src/memory_vector.py`](../src/memory_vector.py); [`routes/skills_routes.py`](../routes/skills_routes.py); [`services/memory/skills.py`](../services/memory/skills.py); [`services/memory/skill_importer.py`](../services/memory/skill_importer.py).
- **Known problems:** None recorded by this mapping.
- **Open question:** Which imported skill content can reach execution-capable paths, and which validation occurs before that point?
## Email, calendar, contacts, notes, and tasks
- **How it works:** Dedicated route modules own email, CalDAV calendar, CardDAV contacts, notes, and tasks. Supporting modules include email helpers and pollers, CalDAV sync and writeback, and the task scheduler.
- **Evidence locations:** [`routes/email_routes.py`](../routes/email_routes.py); [`routes/calendar_routes.py`](../routes/calendar_routes.py); [`routes/contacts/contacts_routes.py`](../routes/contacts/contacts_routes.py); [`routes/note/note_routes.py`](../routes/note/note_routes.py); [`routes/task_routes.py`](../routes/task_routes.py); [`routes/assistant_routes.py`](../routes/assistant_routes.py); [`src/caldav_sync.py`](../src/caldav_sync.py); [`src/caldav_writeback.py`](../src/caldav_writeback.py); [`src/task_scheduler.py`](../src/task_scheduler.py).
- **Known problems:** None recorded by this mapping.
- **Open question:** External account behaviour, writeback, and delivery require controlled credentials and are not runtime-validated here.
## Media, speech, and image work
- **How it works:** Gallery and image routes coordinate media features. Service modules own speech and media integrations; [`src/generated_images.py`](../src/generated_images.py) and [`src/visual_report.py`](../src/visual_report.py) support artifact handling and presentation.
- **Evidence locations:** [`routes/gallery/gallery_routes.py`](../routes/gallery/gallery_routes.py); [`routes/stt_routes.py`](../routes/stt_routes.py); [`routes/tts_routes.py`](../routes/tts_routes.py); [`src/generated_images.py`](../src/generated_images.py); [`services/stt/`](../services/stt/); [`services/tts/`](../services/tts/); [`services/faces/`](../services/faces/); [`src/visual_report.py`](../src/visual_report.py).
- **Known problems:** None recorded by this mapping.
- **Open question:** Hardware- and provider-dependent media workflows were not exercised.
## Authentication, secrets, and privileged administration
- **How it works:** [`core/auth.py`](../core/auth.py) and [`core/middleware.py`](../core/middleware.py) provide identity and request gates. [`src/secret_storage.py`](../src/secret_storage.py) encrypts application-managed database secrets with a local Fernet key. Vault handling is separate: [`routes/vault_routes.py`](../routes/vault_routes.py) and [`src/tools/vault.py`](../src/tools/vault.py) invoke the Bitwarden CLI and persist its session data in the application data area.
- **Evidence locations:** [`core/auth.py`](../core/auth.py); [`core/middleware.py`](../core/middleware.py); [`routes/auth_routes.py`](../routes/auth_routes.py); [`routes/api_token_routes.py`](../routes/api_token_routes.py); [`src/secret_storage.py`](../src/secret_storage.py); [`routes/vault_routes.py`](../routes/vault_routes.py); [`src/tools/vault.py`](../src/tools/vault.py); [`routes/admin_wipe/admin_wipe_routes.py`](../routes/admin_wipe/admin_wipe_routes.py).
- **Known problems:** Vault-command handling and local application secret storage are distinct paths with different storage mechanisms. This is a source-confirmed boundary, not evidence that either path is compromised.
- **Open question:** What are the current confidentiality, ownership, rotation, and backup semantics for vault session data?
## Persistence, background work, and operations
- **How it works:** SQLite models and persistence are centred in [`core/database.py`](../core/database.py); managers use application data paths. The scheduler and background-job monitor can continue work outside a live browser request. Operational routes cover cleanup, backup, and administrative wipe; the repository also provides a backup script and user documentation.
- **Evidence locations:** [`core/database.py`](../core/database.py); [`src/runtime_paths.py`](../src/runtime_paths.py); [`src/task_scheduler.py`](../src/task_scheduler.py); [`src/bg_jobs.py`](../src/bg_jobs.py); [`src/bg_monitor.py`](../src/bg_monitor.py); [`routes/backup_routes.py`](../routes/backup_routes.py); [`routes/cleanup/cleanup_routes.py`](../routes/cleanup/cleanup_routes.py); [`routes/admin_wipe/admin_wipe_routes.py`](../routes/admin_wipe/admin_wipe_routes.py); [`scripts/odysseus-backup`](../scripts/odysseus-backup); [`docs/backup-restore.md`](../docs/backup-restore.md).
- **Known problems:** None recorded by this mapping.
- **Open question:** What current behaviour applies to background execution, cancellation, retries, and authority inheritance?
docs/ARCHITECTURE.md
-87
View File
@@ -1,87 +0,0 @@
# Architecture
> [!NOTE]
> This document is the proposed canonical destination for stable high-level
> architecture facts. It remains subject to maintainer review. Source code,
> tests, and configuration remain authoritative for implementation-sensitive
> behaviour.
## Purpose
This document identifies the stable runtime boundaries and primary ownership
locations used to navigate and extend Odysseus.
It intentionally excludes generated metrics, file-size rankings, refactor
priorities, unresolved investigation findings, and proposed package layouts.
## Runtime structure
| Area | Responsibility |
|---|---|
| `app.py` | FastAPI application composition and primary application entry point |
| `launcher.py` | Application launch support |
| `setup.py` | Native setup workflow |
| `core/` | Authentication, middleware, persistence, sessions, and platform primitives |
| `routes/` | HTTP and API route handlers |
| `src/` | Application orchestration, tools, providers, and runtime helpers |
| `services/` | Domain-oriented service implementations |
| `mcp_servers/` | Built-in MCP server implementations |
| `scripts/` | CLI tools, diagnostics, maintenance, and migration helpers |
| `static/` | No-build browser frontend and bundled assets |
| `tests/` | Automated tests and supporting test infrastructure |
## Subsystem boundaries
| Subsystem | Primary implementation locations |
|---|---|
| Application startup | `app.py`, `src/app_initializer.py`, `core/` |
| Authentication and sessions | `core/auth.py`, `core/middleware.py`, `core/session_manager.py`, `routes/auth_routes.py` |
| Chat and streaming | `routes/chat_routes.py`, `routes/chat_helpers.py`, `src/chat_handler.py`, `src/chat_processor.py`, `src/llm_core.py` |
| Agents and tools | `src/agent_loop.py`, `src/tool_execution.py`, `src/agent_tools/`, `src/tools/`, `src/tool_policy.py`, `src/tool_security.py` |
| Models and providers | `routes/model_routes.py`, `src/model_discovery.py`, `src/model_capabilities.py`, `src/endpoint_resolver.py`, `src/llm_core.py` |
| Cookbook and hardware fit | `routes/cookbook_routes.py`, `routes/cookbook_helpers.py`, `src/cookbook_serve_lifecycle.py`, `services/hwfit/` |
| Search and research | `routes/search_routes.py`, `services/search/`, `routes/research/`, `services/research/`, `src/deep_research.py` |
| Documents and retrieval | `routes/document_routes.py`, `src/document_processor.py`, `src/personal_docs.py`, `src/rag_manager.py`, `src/pdf_runtime.py` |
| Memory and skills | `routes/memory/`, `services/memory/`, `routes/skills_routes.py` |
| Email | `routes/email_routes.py`, `routes/email_helpers.py`, `routes/email_pollers.py`, `mcp_servers/email_server.py` |
| Calendar, contacts, notes, and tasks | `routes/calendar_routes.py`, `routes/contacts/`, `routes/note/`, `routes/task_routes.py`, `src/task_scheduler.py` |
| Media and speech | `routes/gallery/`, `routes/stt_routes.py`, `routes/tts_routes.py`, `services/stt/`, `services/tts/` |
| Persistence and operations | `core/database.py`, `src/runtime_paths.py`, `src/bg_jobs.py`, `routes/backup_routes.py`, `routes/cleanup/` |
## Architectural constraints
- Preserve established import and compatibility paths unless a focused change
explicitly migrates them.
- Keep HTTP concerns in route modules and reusable domain behaviour in runtime
or service modules.
- Treat shared persistence, agent orchestration, tool execution, and application
startup as high-authority boundaries.
- Change one ownership boundary at a time.
- Do not mix structural movement with unrelated feature behaviour.
- Validate affected imports, startup paths, compatibility surfaces, and tests.
## Frontend
The browser frontend is a no-build ES-module application under `static/`.
Its maintained module-level structure is documented in
[`static/js/MODULE_SUMMARY.md`](../static/js/MODULE_SUMMARY.md).
## Investigation and snapshots
Non-canonical investigation material is maintained under [`discovery/`](../discovery/).
The following documents may contain dated observations, metrics, unresolved
questions, or historical planning and must not be treated as specifications:
- [`discovery/system-map.md`](../discovery/system-map.md)
- [`discovery/architecture-runtime-inventory.md`](../discovery/architecture-runtime-inventory.md)
## Documentation authority
- Code, tests, and configuration define implemented behaviour.
- Mature specifications define accepted subsystem behaviour where they exist.
- Following maintainer acceptance, this document will define the high-level
architecture map.
- Discovery documents preserve evidence and uncertainty but remain
non-canonical.
docs/STYLE.md
-72
View File
@@ -1,72 +0,0 @@
# Documentation style
This guide defines the shared structure and writing conventions for Odysseus documentation.
## Principles
- Write for a clear audience and purpose.
- State whether a document is canonical, informational, a snapshot, or planning material.
- Prefer current behaviour over historical explanation.
- Link to source files, tests, issues, or other documentation when useful.
- Separate verified behaviour from assumptions, open questions, and future work.
- Keep headings descriptive and consistent.
- Use Markdown callouts where status or risk must be visible.
- Do not use emojis.
## Document status
Use a status callout near the top when the document is not normal canonical guidance.
### Canonical documentation
> [!IMPORTANT]
> This document describes accepted current behaviour. Verify implementation-sensitive details against the current code and tests.
### Discovery material
> [!NOTE]
> This is non-canonical discovery material. It records code-grounded observations and open questions.
### Snapshot or inventory
> [!WARNING]
> This document is a dated snapshot. Counts, paths, and implementation details may drift as the codebase changes.
### Planning material
> [!NOTE]
> This document records planning context. It does not define current runtime behaviour or guarantee future implementation.
## Recommended structure
Use the following sections where relevant:
1. Title
2. Purpose or status callout
3. Scope
4. Current behaviour or guidance
5. Safety, limitations, or known gaps
6. Validation or evidence
7. Related documentation
Not every document needs every section.
## Writing style
- Use concise sentences.
- Prefer direct language.
- Avoid jokes, filler, and informal warnings.
- Avoid repeating the same guidance across several files.
- Link to the owning document instead of duplicating large sections.
- Use lists for procedures, requirements, and comparisons.
- Use tables only when they improve scanning.
- Use fenced code blocks with an appropriate language identifier.
- Use relative repository links for internal files.
## Authority
The current code, tests, and configuration are the source of truth for implemented behaviour.
Canonical documentation describes accepted behaviour and supported workflows.
Discovery, inventory, and planning documents must identify themselves explicitly and must not silently become behavioural specifications.
docs/attachments.md
+2 -34
View File
@@ -1,13 +1,7 @@
# Attachment References and Upload Storage
> [!NOTE]
> This document records the current attachment-reference and upload-lifecycle
> contract proposed for maintainer acceptance. Source code, tests, and
> configuration remain authoritative for implementation-sensitive behaviour.
Odysseus stores chat and document attachment bytes under the configured upload
directory and passes stable references through chat history, document flows, and
tool context.
Odysseus stores uploaded bytes once under the configured upload directory and
passes stable references through chat history, tools, and future artifact work.
The goal is to avoid duplicating large inline media payloads in
`chat_messages.content` or the SQLite FTS index.
@@ -60,32 +54,6 @@ External MCP/custom tools should treat the URI and attachment ID as the stable
contract and request bytes through an owner-checked server path, not by assuming
host filesystem layout.
## Implementation evidence
The current contract is implemented primarily through:
- `src/upload_handler.py` for upload metadata, owner-aware resolution,
reservations, cleanup, and deletion;
- `src/attachment_refs.py` for compact persisted references and search-index
sanitization;
- `src/document_processor.py` for resolving attachments into chat/model context;
- `src/tool_execution.py` for attachment manifests exposed to tools;
- `routes/upload_routes.py` and `routes/document_helpers.py` for upload and
retrieval paths.
Focused regression coverage includes:
- `tests/test_attachment_refs.py`;
- `tests/test_upload_handler_cleanup.py`;
- `tests/test_replace_messages_upload_reservations.py`;
- `tests/test_resolve_upload_path_nondict.py`;
- `tests/test_chat_preprocess_tool_policy.py`;
- the upload, attachment, and PDF-marker cases in
`tests/test_security_regressions.py`.
These tests cover compact persistence, owner isolation, path containment,
cleanup safety, reservation-before-write behaviour, and traversal resistance.
## Retention and Deletion
Current retention behavior is conservative:
docs/pr-blocker-audit.md
+2 -2
View File
@@ -181,8 +181,8 @@ Dirty, blocked, conflicting, and unknown merge states are shown as risk/caution
## Validation
```bash
venv/bin/python -m py_compile scripts/pr_blocker_audit.py tests/test_pr_blocker_audit.py
venv/bin/python -m pytest tests/test_pr_blocker_audit.py -q
python3 -m py_compile scripts/pr_blocker_audit.py tests/test_pr_blocker_audit.py
python3 -m pytest tests/test_pr_blocker_audit.py -q
python3 scripts/pr_blocker_audit.py --help
git diff --check
```
docs/setup.md
+16 -35
View File
@@ -1,28 +1,6 @@
# Odysseus Setup Guide
This guide covers installation, deployment, troubleshooting, and configuration.
For a minimal Docker installation, start with the
[repository README](../README.md#quick-start).
## On this page
- [Quick Start](#quick-start)
- [Docker](#docker-recommended)
- [Native Linux and macOS](#native-linux--macos)
- [Apple Silicon](#apple-silicon)
- [Native Windows](#native-windows)
- [Troubleshooting and advanced setup](#troubleshooting--advanced-setup)
- [Security notes](#security-notes)
- [Configuration](#configuration)
- [Architecture](#architecture)
- [Data and backups](#data)
Related guidance:
- [Security policy](../SECURITY.md)
- [Architecture overview](ARCHITECTURE.md)
- [Backup and restore guide](backup-restore.md)
- [Contributing guide](../CONTRIBUTING.md)
This page keeps the detailed install, deployment, troubleshooting, and configuration notes out of the front README.
## Quick Start
@@ -37,8 +15,8 @@ On first setup, Odysseus creates an admin account (`admin` unless
For Docker installs, the same line is in `docker compose logs odysseus`.
Use that for the first login, then change it in **Settings**.
Contributing? See the [contributing guide](../CONTRIBUTING.md) for development
setup, testing, and pull request guidelines.
Contributing? See [CONTRIBUTING.md](../CONTRIBUTING.md) for setup, testing, and
pull request guidelines.
### Docker (recommended)
```bash
@@ -463,8 +441,8 @@ uv pip sync requirements.lock # reproduce it exactly la
### Outlook / Office 365 email
Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
passwords will fail. See the [Outlook and Microsoft 365 email guide](email-outlook.md)
for the current limitation and planned integration direction.
passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
current limitation and the planned integration direction.
## Security Notes
Odysseus is a self-hosted workspace with powerful local tools: shell access, file uploads, model downloads, web research, email/calendar integrations, and API tokens. Treat it like an admin console.
@@ -552,16 +530,19 @@ npx -y @playwright/mcp@latest --version
That installs `@playwright/mcp` plus Playwright (~300MB total). Restart Odysseus and the server will register at startup.
## Architecture
For stable high-level runtime structure, subsystem boundaries, and documentation
authority, see the [architecture overview](ARCHITECTURE.md).
Source code, tests, and configuration remain authoritative for
implementation-sensitive behaviour.
```
app.py # FastAPI entry point
core/ auth, database, middleware, constants
src/ llm_core, agent_loop, agent_tools, chat_processor, search/
routes/ chat, session, document, memory, model … endpoints
services/ docs, memory, search, hwfit (Cookbook) …
static/ index.html + app.js + style.css + js/ (modular front-end)
docs/ landing page (index.html) + preview clips
```
## Data
All user data lives in `data/` (gitignored): `app.db` (sessions, messages, documents),
`memory.json`, `presets.json`, `uploads/`, `personal_docs/`, `chroma/`, `settings.json`.
To protect or recover this data, follow the
[backup and restore guide](backup-restore.md).
To back up or restore everything in `data/`, see the
[Backup & Restore guide](backup-restore.md).
routes/personal_routes.py
+61 -18
View File
@@ -1,11 +1,13 @@
# routes/personal_routes.py
"""Routes for personal documents management."""
import asyncio
import os
import logging
import shutil
import uuid
from typing import Any, Dict, List, Tuple
from fastapi import APIRouter, HTTPException, Query, Request, UploadFile, File, Depends
from fastapi.concurrency import run_in_threadpool
from src.request_models import DirectoryRequest
from core.constants import BASE_DIR, PERSONAL_DIR, PERSONAL_UPLOADS_DIR
from src.rag_singleton import get_rag_manager
@@ -18,7 +20,6 @@ UPLOADS_DIR = PERSONAL_UPLOADS_DIR
logger = logging.getLogger(__name__)
def _personal_upload_dir_for_owner(owner: str | None, *, create: bool = True) -> str:
"""Return the per-owner upload directory used for direct RAG uploads."""
owner_segment = secure_filename((owner or "local").strip())[:80] or "local"
@@ -141,6 +142,22 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
"""
router = APIRouter(prefix="/api/personal")
# Serializes directory index jobs across requests. Indexing runs in the
# threadpool (#5558), so concurrent requests would otherwise run in parallel
# and race PersonalDocsManager's unsynchronized list mutations and file
# writes; before the threadpool move they serialized on the blocked event
# loop, so one-at-a-time is behavior parity.
#
# An asyncio.Lock acquired in the async handler BEFORE offloading: a waiting
# request parks on the event loop instead of pinning a threadpool worker (an
# earlier threading.Lock taken INSIDE the worker meant queued jobs held pool
# tokens while blocked, starving every other run_in_threadpool caller).
# add/remove/reload all take this lock, so their mutations never interleave.
# Per-router (not module-global) so each app binds it to its own event loop.
# Scope is the single process: multi-worker deployments would need a shared
# lock (out of scope for #5558).
_index_job_lock = asyncio.Lock()
def _rag():
"""Get the current RAG manager, retrying init if needed."""
return get_rag_manager()
@@ -172,8 +189,12 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
return {"files": files, "directories": directories}
@router.post("/reload")
def api_personal_reload(owner: str = Depends(require_user), _admin: None = Depends(require_admin)):
personal_docs_manager.refresh_index()
async def api_personal_reload(owner: str = Depends(require_user), _admin: None = Depends(require_admin)):
# refresh_index() re-extracts text across every tracked directory —
# blocking work. Take the shared job lock (so it cannot race an add /
# remove) and run it off the event loop.
async with _index_job_lock:
await run_in_threadpool(personal_docs_manager.refresh_index)
return {"ok": True, "count": len(personal_docs_manager.index)}
@router.post("/add_directory")
@@ -207,12 +228,26 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
# Use the RAGManager to index the directory
rag = _rag()
if rag:
result = rag.index_personal_documents(directory, owner=owner)
def _index_directory():
result = rag.index_personal_documents(directory, owner=owner)
if result["success"]:
# Also update the personal_docs_manager to track this
# directory. Kept inside the offloaded call: it triggers
# refresh_index(), which re-extracts text across tracked
# directories.
personal_docs_manager.add_directory(directory, index=False)
return result
# Indexing walks, embeds, and stores the whole tree — minutes
# on a real directory. The handler is async, so calling it
# inline runs it on the event loop and every other request
# queues behind it until it finishes (#5558). Serialize on the
# async job lock BEFORE offloading so a queued request parks on
# the loop instead of pinning a threadpool worker.
async with _index_job_lock:
result = await run_in_threadpool(_index_directory)
if result["success"]:
# Also update the personal_docs_manager to track this directory
personal_docs_manager.add_directory(directory, index=False)
return {
"success": True,
"message": f"Successfully indexed {result['indexed_count']} chunks from {directory}",
@@ -251,17 +286,25 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
logger.info(f"Removing directory from RAG: {directory}")
# Always remove from personal_docs_manager tracking
if hasattr(personal_docs_manager, 'remove_directory'):
personal_docs_manager.remove_directory(directory)
# Remove from RAG vector store (best-effort)
rag = _rag()
if rag:
try:
rag.remove_directory(directory)
except Exception as e:
logger.warning(f"RAG removal failed for directory {directory}: {e}")
def _remove_directory():
# Always remove from personal_docs_manager tracking. This
# mutates the same unsynchronized list/index an add job touches
# and re-extracts text (refresh_index), so it is blocking work.
if hasattr(personal_docs_manager, 'remove_directory'):
personal_docs_manager.remove_directory(directory)
# Remove from RAG vector store (best-effort).
if rag:
try:
rag.remove_directory(directory)
except Exception as e:
logger.warning(f"RAG removal failed for directory {directory}: {e}")
# Same job lock as add/reload so remove cannot interleave with an
# in-flight add; offloaded off the event loop.
async with _index_job_lock:
await run_in_threadpool(_remove_directory)
return {
"success": True,
specs/architecture-runtime-inventory.md
+406 -25
View File
@@ -1,31 +1,412 @@
# Architecture runtime inventory disposition
# Architecture Runtime Inventory
> [!NOTE]
> The dated architecture runtime inventory has been classified and transferred
> to [`discovery/architecture-runtime-inventory.md`](../discovery/architecture-runtime-inventory.md)
> after verification of its stable facts.
>
> Stable runtime structure and subsystem boundaries were transferred to the
> proposed canonical destination,
> [`docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md), for maintainer review.
> **Purpose**: Phase 0 planning baseline for codebase readability improvements (#4071).
> **Parent issue**: [#4082](https://github.com/odysseus-dev/odysseus/issues/4082)
> **Last updated**: dev@b58af42 | 2026-06-16
> **Status**: Draft — to be reviewed before follow-up slices open.
> **Snapshot basis**: Importer / file / import-line counts are refreshed to `dev@b58af42` (2026-06-16) and are recomputable via the commands in §3.4. **Line counts** in §2.1 / §2.2 are a snapshot from an earlier baseline and drift as `dev` moves — recompute any of them with `wc -l <file>`. This inventory tracks structure and risk, not live metrics.
## Classification
This document maps the current runtime module structure, identifies high-risk boundaries, and recommends safe first refactor slices. It does **not** move files, change imports, or alter runtime behavior.
| Content | Disposition |
|---|---|
| Stable runtime structure | Proposed canonical architecture document |
| Stable subsystem boundaries | Proposed canonical architecture document |
| Frontend module organization | `static/js/MODULE_SUMMARY.md` |
| Counts, rankings, and generated metrics | Non-canonical discovery inventory |
| Investigation context and unresolved questions | Non-canonical discovery material |
| Refactor options and prioritization | Issues, Plane, or discovery material |
---
No dated metrics, prioritization, or investigation findings remain in this
specification path.
## 1. Current Structure Overview
## Review status
### 1.1 Top-Level Layout
- Findings and scope have been validated.
- Transfer verification has completed.
- The final canonical status of `docs/ARCHITECTURE.md` requires maintainer review.
- No implementation behaviour, test taxonomy, or runtime files were changed.
```
odysseus/
├── app.py # FastAPI app entrypoint (1,145 lines)
├── conf/ # Configuration (config.py, settings.py, settings_scrub.py)
├── src/ # 95 flat .py files + 2 subdirectories
│ ├── agent_tools/ # Tool helpers: document, filesystem, subprocess, web
│ └── search/ # Search subsystem
├── routes/ # 54 flat .py files — HTTP route handlers
├── core/ # 10 files — database models, auth, middleware, session
├── mcp_servers/ # 5 files — MCP server implementations
├── scripts/ # CLI tools and one-shot scripts
├── static/ # Frontend HTML/CSS/JS
├── tests/ # 583 test files (~54,800 lines)
└── services/ # (exists as needed)
```
### 1.2 Directory Flatness Metric
| Directory | Flat `.py` Files | Subdirectories | Concern |
|-----------|-----------------|----------------|---------|
| `src/` | **95** | 2 (`agent_tools/`, `search/`) | No domain grouping; 95 files in one directory |
| `routes/` | **54** | 0 | All route handlers in one flat directory |
| `core/` | 10 | 0 | Manageable, but `database.py` is oversized |
---
## 2. Largest Runtime Modules
### 2.1 Python Backend
| Rank | File | Lines | Classes | Functions | Risk |
|------|------|-------|---------|-----------|------|
| 1 | `src/tool_implementations.py` | **4,032** | 0 | ~48 | **HIGH** |
| 2 | `routes/email_routes.py` | **3,245** | — | — | **MEDIUM** |
| 3 | `routes/cookbook_routes.py` | **2,969** | — | — | **MEDIUM** |
| 4 | `src/agent_loop.py` | **2,961** | 0 | ~24 | **HIGH** |
| 5 | `src/task_scheduler.py` | **2,330** | — | 5 | MEDIUM |
| 6 | `routes/model_routes.py` | **2,266** | — | — | MEDIUM |
| 7 | `core/database.py` | **2,265** | 28 | ~59 helpers | **HIGH** |
| 8 | `src/builtin_actions.py` | **2,262** | 2 | ~24 | MEDIUM |
| 9 | `src/llm_core.py` | **2,164** | — | — | MEDIUM |
| 10 | `mcp_servers/email_server.py` | 2,197 | — | — | LOW (separate process) |
| 11 | `src/visual_report.py` | 1,918 | — | — | LOW |
| 12 | `routes/gallery_routes.py` | 1,896 | — | — | LOW |
| 13 | `src/ai_interaction.py` | 1,846 | — | — | MEDIUM |
| 14 | `routes/document_routes.py` | 1,717 | — | — | LOW |
| 15 | `routes/skills_routes.py` | 1,648 | — | — | LOW |
**Heuristic**: Files > 2,000 lines with 20+ public symbols and many importers are the highest-risk splits. Files 1,000–2,000 lines are medium-risk if tightly coupled.
### 2.2 Frontend
| File | Lines | Concern |
|------|-------|---------|
| `static/style.css` | **36,653** | Entire app CSS in one file (tracked separately in #2617) |
| `static/js/document.js` | **9,776** | Single JS file for document functionality |
| `static/js/slashCommands.js` | 6,498 | |
| `static/js/settings.js` | 5,266 | |
| `static/js/emailLibrary.js` | 5,217 | |
| `static/js/notes.js` | 5,124 | |
| `static/js/chat.js` | 4,985 | |
| `static/app.js` | 4,090 | |
**Note**: Frontend modularization is tracked separately in #2617 (CSS) and is not the focus of this Phase 0 inventory. Frontend is listed here for completeness but follow-up slices should target Python backend boundaries first.
---
## 3. Import Dependency Graph
### 3.1 Who Depends on `core/database.py`
**102 files** import from `core.database` — this is the most depended-upon module:
- All route handlers (`routes/*.py`)
- Most `src/*.py` files
- `core/session_manager.py`, `core/auth.py`
- Multiple test files
**Implication**: Any split of `core/database.py` is the highest-risk refactor. It should be tackled **last**, never first.
### 3.2 Who Depends on `src/tool_implementations.py`
**17 files** import from `src.tool_implementations`:
- `src/agent_loop.py`, `src/builtin_actions.py`, `src/tool_index.py`
- `src/task_scheduler.py`, `src/tool_policy.py`
- Various tests
### 3.3 Who Depends on `src/agent_loop.py`
**22 files** import from `src.agent_loop`:
- `src/tool_policy.py`, `src/teacher_escalation.py`, `src/bg_monitor.py`
- `src/task_scheduler.py`
- Multiple test files
### 3.4 Cross-Layer Import Violations
**`src/` importing from `routes/`** (backwards dependency — domain logic depending on HTTP layer):
```
src/tool_implementations.py ──→ routes/calendar_routes.py
src/tool_implementations.py ──→ routes/cookbook_helpers.py
src/tool_implementations.py ──→ routes/email_helpers.py
src/tool_implementations.py ──→ routes/email_pollers.py
src/tool_implementations.py ──→ routes/email_routes.py
src/tool_implementations.py ──→ routes/model_routes.py
src/tool_implementations.py ──→ routes/note_routes.py
src/tool_implementations.py ──→ routes/prefs_routes.py
```
> These are **runtime imports** (inside function bodies, not at module top), which mitigates circular import risk but indicates fuzzy layer boundaries. Function-level inline imports from the HTTP layer into business logic are a code smell.
**Import counts (top-level)**:
| Direction | Count | Notes |
|-----------|-------|-------|
| `routes/` → `src/` | **374** | Expected: HTTP handlers call domain logic |
| `routes/` → `core/` | **126** | Expected: handlers access DB models |
| `src/` → `routes/` | **31** | **Unexpected**: domain logic reaching into HTTP layer (direct grep of import lines referencing `routes/`) |
| `src/` → `core/` | **106** | Acceptable but could be reduced with a data-access layer |
> **How the metrics in this document are computed** — recompute against current `dev` before treating any count as authoritative (the tree drifts; these numbers are a snapshot, not a live value):
> - `src/` flat `.py` files: `find src -maxdepth 1 -name '*.py' | wc -l`
> - `tests/` test files: `find tests -name 'test_*.py' | wc -l`
> - `core.database` importers: `grep -rlE '(from|import) +core\.database' --include='*.py' . | grep -v core/database.py | wc -l`
> - `src.agent_loop` importers: `grep -rlE '(from|import) +src\.agent_loop' --include='*.py' . | grep -v src/agent_loop.py | wc -l`
> - Cross-layer import lines: `grep -rhE '(from|import) +<pkg>' --include='*.py' <dir>/ | wc -l` (e.g. `(from|import) +routes` over `src/`)
---
## 4. Route Ownership Map
Routes can be grouped into logical feature domains. Current flat structure obscures these boundaries:
| Domain | Route Files | Total Lines | Review Complexity |
|--------|-------------|-------------|-------------------|
| **Email** | `email_routes.py`, `email_helpers.py`, `email_pollers.py` | 5,936 | HIGH — most complex domain |
| **Chat / Agent** | `chat_routes.py`, `chat_helpers.py`, `shell_routes.py`, `codex_routes.py`, `skills_routes.py` | 6,365 | HIGH — core interaction surface |
| **Cookbook** | `cookbook_routes.py`, `cookbook_helpers.py`, `cookbook_output.py` | 4,110 | MEDIUM |
| **Model / LLM** | `model_routes.py`, `assistant_routes.py`, `copilot_routes.py` | 2,764 | MEDIUM |
| **Calendar / Contacts** | `calendar_routes.py`, `contacts_routes.py` | 2,336 | MEDIUM |
| **Documents** | `document_routes.py`, `document_helpers.py` | 1,954 | LOW |
| **Auth** | `auth_routes.py`, `api_token_routes.py`, `device_flow.py` | 1,171 | LOW |
| **Tasks** | `task_routes.py` (standalone) | 1,157 | LOW |
| **Session** | `session_routes.py` (standalone) | 1,287 | LOW |
| **Gallery** | `gallery_routes.py`, `gallery_helpers.py` | 1,896 | LOW |
| **Memory** | `memory_routes.py` | — | LOW |
| **Research** | `research_routes.py` | — | LOW |
| **MCP** | `mcp_routes.py` | — | LOW |
| **Notes** | `note_routes.py` | — | LOW |
| **Other** | `prefs_routes.py`, `upload_routes.py`, `vault_routes.py`, `webhook_routes.py`, `workspace_routes.py`, `search_routes.py`, `history_routes.py`, `hwfit_routes.py`, `preset_routes.py`, `signature_routes.py`, `backup_routes.py`, `cleanup_routes.py`, `diagnostics_routes.py`, `embedding_routes.py`, `emoji_routes.py`, `font_routes.py`, `stt_routes.py`, `tts_routes.py`, `compare_routes.py`, `personal_routes.py`, `editor_draft_routes.py`, `admin_wipe_routes.py`, `chatgpt_subscription_routes.py` | 2,000+ | LOW individual, HIGH cumulative |
---
## 5. Tool Registry & Implementation Boundaries
### 5.1 Current Tool Architecture
| Component | File | Lines | Role |
|-----------|------|-------|------|
| Tool schemas | `src/tool_schemas.py` | 1,392 | JSON Schema tool definitions (Duck-TypedDict) |
| Tool index | `src/tool_index.py` | 542 | RAG-based tool retrieval from ChromaDB |
| Tool implementations | `src/tool_implementations.py` | 4,032 | 33 `do_*` functions — all tool execution logic |
| Tool security | `src/tool_security.py` | — | Owner-scoped tool blocking |
| Tool policy | `src/tool_policy.py` | — | Guide-only directive, plan-mode disabled tools |
| Tool utils | `src/tool_utils.py` | — | Shared tool helpers |
### 5.2 Tool Implementation Categories
The 33 `do_*` functions in `tool_implementations.py` fall into natural domain groups — the basis for slice 1's split in §6.2:
| Category | `do_*` functions | Count |
|----------|------------------|-------|
| **System / config** | `do_manage_skills`, `do_manage_tasks`, `do_manage_endpoints`, `do_manage_mcp`, `do_manage_webhooks`, `do_manage_tokens`, `do_manage_settings`, `do_api_call`, `do_app_api` | 9 |
| **Cookbook / model serving** | `do_download_model`, `do_serve_model`, `do_list_served_models`, `do_stop_served_model`, `do_tail_serve_output`, `do_list_downloads`, `do_cancel_download`, `do_search_hf_models`, `do_adopt_served_model`, `do_list_cookbook_servers`, `do_list_serve_presets`, `do_serve_preset`, `do_list_cached_models` | 13 |
| **Notes** | `do_manage_notes` | 1 |
| **Calendar** | `do_manage_calendar` | 1 |
| **Search** | `do_search_chats` | 1 |
| **Research** | `do_manage_research`, `do_trigger_research` | 2 |
| **Contacts** | `do_resolve_contact`, `do_manage_contact` | 2 |
| **Vault** | `do_vault_search`, `do_vault_get`, `do_vault_unlock` | 3 |
| **Image** | `do_edit_image` | 1 |
| | **Total** | **33** |
> Low-level tools (filesystem, subprocess, web fetch, document parsing) live in `src/agent_tools/`, **not** in `tool_implementations.py` — out of scope for this split.
---
## 6. Risk Assessment & Candidate Slice Ranking
> **Candidate proposals, not a committed plan.** The rankings, package shapes (e.g. `src/pkg/`, `src/domain/`, `src/infra/`, `src/api/`), split ordering, and route-grouping strategy below are **options for maintainer discussion**. Per #4082/#4071, slice ownership and order are settled by maintainers before any follow-up PR. §1–§3 above are the factual current-state inventory.
### 6.1 Risk Scale
| Level | Criteria |
|-------|----------|
| **LOW** | File has ≤3 importers AND ≤500 lines, OR is a pure refactor with clear boundaries |
| **MEDIUM** | File has 4–15 importers OR 500–1,500 lines |
| **HIGH** | File has 16+ importers OR >2,000 lines, OR has cross-layer import violations |
### 6.2 Ranked Split Candidates
| Priority | Target | Risk | Rationale |
|----------|--------|------|-----------|
| **1** | `src/tool_implementations.py` → `src/tools/*.py` | **MEDIUM** | 4,032 lines → ~10 files by tool category. Already has natural boundaries. 17 importers, tracked in #3629. Use `__init__.py` shim to keep existing imports working. |
| **2** | `routes/` → domain subdirectories (one domain per PR) | **MEDIUM** | 54 flat files. Done **one domain at a time** (e.g. a standalone PR for the email domain, then chat, …), not a broad reorganization — route modules carry helper imports, registration assumptions, and test import paths. |
| **3** | `src/agent_loop.py` → `src/agent/loop.py` + submodules | **MEDIUM-HIGH** | 2,961 lines, 24 functions. Can extract prompt building, classification, verification, and runaway detection. Tracked in #3266. |
| **4** | `src/` → `src/pkg/`, `src/domain/`, `src/infra/`, `src/api/` | **MEDIUM** | Structural reorganization. Split flat `src/` into layered packages. Must come after routes and tools are stable. |
| **5** | `routes/email_*.py` consolidation | **LOW** | Already grouped by filename prefix. Low-risk cleanup within the email domain. |
| **6** | `core/database.py` → `src/infra/database/models/*.py` | **HIGH** | 28 classes, 102 importers. Highest-risk split. Must be **last** in any sequence. Requires careful import shim strategy. |
| **7** | Frontend CSS modularization | **MEDIUM** | 36,653 lines. Tracked in #2617. Separate timeline from backend work. |
| **8** | Frontend JS modularization | **MEDIUM** | 9,776 lines in `document.js`. Introduce ES modules at minimum. |
### 6.3 Candidate First 3 Behavior-Preserving Slices
**Slice 1: Split `tool_implementations.py`** (Lowest-risk high-impact)
- Create `src/tools/` package with one file per tool category
- Add `src/tools/__init__.py` re-exporting all symbols with current names
- Update 17 importers to use new paths (can be deferred via shim)
- Validation: `python -m pytest tests/ -x -q` + manual smoke test of tool execution
- Reference: #3629
**Slice 2: Group `routes/` by domain** (one domain per PR, not a broad sweep)
Route modules carry helper imports, router registration assumptions, and test import paths, so this must be done **one domain at a time** rather than as a single reorganization PR. Example sequence (each its own PR):
- PR 2a: move the **email** domain (`email_routes.py`, `email_helpers.py`, `email_pollers.py`) → `routes/email/` + shim
- PR 2b: move the **chat/agent** domain → `routes/chat/` + shim
- PR 2c: move the **cookbook** domain → `routes/cookbook/` + shim
- …and so on per domain from §4
Each PR: add `__init__.py` re-exporting old names, update `app.py` router imports, validation `python app.py` starts clean. **No behavior change** — pure file reorganization.
**Slice 3: Extract `agent_loop.py` submodules** (Improve reviewability)
- Move prompt assembly → `src/agent/prompt.py`
- Move request classification → `src/agent/classifier.py`
- Move sub-agent verification → `src/agent/verifier.py`
- Move runaway detection → `src/agent/runaway.py`
- Move context management → `src/agent/context.py`
- Keep `src/agent/loop.py` as the main orchestration module
- Validation: `python -m pytest tests/test_agent_loop.py tests/test_loop_breaker_runaway.py -v`
---
## 7. Safety Guardrails for Follow-Up Work
Per maintainer guidance in #4082 and #4071:
- [ ] **One domain/slice per PR** — never mix multiple reorganizations
- [ ] **No behavior changes** mixed with file moves — pure reorganization only
- [ ] **Keep compatibility shims** — `__init__.py` re-exports for all existing import paths
- [ ] **Add or identify focused tests** before risky splits
- [ ] **Do not start with `core/database.py`** or broad route movement unless this inventory shows a safe boundary
- [ ] **Prefer small, reviewable slices** over large restructures
- [ ] **No packaging/runtime/tooling migration** mixed into file moves
- [ ] **No frontend framework migration** inside this stabilization lane
- [ ] **Validate with `python -m compileall`** — every PR must pass CI checks
- [ ] **Validate with `pytest`** — run the full test suite before opening each PR
---
## 8. Validation Commands
Each follow-up PR should be verifiable with these commands before submission:
```bash
# Syntax check — must pass with zero errors
python -m compileall src/ routes/ core/ conf/
# Full test suite — must match baseline pass rate
python -m pytest tests/ -x -q
# Import shim verification — existing import paths must still work
python -c "from src.tool_implementations import do_search_chats; print('OK')"
# App startup smoke test (if backend touched)
timeout 5 python app.py 2>&1 | head -5 || true
```
---
## 9. Open Questions
1. Is `#2538` (specs ground truth) the canonical behavior map baseline, and should this inventory be kept in sync with those specs once merged?
2. Should route grouping follow the domain map proposed here, or is there a different taxonomy preferred by maintainers?
3. For the `tool_implementations.py` split (#3629), is the tool categorization in §5.2 acceptable, or should it follow a different grouping?
4. Should compatibility shims (`__init__.py`) be temporary (removed in a follow-up wave) or permanent?
5. Should an ADR (Architecture Decision Record) document be started to track decisions made during this process?
---
## 10. Future Direction (NOT current state)
The following are **future refactor targets** (candidate directions **pending maintainer agreement**, not committed), recorded here so this inventory does not imply they exist today. None of them are present in the current `dev` tree:
- `main.py` — proposed rename of the `app.py` entrypoint. Today the app boots via `app.py`.
- `src/agent/` — proposed package to hold `agent_loop.py` submodules (prompt/classifier/verifier/runaway/context). Today `agent_loop.py` is a single flat file in `src/`.
- `src/infra/`, `src/domain/`, `src/pkg/`, `src/api/` — proposed layered reorganization of the flat `src/` directory (slice 4 in §6).
These become real only when the corresponding slices land.
---
## Appendix A: File Listing
### `src/` (95 files — 61 shown; run `ls src/*.py` for the full list)
```
agent_loop.py tool_implementations.py tool_schemas.py
tool_index.py tool_security.py tool_policy.py
tool_utils.py builtin_actions.py task_scheduler.py
llm_core.py model_context.py model_discovery.py
session_search.py context_budget.py context_compactor.py
ai_interaction.py action_intents.py agent_runs.py
app_helpers.py app_initializer.py config.py
database.py memory.py memory_provider.py
secret_storage.py prompt_security.py url_security.py
url_safety.py rate_limiter.py cleanup_service.py
readiness.py service_health.py exceptions.py
request_models.py assistant_log.py bg_monitor.py
builtin_mcp.py chat_helpers.py chroma_client.py
document_processor.py embedding_lanes.py deep_research.py
research_handler.py research_utils.py personal_docs.py
rag_manager.py rag_singleton.py topic_analyzer.py
visual_report.py youtube_handler.py pdf_forms.py
pdf_form_doc.py pdf_runtime.py caldav_writeback.py
email_thread_parser.py text_helpers.py user_time.py
teacher_escalation.py cookbook_serve_lifecycle.py
chatgpt_subscription.py mcp_manager.py
```
### `routes/` (54 files)
```
__init__.py _validators.py
auth_routes.py api_token_routes.py device_flow.py
chat_routes.py chat_helpers.py shell_routes.py
codex_routes.py skills_routes.py
email_routes.py email_helpers.py email_pollers.py
cookbook_routes.py cookbook_helpers.py cookbook_output.py
model_routes.py assistant_routes.py copilot_routes.py
calendar_routes.py contacts_routes.py
document_routes.py document_helpers.py
gallery_routes.py gallery_helpers.py
task_routes.py session_routes.py
note_routes.py memory_routes.py research_routes.py
mcp_routes.py search_routes.py history_routes.py
webhook_routes.py workspace_routes.py upload_routes.py
vault_routes.py prefs_routes.py preset_routes.py
signature_routes.py personal_routes.py hwfit_routes.py
backup_routes.py cleanup_routes.py diagnostics_routes.py
embedding_routes.py emoji_routes.py font_routes.py
stt_routes.py tts_routes.py compare_routes.py
editor_draft_routes.py chatgpt_subscription_routes.py admin_wipe_routes.py
```
### `core/` (10 files)
```
__init__.py constants.py database.py models.py
auth.py middleware.py session_manager.py exceptions.py
atomic_io.py platform_compat.py
```
---
## Appendix B: Key Import Relationships
```
core/database.py ←── 102 importers (routes/*, src/*, core/*, tests/*)
↑
├── routes/auth_routes.py
├── routes/email_routes.py
├── src/builtin_actions.py
├── src/task_scheduler.py
├── src/tool_implementations.py (inline)
└── ...97 more
src/tool_implementations.py ←── 17 importers
↑
├── src/agent_loop.py
├── src/builtin_actions.py
├── src/tool_index.py
├── src/task_scheduler.py
├── src/tool_policy.py
└── ...12 more (mostly tests)
src/agent_loop.py ←── 22 importers
↑
├── src/tool_policy.py
├── src/teacher_escalation.py
├── src/bg_monitor.py
├── src/task_scheduler.py
└── 18 more (incl. tests)
```
tests/LAYOUT_INVENTORY.md
+175 -142
View File
@@ -1,169 +1,202 @@
# Test layout inventory
> [!NOTE]
> This document is a completed implementation snapshot for the first low-risk
> test-directory reorganization. It does not propose additional file moves and
> is not the canonical definition of the current test taxonomy.
- **Original inventory:** #3712
- **Parent tracker:** #2523
- **Implemented by:** #3842
- **Status:** completed historical snapshot
- **Durable testing policy:** [`TESTING_STANDARD.md`](./TESTING_STANDARD.md)
- **Current testing mechanics:** [`README.md`](./README.md)
## Disposition
This document is retained as a non-canonical historical record of the completed
CLI test relocation.
Reusable testing rules are maintained in `TESTING_STANDARD.md`; current helper,
taxonomy-runner, and execution mechanics are maintained in `README.md`, the test
suite, and its supporting code.
Counts, residual-file observations, and path listings in this document are
snapshot evidence only. They must not be maintained as durable repository truth.
Any additional test relocation requires a separately approved issue or work item.
# Test Layout Inventory
## Purpose
This document preserves the reasoning, scope, and validation contract for the
first mechanical reorganization of the formerly flat `tests/` directory.
Inventory for the first low-risk split of the flat `tests/` directory
(issue #3712, parent #2523). This document only records *what* should move
first and *why*; it moves nothing. The actual move is a separate, mechanical
PR that relocates the listed files verbatim and changes no test content.
The original change moved 28 tests classified as `area_cli` into `tests/cli/`
without changing their assertions, helpers, taxonomy, or runtime behaviour.
The target layout and category definitions come from
[`TESTING_STANDARD.md`](./TESTING_STANDARD.md); the collection-time markers
come from [`_taxonomy.py`](./_taxonomy.py), which classifies by **filename
tokens only** (paths are ignored, except the `tests/helpers/` rule). A file
keeps its `area_*`/`sub_*` markers when moved into a subdirectory, and
`conftest.py` discovers marker names recursively (`rglob`), so a move does not
disturb marker registration or focused selection.
Current testing guidance and classification rules are maintained in:
## Current low-risk candidate groups
- [`README.md`](./README.md)
- [`TESTING_STANDARD.md`](./TESTING_STANDARD.md)
- [`_taxonomy.py`](./_taxonomy.py)
Groups whose tests need no route/app setup and no real DB/session setup:
The current repository and test suite remain authoritative when this snapshot
and the implementation differ.
1. **CLI / script tests** (`area_cli`, 28 files) - load `scripts/` entry
points via `tests.helpers.cli_loader.load_script`; DB access is stubbed
with `tests.helpers.db_stubs` (`SessionLocal` is a plain stub attribute).
No `TestClient`, no FastAPI app import, no SQLite files.
2. **Helper self-tests** (`area_helpers`) - e.g. `test_helpers_import_state.py`,
`test_db_stubs_helper.py`. Safe but tiny (two files), and they test the
shared helpers from the #3685 audit (merged) that the rest of the suite
depends on; little payoff as a first slice.
3. **Pure unit / parsing tests** (`area_unit`) - `*_nonstring.py`,
`*_nondict.py`, parsing tests. Large and heterogeneous; some touch
provider/session modules, so the boundary is less crisp.
4. **Static checks** - e.g. `test_readme_ascii_fenced.py`,
`test_docs_no_orphan_images.py`. Safe but tiny and `uncategorized` in the
taxonomy, so a move buys little and matches no existing marker.
## Why CLI tests were selected
Not candidates for the first move (per #3712 guidance): security/owner-scope
tests, route/API tests, DB/session-heavy tests, auth/session concurrency
tests, and the taxonomy/runner infrastructure tests that changed recently
(#3491, #3556, #3659, #3711).
The CLI and script tests were selected as the first low-risk group because they
formed a narrow, mechanically identifiable boundary:
## Recommended first move
- script entry points were loaded through shared CLI helpers;
- database access used controlled stubs rather than persistent sessions;
- the tests did not require FastAPI application or route setup;
- the tests did not require a real database;
- taxonomy classification could be derived consistently from filenames;
- absolute helper imports remained valid after relocation.
**CLI / script tests → `tests/cli/`**
This reduced the risk relative to route, security, session, provider, and
heterogeneous unit-test groups.
Why this group over the alternatives:
## Completed implementation
- Lowest coupling: every file imports only the script under test (via
`cli_loader`) plus `tests.helpers` stubs - no app, no routes, no real DB.
- Crisp, machine-checkable boundary: the set is exactly the files classified
`area_cli` by `_taxonomy.py`, so before/after selection counts can be
compared mechanically.
- Already the planned target dir for this category in `TESTING_STANDARD.md`
(`tests/cli/`).
- Absolute imports (`from tests.helpers...`) and unique basenames mean no
import-order or module-name collisions after the move.
- Lower risk than helper self-tests (tiny group, little payoff), unit tests
(fuzzy boundary), or anything security/route/session-shaped.
Repository history records the implementation as:
## Files included in the first move
- `83af3ca test: move area_cli tests into cli directory (#3842)`
The 28 files classified `area_cli` (verified against `_taxonomy.py`):
The following 28 files were included in that completed move:
Note: this inventory was refreshed against current `dev` after `tests/test_research_cli_status.py` was added to the `area_cli` set.
- `tests/cli/test_calendar_cli_name.py`
- `tests/cli/test_contacts_cli_rows.py`
- `tests/cli/test_cookbook_cli_state.py`
- `tests/cli/test_docs_cli_content_length.py`
- `tests/cli/test_gallery_cli_album_count.py`
- `tests/cli/test_gallery_cli_preview.py`
- `tests/cli/test_logs_cli_resolve_nonstring.py`
- `tests/cli/test_mail_cli_read_empty_fetch.py`
- `tests/cli/test_mail_cli_recipients.py`
- `tests/cli/test_mcp_cli_env_serialize.py`
- `tests/cli/test_mcp_cli_json.py`
- `tests/cli/test_memory_cli_rows.py`
- `tests/cli/test_notes_cli_items.py`
- `tests/cli/test_personal_cli_rows.py`
- `tests/cli/test_preset_cli_invalid_entries.py`
- `tests/cli/test_preset_cli_set_corrupt_entry.py`
- `tests/cli/test_preset_cli_store.py`
- `tests/cli/test_research_cli_preview.py`
- `tests/cli/test_research_cli_status.py`
- `tests/cli/test_research_cli_status_filter.py`
- `tests/cli/test_research_cli_store.py`
- `tests/cli/test_sessions_cli.py`
- `tests/cli/test_signature_cli_export.py`
- `tests/cli/test_skills_cli_preview.py`
- `tests/cli/test_skills_cli_rows.py`
- `tests/cli/test_tasks_cli_preview.py`
- `tests/cli/test_theme_cli_store.py`
- `tests/cli/test_webhook_cli_mask.py`
- `tests/test_calendar_cli_name.py`
- `tests/test_contacts_cli_rows.py`
- `tests/test_cookbook_cli_state.py`
- `tests/test_docs_cli_content_length.py`
- `tests/test_gallery_cli_album_count.py`
- `tests/test_gallery_cli_preview.py`
- `tests/test_logs_cli_resolve_nonstring.py`
- `tests/test_mail_cli_read_empty_fetch.py`
- `tests/test_mail_cli_recipients.py`
- `tests/test_mcp_cli_env_serialize.py`
- `tests/test_mcp_cli_json.py`
- `tests/test_memory_cli_rows.py`
- `tests/test_notes_cli_items.py`
- `tests/test_personal_cli_rows.py`
- `tests/test_preset_cli_invalid_entries.py`
- `tests/test_preset_cli_set_corrupt_entry.py`
- `tests/test_preset_cli_store.py`
- `tests/test_research_cli_preview.py`
- `tests/test_research_cli_status_filter.py`
- `tests/test_research_cli_status.py`
- `tests/test_research_cli_store.py`
- `tests/test_sessions_cli.py`
- `tests/test_signature_cli_export.py`
- `tests/test_skills_cli_preview.py`
- `tests/test_skills_cli_rows.py`
- `tests/test_tasks_cli_preview.py`
- `tests/test_theme_cli_store.py`
- `tests/test_webhook_cli_mask.py`
## Original exclusions
## Files intentionally excluded
The original inventory deliberately excluded:
- `tests/test_backup_cli_security.py` - classifies as `area_security`
(security outranks cli in the taxonomy); moving it into `tests/cli/` would
make the directory disagree with its marker. It belongs with the security
group in a later phase.
- `tests/test_run_focus.py`, `tests/test_taxonomy.py` - taxonomy/runner
infrastructure tests, recently changed (#3556, #3659); they also pin
flat-layout paths (e.g. `tests/test_auth_config_lock_concurrency.py` in
`test_run_focus.py`), so they stay put.
- Script-like but `uncategorized` files - `test_pr_blocker_audit.py`,
`test_update_database_script.py`, `test_windows_update_script.py`,
`test_setup_admin_user.py`, `test_amd_gpu_check_args.py`, `test_hwfit_*.py`.
They exercise `scripts/` too, but moving them would make `tests/cli/`
diverge from the `area_cli` marker set. Reclassify or move them in a later,
separate slice.
- Everything else (security, routes, services, unit, js, helpers) - out of
scope for the first move by design.
- `tests/test_backup_cli_security.py`, because its security classification
outranked the CLI token in its filename;
- taxonomy and focused-runner infrastructure, including
`tests/test_taxonomy.py` and `tests/test_run_focus.py`;
- script-oriented tests not classified as `area_cli`;
- route, service, security, session, JavaScript, helper, and broad unit-test
groups.
## How this was verified
These exclusions kept the implementation mechanical and prevented it from
mixing taxonomy changes, assertion changes, helper extraction, or unrelated
test-suite restructuring.
## Current repository observation
The current taxonomy classifies 30 files as `area_cli`:
| Location | Files |
|---|---:|
| `tests/cli/` | 28 |
| Flat `tests/` directory | 2 |
The two currently classified CLI tests outside `tests/cli/` are:
- `tests/test_calendar_cli_overlap.py`
- `tests/test_memory_cli_add_nondict.py`
These files were not part of the original 28-file implementation.
Their presence is recorded solely to keep this snapshot accurate. This
documentation update does not recommend moving them, changing their taxonomy,
or opening implementation work. Any future test-layout change requires separate
maintainer agreement, scope, and validation.
## Validation contract preserved by the original move
The completed implementation was designed to preserve:
- identical `area_cli` selection before and after relocation;
- successful direct execution of the relocated tests;
- successful whole-suite collection;
- unchanged taxonomy and focused-runner behaviour;
- no stale references to the former flat paths;
- no production-code changes.
Relevant current verification commands include:
Read-only checks, run from the repo root on this branch. Note the real API is
`classify_test_path` (there is no `classify_test_file`).
```bash
venv/bin/python tests/run_focus.py --dry-run --area cli
venv/bin/python -m pytest -m area_cli -q
venv/bin/python -m pytest tests/cli/ -q
venv/bin/python -m pytest tests/test_taxonomy.py tests/test_run_focus.py -q
venv/bin/python -m pytest --collect-only -q
# Compute the area_cli set and confirm test_backup_cli_security.py is
# area_security. Expected: 28 files, then "security".
./venv/bin/python - <<'PY'
from pathlib import Path
from tests._taxonomy import classify_test_path
cli = [p for p in sorted(Path("tests").glob("test_*.py"))
if classify_test_path(p).area == "cli"]
print(len(cli))
for p in cli:
print(p)
print(classify_test_path("tests/test_backup_cli_security.py").area)
PY
# Coupling check across the CLI files. Expected: the only hits are
# "SessionLocal" as stub attribute names passed to tests.helpers.db_stubs;
# no TestClient, FastAPI, create_app, sqlite, or dependency_overrides.
rg -n "TestClient|FastAPI|create_app|SessionLocal|sqlite|dependency_overrides" \
tests/test_*cli*.py tests/test_sessions_cli.py
# Hard-coded flat paths to the exact CLI files outside tests/. Expected: no matches.
./venv/bin/python - <<'PY2' > /tmp/area_cli_paths.txt
from pathlib import Path
from tests._taxonomy import classify_test_path
for path in sorted(Path("tests").glob("test_*.py")):
if classify_test_path(path).area == "cli":
print(path)
PY2
rg -n -F -f /tmp/area_cli_paths.txt .github scripts docs \
tests/README.md tests/TESTING_STANDARD.md pyproject.toml 2>/dev/null || true
```
These commands describe validation coverage. This snapshot does not claim that
they were executed as part of the present documentation update.
Also checked by reading the code: `tests/conftest.py` registers sub-markers
from a recursive `rglob` scan, and `tests/_taxonomy.py` classifies by filename
tokens only (plus the `tests/helpers/` directory rule), so the markers of the
28 files do not change when they move into `tests/cli/`.
## Validation for the future move PR
Run with the project venv (`./venv/bin/python`); system `python3` may miss
pinned deps. Before the move, record the baseline; after, compare:
```bash
# Selection must match the 28 files before and after the move.
./venv/bin/python tests/run_focus.py --dry-run --area cli
./venv/bin/python -m pytest -m area_cli -q
# Moved files pass when targeted directly.
./venv/bin/python -m pytest tests/cli/ -q
# Whole-suite collection still succeeds (catches import/path breakage).
./venv/bin/python -m pytest --collect-only -q
# Taxonomy/runner infrastructure is unaffected.
./venv/bin/python -m pytest tests/test_taxonomy.py tests/test_run_focus.py -q
# No stale flat-path references to the moved files. Expected: no matches
# outside tests/cli/ itself.
./venv/bin/python - <<'PY2' > /tmp/area_cli_paths.txt
from pathlib import Path
from tests._taxonomy import classify_test_path
for path in sorted(Path("tests").glob("test_*.py")):
if classify_test_path(path).area == "cli":
print(path)
PY2
rg -n -F -f /tmp/area_cli_paths.txt .github scripts docs \
tests/README.md tests/TESTING_STANDARD.md pyproject.toml 2>/dev/null || true
```
Pass criteria: identical test counts for `-m area_cli` before/after, zero
collection errors, and no changes outside the moved files.
## Non-goals
This document does not:
- propose moving or renaming tests;
- redefine taxonomy classifications;
- change test behaviour or assertions;
- prescribe the next test-suite refactor;
- replace the testing standard;
- create implementation work from the two current residual files.
- No file moves, renames, or deletions in this PR.
- No changes to `conftest.py`, `_taxonomy.py`, `run_focus.py`, helpers,
markers, CI workflows, or production code.
- No recommendation to split the whole suite at once; later groups get their
own inventory-then-move slices.
tests/OVERSIZED_TEST_SPLIT_PLAN.md
+148 -181
View File
@@ -1,13 +1,5 @@
# Oversized Test File Split Plan
> [!NOTE]
> This is generated planning material based on the current test tree.
>
> It is non-canonical and disposable. Durable testing policy belongs in
> `tests/TESTING_STANDARD.md`; active split selection and prioritization
> belong in approved issues or the project tracker.
> It does not define runtime behaviour or approve any test-file split.
## Purpose
This document plans future oversized test-file splits using current repo data.
@@ -34,8 +26,8 @@ Future split PRs must still inspect each file manually before editing.
## Current summary
- test files scanned: 746
- collected pytest items counted: 4764
- test files scanned: 583
- collected pytest items counted: 3586
- large-file threshold: 300 lines
- large-collected threshold: 20 collected items
@@ -43,123 +35,123 @@ Area distribution:
| Value | Files |
|---|---:|
| cli | 30 |
| cli | 28 |
| helpers | 1 |
| js | 49 |
| routes | 35 |
| security | 88 |
| services | 197 |
| uncategorized | 306 |
| unit | 40 |
| js | 39 |
| routes | 23 |
| security | 77 |
| services | 144 |
| uncategorized | 234 |
| unit | 37 |
Sub-area distribution:
| Value | Files |
|---|---:|
| api | 8 |
| api | 6 |
| atomic | 3 |
| auth | 11 |
| calendar | 12 |
| cli | 30 |
| confinement | 8 |
| cookbook | 20 |
| document | 13 |
| email | 23 |
| embedding | 6 |
| gallery | 6 |
| auth | 9 |
| calendar | 10 |
| cli | 28 |
| confinement | 7 |
| cookbook | 13 |
| document | 11 |
| email | 12 |
| embedding | 3 |
| gallery | 5 |
| history | 3 |
| js | 49 |
| llm | 21 |
| mcp | 12 |
| memory | 17 |
| js | 39 |
| llm | 16 |
| mcp | 8 |
| memory | 15 |
| nondict | 7 |
| nonstring | 23 |
| owner | 15 |
| owner_scope | 27 |
| parse | 5 |
| provider | 16 |
| research | 17 |
| nonstring | 22 |
| owner | 14 |
| owner_scope | 23 |
| parse | 4 |
| provider | 6 |
| research | 16 |
| route | 6 |
| routes | 19 |
| scheduler | 4 |
| routes | 9 |
| scheduler | 3 |
| scope | 5 |
| security | 9 |
| session | 18 |
| ssrf | 6 |
| webhook | 4 |
| session | 16 |
| ssrf | 3 |
| webhook | 3 |
| xss | 5 |
Values below 2 files: 318 values covering 318 files.
Values below 2 files: 244 values covering 244 files.
## Top files by collected pytest items
| File | Lines | Collected tests | Test defs | Test classes | Area | Sub-area | Signals |
|---|---:|---:|---:|---:|---|---|---|
| `tests/test_model_routes.py` | 2099 | 163 | 134 | 10 | routes | routes | route/api, db/session, import-state, async/threading |
| `tests/test_security_regressions.py` | 1515 | 98 | 74 | 0 | security | security | route/api, db/session, import-state, security, filesystem, async/threading, ui/static |
| `tests/test_cookbook_helpers.py` | 958 | 69 | 69 | 0 | services | cookbook | route/api, filesystem, subprocess/script, async/threading, ui/static |
| `tests/test_shell_routes.py` | 538 | 69 | 53 | 9 | routes | routes | route/api, import-state, filesystem |
| `tests/test_endpoint_probing.py` | 497 | 61 | 36 | 7 | uncategorized | endpoint_probing | route/api, db/session, import-state |
| `tests/test_model_routes.py` | 1778 | 139 | 116 | 10 | routes | routes | route/api, db/session, import-state, async/threading |
| `tests/test_security_regressions.py` | 1224 | 92 | 68 | 0 | security | security | route/api, db/session, import-state, security, filesystem, async/threading, ui/static |
| `tests/test_provider_classification.py` | 188 | 67 | 21 | 4 | services | provider | - |
| `tests/test_cookbook_helpers.py` | 912 | 65 | 65 | 0 | services | cookbook | route/api, filesystem, subprocess/script, async/threading, ui/static |
| `tests/test_shell_routes.py` | 481 | 63 | 48 | 8 | routes | routes | route/api, import-state, filesystem |
| `tests/test_pr_blocker_audit.py` | 964 | 58 | 58 | 0 | uncategorized | pr_blocker_audit | import-state, security, filesystem |
| `tests/test_research_routes_path_confinement.py` | 565 | 58 | 26 | 0 | security | confinement | route/api, security, filesystem, async/threading |
| `tests/test_run_focus.py` | 492 | 55 | 52 | 0 | uncategorized | run_focus | security, filesystem, subprocess/script, ui/static |
| `tests/test_agent_loop.py` | 500 | 54 | 54 | 5 | uncategorized | agent_loop | db/session, import-state |
| `tests/test_manage_mcp_command_allowlist.py` | 168 | 45 | 10 | 0 | services | mcp | db/session, subprocess/script, async/threading, ui/static |
| `tests/test_provider_classification.py` | 115 | 42 | 9 | 2 | services | provider | - |
| `tests/test_review_regressions.py` | 1429 | 41 | 41 | 0 | uncategorized | review_regressions | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_chat_helpers.py` | 582 | 37 | 24 | 0 | uncategorized | chat_helpers | route/api, filesystem, async/threading |
| `tests/test_model_context.py` | 314 | 36 | 34 | 4 | uncategorized | model_context | db/session, import-state |
| `tests/test_email_oauth.py` | 798 | 34 | 29 | 0 | services | email | route/api, db/session, security, async/threading |
| `tests/test_provider_endpoints.py` | 241 | 58 | 18 | 1 | services | provider | subprocess/script |
| `tests/test_agent_loop.py` | 469 | 52 | 52 | 5 | uncategorized | agent_loop | db/session, import-state |
| `tests/test_service_health.py` | 472 | 47 | 42 | 0 | uncategorized | service_health | async/threading |
| `tests/test_run_focus.py` | 399 | 47 | 44 | 0 | uncategorized | run_focus | security, filesystem, subprocess/script, ui/static |
| `tests/test_llm_core_temperature.py` | 196 | 41 | 17 | 0 | services | llm | - |
| `tests/test_endpoint_probing.py` | 411 | 34 | 30 | 6 | uncategorized | endpoint_probing | route/api, db/session, import-state |
| `tests/test_llm_core_anthropic_temp_omit.py` | 94 | 32 | 6 | 0 | services | llm | db/session |
| `tests/test_redos_verdict_continuation.py` | 80 | 30 | 5 | 0 | uncategorized | redos_verdict_continuation | route/api |
| `tests/test_chat_helpers.py` | 264 | 31 | 18 | 0 | uncategorized | chat_helpers | route/api |
| `tests/test_provider_detection.py` | 148 | 31 | 31 | 5 | services | provider | - |
| `tests/test_model_context.py` | 251 | 30 | 30 | 4 | uncategorized | model_context | db/session, import-state |
| `tests/test_endpoint_resolver.py` | 148 | 30 | 30 | 6 | uncategorized | endpoint_resolver | - |
| `tests/test_embedding_lanes.py` | 1104 | 29 | 29 | 0 | services | embedding | filesystem |
| `tests/test_upload_limits_centralized.py` | 110 | 29 | 5 | 0 | uncategorized | upload_limits_centralized | import-state, filesystem |
| `tests/test_cookbook_docker_access.py` | 339 | 28 | 13 | 0 | services | cookbook | route/api, filesystem, async/threading |
| `tests/test_rename_user_owner_sync.py` | 740 | 27 | 27 | 0 | security | owner | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_taxonomy.py` | 151 | 27 | 17 | 0 | uncategorized | taxonomy | security, ui/static |
| `tests/test_endpoint_resolver_urls.py` | 95 | 27 | 23 | 3 | uncategorized | endpoint_resolver_urls | - |
| `tests/test_email_oauth.py` | 580 | 28 | 25 | 0 | services | email | route/api, db/session, security, async/threading |
| `tests/test_review_regressions.py` | 930 | 26 | 26 | 0 | uncategorized | review_regressions | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_rename_user_owner_sync.py` | 686 | 26 | 26 | 0 | security | owner | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_helpers_import_state.py` | 426 | 26 | 26 | 0 | helpers | helpers | route/api, db/session, import-state |
| `tests/test_workspace_confine.py` | 527 | 25 | 25 | 0 | uncategorized | workspace_confine | route/api, filesystem, subprocess/script, async/threading |
| `tests/test_chat_route_tool_policy.py` | 344 | 25 | 23 | 0 | routes | route | filesystem, ui/static |
| `tests/test_tool_path_confinement.py` | 303 | 25 | 25 | 0 | security | confinement | import-state, filesystem, async/threading |
| `tests/test_copilot.py` | 182 | 25 | 18 | 0 | uncategorized | copilot | - |
| `tests/test_llm_core_temperature_reasoning.py` | 133 | 25 | 9 | 0 | services | llm | - |
| `tests/test_provider_endpoints_url_building.py` | 86 | 25 | 3 | 0 | services | provider | - |
| `tests/test_bg_job_tools.py` | 174 | 23 | 16 | 0 | uncategorized | bg_job_tools | filesystem, async/threading |
| `tests/test_taxonomy.py` | 145 | 26 | 16 | 0 | uncategorized | taxonomy | security, ui/static |
| `tests/test_tool_path_confinement.py` | 282 | 24 | 24 | 0 | security | confinement | import-state, filesystem, async/threading |
| `tests/test_copilot.py` | 170 | 23 | 16 | 0 | uncategorized | copilot | - |
| `tests/test_research_utils.py` | 97 | 23 | 23 | 2 | services | research | - |
| `tests/test_api_chat_security.py` | 401 | 22 | 8 | 0 | security | security | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_tool_support_heuristic.py` | 166 | 22 | 22 | 3 | uncategorized | tool_support_heuristic | - |
| `tests/test_platform_compat.py` | 318 | 21 | 21 | 0 | uncategorized | platform_compat | import-state, filesystem, subprocess/script |
## Top files by physical line count
| File | Lines | Collected tests | Test defs | Test classes | Area | Sub-area | Signals |
|---|---:|---:|---:|---:|---|---|---|
| `tests/test_model_routes.py` | 2099 | 163 | 134 | 10 | routes | routes | route/api, db/session, import-state, async/threading |
| `tests/test_security_regressions.py` | 1515 | 98 | 74 | 0 | security | security | route/api, db/session, import-state, security, filesystem, async/threading, ui/static |
| `tests/test_review_regressions.py` | 1429 | 41 | 41 | 0 | uncategorized | review_regressions | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_model_routes.py` | 1778 | 139 | 116 | 10 | routes | routes | route/api, db/session, import-state, async/threading |
| `tests/test_security_regressions.py` | 1224 | 92 | 68 | 0 | security | security | route/api, db/session, import-state, security, filesystem, async/threading, ui/static |
| `tests/test_embedding_lanes.py` | 1104 | 29 | 29 | 0 | services | embedding | filesystem |
| `tests/test_pr_blocker_audit.py` | 964 | 58 | 58 | 0 | uncategorized | pr_blocker_audit | import-state, security, filesystem |
| `tests/test_cookbook_helpers.py` | 958 | 69 | 69 | 0 | services | cookbook | route/api, filesystem, subprocess/script, async/threading, ui/static |
| `tests/test_upload_handler_cleanup.py` | 831 | 13 | 13 | 0 | uncategorized | upload_handler_cleanup | route/api, db/session, filesystem, async/threading |
| `tests/test_email_oauth.py` | 798 | 34 | 29 | 0 | services | email | route/api, db/session, security, async/threading |
| `tests/test_rename_user_owner_sync.py` | 740 | 27 | 27 | 0 | security | owner | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_email_owner_scope.py` | 656 | 13 | 13 | 0 | security | owner_scope | route/api, db/session, filesystem, async/threading |
| `tests/test_model_capability_readers.py` | 646 | 20 | 20 | 0 | uncategorized | model_capability_readers | - |
| `tests/test_chat_helpers.py` | 582 | 37 | 24 | 0 | uncategorized | chat_helpers | route/api, filesystem, async/threading |
| `tests/test_review_regressions.py` | 930 | 26 | 26 | 0 | uncategorized | review_regressions | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_cookbook_helpers.py` | 912 | 65 | 65 | 0 | services | cookbook | route/api, filesystem, subprocess/script, async/threading, ui/static |
| `tests/test_rename_user_owner_sync.py` | 686 | 26 | 26 | 0 | security | owner | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_email_oauth.py` | 580 | 28 | 25 | 0 | services | email | route/api, db/session, security, async/threading |
| `tests/test_api_token_routes.py` | 578 | 17 | 17 | 0 | routes | api_routes | route/api, db/session, import-state, async/threading |
| `tests/test_research_routes_path_confinement.py` | 565 | 58 | 26 | 0 | security | confinement | route/api, security, filesystem, async/threading |
| `tests/test_shell_routes.py` | 538 | 69 | 53 | 9 | routes | routes | route/api, import-state, filesystem |
| `tests/test_workspace_confine.py` | 527 | 25 | 25 | 0 | uncategorized | workspace_confine | route/api, filesystem, subprocess/script, async/threading |
| `tests/test_agent_loop.py` | 500 | 54 | 54 | 5 | uncategorized | agent_loop | db/session, import-state |
| `tests/test_endpoint_probing.py` | 497 | 61 | 36 | 7 | uncategorized | endpoint_probing | route/api, db/session, import-state |
| `tests/test_run_focus.py` | 492 | 55 | 52 | 0 | uncategorized | run_focus | security, filesystem, subprocess/script, ui/static |
| `tests/test_tool_policy.py` | 475 | 18 | 18 | 0 | uncategorized | tool_policy | import-state, async/threading |
| `tests/test_shell_routes.py` | 481 | 63 | 48 | 8 | routes | routes | route/api, import-state, filesystem |
| `tests/test_email_owner_scope.py` | 474 | 9 | 9 | 0 | security | owner_scope | route/api, db/session, filesystem, async/threading |
| `tests/test_service_health.py` | 472 | 47 | 42 | 0 | uncategorized | service_health | async/threading |
| `tests/test_agent_loop.py` | 469 | 52 | 52 | 5 | uncategorized | agent_loop | db/session, import-state |
| `tests/test_kv_cache_invalidation_2927.py` | 463 | 8 | 8 | 0 | uncategorized | kv_cache_invalidation_2927 | route/api, db/session, import-state, async/threading |
| `tests/test_email_test_connection_oauth.py` | 436 | 9 | 7 | 0 | services | email | route/api, db/session, async/threading |
| `tests/test_helpers_import_state.py` | 426 | 26 | 26 | 0 | helpers | helpers | route/api, db/session, import-state |
| `tests/test_endpoint_owner_scope_followup.py` | 414 | 11 | 11 | 0 | security | owner_scope | route/api, db/session, filesystem |
| `tests/test_api_chat_security.py` | 404 | 22 | 8 | 0 | security | security | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_endpoint_probing.py` | 411 | 34 | 30 | 6 | uncategorized | endpoint_probing | route/api, db/session, import-state |
| `tests/test_imap_leak_fixes.py` | 404 | 15 | 15 | 0 | uncategorized | imap_leak_fixes | route/api, db/session, security, filesystem |
| `tests/test_companion_readonly.py` | 402 | 17 | 17 | 0 | uncategorized | companion_readonly | db/session, import-state |
| `tests/test_api_chat_security.py` | 401 | 22 | 8 | 0 | security | security | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_upload_handler_atomicity.py` | 401 | 9 | 9 | 0 | uncategorized | upload_handler_atomicity | filesystem, async/threading |
| `tests/test_auth_regressions.py` | 370 | 15 | 15 | 0 | security | auth | route/api, db/session, import-state, async/threading |
| `tests/test_calendar_recurrence.py` | 356 | 20 | 20 | 0 | services | calendar | - |
| `tests/test_task_cookbook_admin_gate.py` | 350 | 10 | 10 | 0 | services | cookbook | route/api, db/session, import-state, filesystem, async/threading |
| `tests/test_run_focus.py` | 399 | 47 | 44 | 0 | uncategorized | run_focus | security, filesystem, subprocess/script, ui/static |
| `tests/test_auth_regressions.py` | 375 | 15 | 15 | 0 | security | auth | route/api, db/session, import-state, async/threading |
| `tests/test_calendar_owner_scope.py` | 345 | 7 | 7 | 0 | security | owner_scope | route/api, db/session, import-state, filesystem, async/threading, ui/static |
| `tests/test_null_owner_gates.py` | 342 | 20 | 20 | 0 | security | owner | route/api, db/session, import-state |
| `tests/test_agent_migration_manifest.py` | 340 | 15 | 15 | 0 | uncategorized | agent_migration_manifest | import-state, filesystem |
| `tests/test_calendar_recurrence.py` | 338 | 19 | 19 | 0 | services | calendar | - |
| `tests/test_tool_policy.py` | 330 | 13 | 13 | 0 | uncategorized | tool_policy | import-state, async/threading |
| `tests/test_workspace_confine.py` | 328 | 18 | 18 | 0 | uncategorized | workspace_confine | route/api, filesystem, subprocess/script, async/threading |
| `tests/test_diffusion_server_security.py` | 325 | 14 | 14 | 0 | security | security | route/api, import-state, security, filesystem, async/threading, ui/static |
## Split planning candidates
@@ -173,69 +165,53 @@ These are planning candidates only. A later split PR still needs a focused manua
| File | Why included | Setup/risk signals | Suggested handling |
|---|---|---|---|
| `tests/test_model_routes.py` | 2099 lines, 163 collected tests | route/api, db/session, import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_security_regressions.py` | 1515 lines, 98 collected tests | route/api, db/session, import-state, security, filesystem, async/threading, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_cookbook_helpers.py` | 958 lines, 69 collected tests | route/api, filesystem, subprocess/script, async/threading, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_shell_routes.py` | 538 lines, 69 collected tests | route/api, import-state, filesystem | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_endpoint_probing.py` | 497 lines, 61 collected tests | route/api, db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_model_routes.py` | 1778 lines, 139 collected tests | route/api, db/session, import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_security_regressions.py` | 1224 lines, 92 collected tests | route/api, db/session, import-state, security, filesystem, async/threading, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_provider_classification.py` | 67 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_cookbook_helpers.py` | 912 lines, 65 collected tests | route/api, filesystem, subprocess/script, async/threading, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_shell_routes.py` | 481 lines, 63 collected tests | route/api, import-state, filesystem | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_pr_blocker_audit.py` | 964 lines, 58 collected tests | import-state, security, filesystem | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_research_routes_path_confinement.py` | 565 lines, 58 collected tests | route/api, security, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_run_focus.py` | 492 lines, 55 collected tests | security, filesystem, subprocess/script, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_agent_loop.py` | 500 lines, 54 collected tests | db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_manage_mcp_command_allowlist.py` | 45 collected tests | db/session, subprocess/script, async/threading, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_provider_classification.py` | 42 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_review_regressions.py` | 1429 lines, 41 collected tests | route/api, db/session, import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_chat_helpers.py` | 582 lines, 37 collected tests | route/api, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_model_context.py` | 314 lines, 36 collected tests | db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_email_oauth.py` | 798 lines, 34 collected tests | route/api, db/session, security, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_provider_endpoints.py` | 58 collected tests | subprocess/script | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_agent_loop.py` | 469 lines, 52 collected tests | db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_service_health.py` | 472 lines, 47 collected tests | async/threading | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_run_focus.py` | 399 lines, 47 collected tests | security, filesystem, subprocess/script, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_llm_core_temperature.py` | 41 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_endpoint_probing.py` | 411 lines, 34 collected tests | route/api, db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_llm_core_anthropic_temp_omit.py` | 32 collected tests | db/session | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_redos_verdict_continuation.py` | 30 collected tests | route/api | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_chat_helpers.py` | 31 collected tests | route/api | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_provider_detection.py` | 31 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_model_context.py` | 30 collected tests | db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_endpoint_resolver.py` | 30 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_embedding_lanes.py` | 1104 lines, 29 collected tests | filesystem | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_upload_limits_centralized.py` | 29 collected tests | import-state, filesystem | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_cookbook_docker_access.py` | 339 lines, 28 collected tests | route/api, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_rename_user_owner_sync.py` | 740 lines, 27 collected tests | route/api, db/session, import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_taxonomy.py` | 27 collected tests | security, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_endpoint_resolver_urls.py` | 27 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_email_oauth.py` | 580 lines, 28 collected tests | route/api, db/session, security, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_review_regressions.py` | 930 lines, 26 collected tests | route/api, db/session, import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_rename_user_owner_sync.py` | 686 lines, 26 collected tests | route/api, db/session, import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_helpers_import_state.py` | 426 lines, 26 collected tests | route/api, db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_workspace_confine.py` | 527 lines, 25 collected tests | route/api, filesystem, subprocess/script, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_chat_route_tool_policy.py` | 344 lines, 25 collected tests | filesystem, ui/static | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_tool_path_confinement.py` | 303 lines, 25 collected tests | import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_copilot.py` | 25 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_llm_core_temperature_reasoning.py` | 25 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_provider_endpoints_url_building.py` | 25 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_bg_job_tools.py` | 23 collected tests | filesystem, async/threading | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_taxonomy.py` | 26 collected tests | security, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_tool_path_confinement.py` | 24 collected tests | import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_copilot.py` | 23 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_research_utils.py` | 23 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_api_chat_security.py` | 404 lines, 22 collected tests | route/api, db/session, import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_api_chat_security.py` | 401 lines, 22 collected tests | route/api, db/session, import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_tool_support_heuristic.py` | 22 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_platform_compat.py` | 322 lines, 21 collected tests | import-state, filesystem, subprocess/script | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_platform_compat.py` | 318 lines, 21 collected tests | import-state, filesystem, subprocess/script | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_context_compactor.py` | 21 collected tests | db/session, import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_prompt_security.py` | 21 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_fenced_inline_args.py` | 21 collected tests | db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_cache_affinity_local_only.py` | 21 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_model_capability_readers.py` | 646 lines, 20 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_calendar_recurrence.py` | 356 lines, 20 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_null_owner_gates.py` | 332 lines, 20 collected tests | route/api, db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_gallery_endpoint_hardening.py` | 326 lines, 20 collected tests | route/api, security, filesystem, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_redos_llm_parsers.py` | 20 collected tests | No obvious setup signals from static scan. | Good first manual-review candidate if test themes are cohesive. |
| `tests/test_code_nav_tools.py` | 20 collected tests | db/session, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_null_owner_gates.py` | 342 lines, 20 collected tests | route/api, db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_youtube_handler_consolidation.py` | 20 collected tests | route/api, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_tool_policy.py` | 475 lines | import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_fenced_example_not_executed_for_native_models.py` | 345 lines | async/threading | Plan split boundaries before editing. |
| `tests/test_calendar_recurrence.py` | 338 lines | No obvious setup signals from static scan. | Plan split boundaries before editing. |
| `tests/test_workspace_confine.py` | 328 lines | route/api, filesystem, subprocess/script, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_api_token_routes.py` | 578 lines | route/api, db/session, import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_companion_readonly.py` | 402 lines | db/session, import-state | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_set_admin.py` | 317 lines | route/api, import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_llm_core_ollama.py` | 324 lines | async/threading | Plan split boundaries before editing. |
| `tests/test_imap_leak_fixes.py` | 404 lines | route/api, db/session, security, filesystem | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_auth_regressions.py` | 370 lines | route/api, db/session, import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_auth_regressions.py` | 375 lines | route/api, db/session, import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_agent_migration_manifest.py` | 340 lines | import-state, filesystem | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_mcp_email_decode_header_spaces.py` | 327 lines | route/api, db/session, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_diffusion_server_security.py` | 325 lines | route/api, import-state, security, filesystem, async/threading, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_upload_handler_cleanup.py` | 831 lines | route/api, db/session, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_email_owner_scope.py` | 656 lines | route/api, db/session, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_upload_routes_owner_scope.py` | 344 lines | route/api, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_tool_policy.py` | 330 lines | import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_endpoint_owner_scope_followup.py` | 414 lines | route/api, db/session, filesystem | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_composer_arrow_up_recall_js.py` | 308 lines | filesystem, subprocess/script, ui/static | Plan split boundaries before editing. |
| `tests/test_task_cookbook_admin_gate.py` | 350 lines | route/api, db/session, import-state, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_email_test_connection_oauth.py` | 436 lines | route/api, db/session, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_upload_routes_owner_scope.py` | 315 lines | route/api, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_email_owner_scope.py` | 474 lines | route/api, db/session, filesystem, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_upload_handler_atomicity.py` | 401 lines | filesystem, async/threading | Plan split boundaries before editing. |
| `tests/test_kv_cache_invalidation_2927.py` | 463 lines | route/api, db/session, import-state, async/threading | Defer mechanical split until setup/risk boundaries are mapped. |
| `tests/test_calendar_owner_scope.py` | 345 lines | route/api, db/session, import-state, filesystem, async/threading, ui/static | Defer mechanical split until setup/risk boundaries are mapped. |
@@ -250,35 +226,28 @@ Before using any of these files as a split target, first decide whether the taxo
| File | Lines | Collected tests | Sub-area | Signals | Suggested follow-up |
|---|---:|---:|---|---|---|
| `tests/test_endpoint_probing.py` | 497 | 61 | endpoint_probing | route/api, db/session, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_pr_blocker_audit.py` | 964 | 58 | pr_blocker_audit | import-state, security, filesystem | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_run_focus.py` | 492 | 55 | run_focus | security, filesystem, subprocess/script, ui/static | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_agent_loop.py` | 500 | 54 | agent_loop | db/session, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_review_regressions.py` | 1429 | 41 | review_regressions | route/api, db/session, import-state, filesystem, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_chat_helpers.py` | 582 | 37 | chat_helpers | route/api, filesystem, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_model_context.py` | 314 | 36 | model_context | db/session, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_redos_verdict_continuation.py` | 80 | 30 | redos_verdict_continuation | route/api | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_agent_loop.py` | 469 | 52 | agent_loop | db/session, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_service_health.py` | 472 | 47 | service_health | async/threading | Review taxonomy mapping before using as a split target. |
| `tests/test_run_focus.py` | 399 | 47 | run_focus | security, filesystem, subprocess/script, ui/static | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_endpoint_probing.py` | 411 | 34 | endpoint_probing | route/api, db/session, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_chat_helpers.py` | 264 | 31 | chat_helpers | route/api | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_model_context.py` | 251 | 30 | model_context | db/session, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_endpoint_resolver.py` | 148 | 30 | endpoint_resolver | - | Review taxonomy mapping before using as a split target. |
| `tests/test_upload_limits_centralized.py` | 110 | 29 | upload_limits_centralized | import-state, filesystem | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_taxonomy.py` | 151 | 27 | taxonomy | security, ui/static | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_endpoint_resolver_urls.py` | 95 | 27 | endpoint_resolver_urls | - | Review taxonomy mapping before using as a split target. |
| `tests/test_workspace_confine.py` | 527 | 25 | workspace_confine | route/api, filesystem, subprocess/script, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_copilot.py` | 182 | 25 | copilot | - | Review taxonomy mapping before using as a split target. |
| `tests/test_bg_job_tools.py` | 174 | 23 | bg_job_tools | filesystem, async/threading | Review taxonomy mapping before using as a split target. |
| `tests/test_review_regressions.py` | 930 | 26 | review_regressions | route/api, db/session, import-state, filesystem, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_taxonomy.py` | 145 | 26 | taxonomy | security, ui/static | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_copilot.py` | 170 | 23 | copilot | - | Review taxonomy mapping before using as a split target. |
| `tests/test_tool_support_heuristic.py` | 166 | 22 | tool_support_heuristic | - | Review taxonomy mapping before using as a split target. |
| `tests/test_platform_compat.py` | 322 | 21 | platform_compat | import-state, filesystem, subprocess/script | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_platform_compat.py` | 318 | 21 | platform_compat | import-state, filesystem, subprocess/script | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_context_compactor.py` | 233 | 21 | context_compactor | db/session, import-state, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_fenced_inline_args.py` | 186 | 21 | fenced_inline_args | db/session, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_cache_affinity_local_only.py` | 104 | 21 | cache_affinity_local_only | - | Review taxonomy mapping before using as a split target. |
| `tests/test_model_capability_readers.py` | 646 | 20 | model_capability_readers | - | Review taxonomy mapping before using as a split target. |
| `tests/test_code_nav_tools.py` | 198 | 20 | code_nav_tools | db/session, filesystem, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_youtube_handler_consolidation.py` | 104 | 20 | youtube_handler_consolidation | route/api, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_tool_policy.py` | 475 | 18 | tool_policy | import-state, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_fenced_example_not_executed_for_native_models.py` | 345 | 18 | fenced_example_not_executed_for_native_models | async/threading | Review taxonomy mapping before using as a split target. |
| `tests/test_workspace_confine.py` | 328 | 18 | workspace_confine | route/api, filesystem, subprocess/script, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_companion_readonly.py` | 402 | 17 | companion_readonly | db/session, import-state | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_set_admin.py` | 317 | 17 | set_admin | route/api, import-state, filesystem, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_imap_leak_fixes.py` | 404 | 15 | imap_leak_fixes | route/api, db/session, security, filesystem | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_agent_migration_manifest.py` | 340 | 15 | agent_migration_manifest | import-state, filesystem | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_upload_handler_cleanup.py` | 831 | 13 | upload_handler_cleanup | route/api, db/session, filesystem, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_tool_policy.py` | 330 | 13 | tool_policy | import-state, async/threading | Review taxonomy and setup/risk boundaries before any split. |
| `tests/test_upload_handler_atomicity.py` | 401 | 9 | upload_handler_atomicity | filesystem, async/threading | Review taxonomy mapping before using as a split target. |
| `tests/test_kv_cache_invalidation_2927.py` | 463 | 8 | kv_cache_invalidation_2927 | route/api, db/session, import-state, async/threading | Review taxonomy and setup/risk boundaries before any split. |
@@ -290,16 +259,14 @@ Files still in the `uncategorized` taxonomy area are listed separately below so
| File | Lines | Collected tests | Area | Sub-area | Signals | Why this is a candidate |
|---|---:|---:|---|---|---|---|
| `tests/test_provider_classification.py` | 115 | 42 | services | provider | - | 42 collected tests |
| `tests/test_chat_route_tool_policy.py` | 344 | 25 | routes | route | filesystem, ui/static | 344 lines, 25 collected tests |
| `tests/test_llm_core_temperature_reasoning.py` | 133 | 25 | services | llm | - | 25 collected tests |
| `tests/test_provider_endpoints_url_building.py` | 86 | 25 | services | provider | - | 25 collected tests |
| `tests/test_provider_classification.py` | 188 | 67 | services | provider | - | 67 collected tests |
| `tests/test_provider_endpoints.py` | 241 | 58 | services | provider | subprocess/script | 58 collected tests |
| `tests/test_llm_core_temperature.py` | 196 | 41 | services | llm | - | 41 collected tests |
| `tests/test_provider_detection.py` | 148 | 31 | services | provider | - | 31 collected tests |
| `tests/test_embedding_lanes.py` | 1104 | 29 | services | embedding | filesystem | 1104 lines, 29 collected tests |
| `tests/test_research_utils.py` | 97 | 23 | services | research | - | 23 collected tests |
| `tests/test_prompt_security.py` | 203 | 21 | security | security | - | 21 collected tests |
| `tests/test_calendar_recurrence.py` | 356 | 20 | services | calendar | - | 356 lines, 20 collected tests |
| `tests/test_redos_llm_parsers.py` | 201 | 20 | services | llm | - | 20 collected tests |
| `tests/test_llm_core_ollama.py` | 324 | 16 | services | llm | async/threading | 324 lines |
| `tests/test_composer_arrow_up_recall_js.py` | 308 | 11 | js | js | filesystem, subprocess/script, ui/static | 308 lines |
| `tests/test_calendar_recurrence.py` | 338 | 19 | services | calendar | - | 338 lines |
## High-risk candidates to defer first
@@ -307,21 +274,21 @@ These files may still be split later, but not as the first implementation slice
| File | Lines | Collected tests | High-risk signals |
|---|---:|---:|---|
| `tests/test_model_routes.py` | 2099 | 163 | db/session, import-state, route/api |
| `tests/test_security_regressions.py` | 1515 | 98 | db/session, import-state, route/api, security |
| `tests/test_cookbook_helpers.py` | 958 | 69 | route/api |
| `tests/test_shell_routes.py` | 538 | 69 | import-state, route/api |
| `tests/test_endpoint_probing.py` | 497 | 61 | db/session, import-state, route/api |
| `tests/test_model_routes.py` | 1778 | 139 | db/session, import-state, route/api |
| `tests/test_security_regressions.py` | 1224 | 92 | db/session, import-state, route/api, security |
| `tests/test_cookbook_helpers.py` | 912 | 65 | route/api |
| `tests/test_shell_routes.py` | 481 | 63 | import-state, route/api |
| `tests/test_pr_blocker_audit.py` | 964 | 58 | import-state, security |
| `tests/test_research_routes_path_confinement.py` | 565 | 58 | route/api, security |
| `tests/test_run_focus.py` | 492 | 55 | security |
| `tests/test_agent_loop.py` | 500 | 54 | db/session, import-state |
| `tests/test_manage_mcp_command_allowlist.py` | 168 | 45 | db/session |
| `tests/test_review_regressions.py` | 1429 | 41 | db/session, import-state, route/api |
| `tests/test_chat_helpers.py` | 582 | 37 | route/api |
| `tests/test_model_context.py` | 314 | 36 | db/session, import-state |
| `tests/test_email_oauth.py` | 798 | 34 | db/session, route/api, security |
| `tests/test_agent_loop.py` | 469 | 52 | db/session, import-state |
| `tests/test_run_focus.py` | 399 | 47 | security |
| `tests/test_endpoint_probing.py` | 411 | 34 | db/session, import-state, route/api |
| `tests/test_llm_core_anthropic_temp_omit.py` | 94 | 32 | db/session |
| `tests/test_chat_helpers.py` | 264 | 31 | route/api |
| `tests/test_model_context.py` | 251 | 30 | db/session, import-state |
| `tests/test_upload_limits_centralized.py` | 110 | 29 | import-state |
| `tests/test_email_oauth.py` | 580 | 28 | db/session, route/api, security |
| `tests/test_review_regressions.py` | 930 | 26 | db/session, import-state, route/api |
| `tests/test_rename_user_owner_sync.py` | 686 | 26 | db/session, import-state, route/api |
## Rules for future split PRs
@@ -346,7 +313,7 @@ Do not start a split PR from this planning issue alone if the file's boundaries
This document was generated with:
```bash
venv/bin/python tests/tools/build_oversized_test_split_plan.py
.venv/bin/python tests/tools/build_oversized_test_split_plan.py
```
## Freshness check
@@ -354,6 +321,6 @@ venv/bin/python tests/tools/build_oversized_test_split_plan.py
After editing the builder or rebasing the branch, regenerate the plan and confirm no unexpected plan drift:
```bash
venv/bin/python tests/tools/build_oversized_test_split_plan.py
.venv/bin/python tests/tools/build_oversized_test_split_plan.py
git diff --exit-code -- tests/OVERSIZED_TEST_SPLIT_PLAN.md
```
tests/test_add_directory_event_loop.py
+253
View File
@@ -0,0 +1,253 @@
"""Regression guard for #5558 — POST /api/personal/add_directory must not run
the indexing job on the event loop.
The handler is ``async def`` but called ``rag.index_personal_documents``
(os.walk + file reads + per-chunk embedding + Chroma inserts) inline, so
FastAPI ran the whole job on the event loop and every other request queued
behind it: indexing a real directory froze the UI and API for 25+ minutes.
``personal_docs_manager.add_directory`` sits in the same blocking section — it
triggers ``refresh_index()``, which re-extracts text across tracked dirs.
These tests build the real router with fake managers and compare the thread
the indexing work runs on against the event loop's thread.
"""
import asyncio
import os
import threading
os.environ.setdefault("DATABASE_URL", "sqlite:///:memory:")
import httpx
from fastapi import FastAPI
from fastapi.testclient import TestClient
def _serialization_probe():
"""Shared counter proving two critical sections never overlap."""
state = {"active": 0, "max_active": 0}
lock = threading.Lock()
def enter():
with lock:
state["active"] += 1
state["max_active"] = max(state["max_active"], state["active"])
def leave():
with lock:
state["active"] -= 1
return state, enter, leave
# Concurrency tests are `async def` (pyproject asyncio_mode="auto") and drive the
# ASGI app through httpx.ASGITransport + AsyncClient + asyncio.gather, NOT starlette
# TestClient + ThreadPoolExecutor: the job lock is an asyncio.Lock acquired in the
# async handler, and TestClient's portal-thread dispatch deadlocks against it (same
# reason test_notes_fail_closed_auth.py uses ASGITransport). asyncio.gather runs both
# requests on the test's own loop.
def _async_client(app):
return httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://t")
import routes.personal_routes as personal_routes
from core.middleware import require_admin
from src.auth_helpers import require_user
class _FakeRag:
def __init__(self, record):
self._record = record
def index_personal_documents(self, directory, owner=None):
self._record["index_thread"] = threading.get_ident()
return {"success": True, "indexed_count": 3, "failed_count": 0}
class _FakeDocsManager:
def __init__(self, record):
self._record = record
self.index = []
def add_directory(self, directory, *, index=True, owner=None):
self._record["bookkeeping_thread"] = threading.get_ident()
self._record["bookkeeping_index_flag"] = index
def _build_app(tmp_path, monkeypatch, record):
monkeypatch.setattr(personal_routes, "PERSONAL_DIR", str(tmp_path))
monkeypatch.setattr(personal_routes, "get_rag_manager", lambda: _FakeRag(record))
app = FastAPI()
app.include_router(
personal_routes.setup_personal_routes(_FakeDocsManager(record), None, True)
)
app.dependency_overrides[require_user] = lambda: "tester"
app.dependency_overrides[require_admin] = lambda: None
@app.get("/loop-thread")
async def loop_thread_probe():
return {"thread": threading.get_ident()}
return app
def test_indexing_runs_off_the_event_loop(tmp_path, monkeypatch):
record = {}
app = _build_app(tmp_path, monkeypatch, record)
target = tmp_path / "docs"
target.mkdir()
# Context-manager client: one portal/event loop serves both requests, so
# the probe and the POST are guaranteed to see the same loop thread.
with TestClient(app) as client:
loop_thread = client.get("/loop-thread").json()["thread"]
resp = client.post(
"/api/personal/add_directory", json={"directory": str(target)}
)
assert resp.status_code == 200
assert record["index_thread"] != loop_thread, (
"index_personal_documents ran on the event loop thread — every other "
"request queues behind the indexing job (#5558)"
)
assert record["bookkeeping_thread"] != loop_thread, (
"personal_docs_manager.add_directory (refresh_index) ran on the event "
"loop thread"
)
def test_response_and_bookkeeping_unchanged(tmp_path, monkeypatch):
record = {}
app = _build_app(tmp_path, monkeypatch, record)
target = tmp_path / "docs"
target.mkdir()
client = TestClient(app)
resp = client.post("/api/personal/add_directory", json={"directory": str(target)})
assert resp.status_code == 200
body = resp.json()
assert body["success"] is True
assert body["indexed_count"] == 3
assert body["failed_count"] == 0
assert body["directory"] == os.path.realpath(str(target))
assert record["bookkeeping_index_flag"] is False
async def test_concurrent_add_directory_requests_serialize_indexing(tmp_path, monkeypatch):
"""Off-loop execution must not mean parallel index jobs: concurrent
requests would race PersonalDocsManager's unsynchronized list mutations
and file writes (save_directories/_save_excluded are plain open('w'))."""
import time
state, enter, leave = _serialization_probe()
def _slow_index(self, directory, owner=None):
enter(); time.sleep(0.2); leave()
return {"success": True, "indexed_count": 1, "failed_count": 0}
monkeypatch.setattr(_FakeRag, "index_personal_documents", _slow_index)
record = {}
app = _build_app(tmp_path, monkeypatch, record)
for name in ("docs_a", "docs_b"):
(tmp_path / name).mkdir()
async with _async_client(app) as ac:
results = await asyncio.gather(
ac.post("/api/personal/add_directory", json={"directory": str(tmp_path / "docs_a")}),
ac.post("/api/personal/add_directory", json={"directory": str(tmp_path / "docs_b")}),
)
assert all(r.status_code == 200 for r in results)
assert state["max_active"] == 1, (
f"{state['max_active']} index jobs ran in parallel — concurrent "
"add_directory requests must serialize"
)
def test_failed_indexing_still_returns_500(tmp_path, monkeypatch):
record = {}
app = _build_app(tmp_path, monkeypatch, record)
target = tmp_path / "docs"
target.mkdir()
def _fail(directory, owner=None):
return {"success": False, "message": "boom"}
monkeypatch.setattr(_FakeRag, "index_personal_documents", staticmethod(_fail))
client = TestClient(app)
resp = client.post("/api/personal/add_directory", json={"directory": str(target)})
assert resp.status_code == 500
assert "boom" in resp.json()["detail"]
async def test_add_and_remove_serialize(tmp_path, monkeypatch):
"""#5634: remove must hold the SAME job lock as add. Otherwise a remove
running while an add job is in flight races PersonalDocsManager's
unsynchronized list/index mutations — the inconsistent state the PR's
'add/remove are serialized' guarantee claims to prevent."""
import time
state, enter, leave = _serialization_probe()
def _slow_index(self, directory, owner=None):
enter(); time.sleep(0.25); leave()
return {"success": True, "indexed_count": 1, "failed_count": 0}
def _slow_remove(self, directory):
enter(); time.sleep(0.25); leave()
monkeypatch.setattr(_FakeRag, "index_personal_documents", _slow_index)
monkeypatch.setattr(_FakeDocsManager, "remove_directory", _slow_remove, raising=False)
record = {}
app = _build_app(tmp_path, monkeypatch, record)
(tmp_path / "docs_a").mkdir()
(tmp_path / "docs_b").mkdir()
async with _async_client(app) as ac:
results = await asyncio.gather(
ac.post("/api/personal/add_directory", json={"directory": str(tmp_path / "docs_a")}),
ac.delete("/api/personal/remove_directory", params={"directory": str(tmp_path / "docs_b")}),
)
assert all(r.status_code == 200 for r in results)
assert state["max_active"] == 1, (
f"{state['max_active']} add/remove critical sections overlapped — "
"remove must hold the same index job lock as add"
)
async def test_reload_serializes_with_add(tmp_path, monkeypatch):
"""#5634: POST /reload rebuilds the index via refresh_index(); it must hold
the same job lock so it cannot race an in-flight add job."""
import time
state, enter, leave = _serialization_probe()
def _slow_index(self, directory, owner=None):
enter(); time.sleep(0.25); leave()
return {"success": True, "indexed_count": 1, "failed_count": 0}
def _slow_refresh(self):
enter(); time.sleep(0.25); leave()
monkeypatch.setattr(_FakeRag, "index_personal_documents", _slow_index)
monkeypatch.setattr(_FakeDocsManager, "refresh_index", _slow_refresh, raising=False)
record = {}
app = _build_app(tmp_path, monkeypatch, record)
(tmp_path / "docs_a").mkdir()
async with _async_client(app) as ac:
results = await asyncio.gather(
ac.post("/api/personal/add_directory", json={"directory": str(tmp_path / "docs_a")}),
ac.post("/api/personal/reload"),
)
assert all(r.status_code == 200 for r in results)
assert state["max_active"] == 1, (
f"{state['max_active']} add/reload critical sections overlapped — "
"reload must hold the same index job lock as add"
)
tests/test_provider_endpoints_url_building.py
+3 -3
View File
@@ -1,8 +1,8 @@
"""Provider endpoint URL-building tests.
Covers ``build_chat_url`` and ``build_models_url`` for the provider families
exercised by this module: Anthropic, Gemini, Groq, xAI, OpenRouter, OpenAI,
DeepSeek, and Ollama (local and cloud).
Covers ``build_chat_url`` and ``build_models_url`` for every provider named in
ROADMAP.md: Anthropic, Gemini, Groq, xAI, OpenRouter, OpenAI, DeepSeek, Ollama
(local + cloud).
"""
import pytest
tests/tools/build_oversized_test_split_plan.py
+2 -10
View File
@@ -441,14 +441,6 @@ def write_report(metrics: list[FileMetric], node_count_total: int) -> None:
lines = [
"# Oversized Test File Split Plan",
"",
"> [!NOTE]",
"> This is generated planning material based on the current test tree.",
">",
"> It is non-canonical and disposable. Durable testing policy belongs in",
"> `tests/TESTING_STANDARD.md`; active split selection and prioritization",
"> belong in approved issues or the project tracker.",
"> It does not define runtime behaviour or approve any test-file split.",
"",
"## Purpose",
"",
"This document plans future oversized test-file splits using current repo data.",
@@ -516,7 +508,7 @@ def write_report(metrics: list[FileMetric], node_count_total: int) -> None:
"This document was generated with:",
"",
"```bash",
"venv/bin/python tests/tools/build_oversized_test_split_plan.py",
".venv/bin/python tests/tools/build_oversized_test_split_plan.py",
"```",
"",
"## Freshness check",
@@ -524,7 +516,7 @@ def write_report(metrics: list[FileMetric], node_count_total: int) -> None:
"After editing the builder or rebasing the branch, regenerate the plan and confirm no unexpected plan drift:",
"",
"```bash",
"venv/bin/python tests/tools/build_oversized_test_split_plan.py",
".venv/bin/python tests/tools/build_oversized_test_split_plan.py",
"git diff --exit-code -- tests/OVERSIZED_TEST_SPLIT_PLAN.md",
"```",
"",