fix(personal): serialize add/remove/reload on an async job lock

The #5558 fix took the job lock INSIDE the threadpool worker and only on the
add path, so (1) remove_directory and /reload mutated PersonalDocsManager's
unsynchronized list/index concurrently with an in-flight add — the inconsistent
state the PR claimed to prevent — and (2) a queued add blocked on the lock while
holding an AnyIO threadpool token, starving the shared pool.

Move the lock to an asyncio.Lock acquired in the async handler BEFORE offloading,
and route add, remove and reload through it. A waiting request now parks on the
event loop instead of pinning a worker, and all three mutators are serialized so
the 'add/remove are serialized and cannot leave inconsistent state' guarantee
holds. remove and reload also run their blocking work off the event loop. The
lock is per-router so each app binds it to its own loop; single-process scope.

Tests: add-vs-remove and add-vs-reload serialization regressions (async via
ASGITransport, since asyncio.Lock deadlocks starlette TestClient's portal); the
existing add-vs-add test converted to the same driver.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -8,8 +8,8 @@ body:
       value: |
         **Before submitting:** search [open issues](https://github.com/odysseus-dev/odysseus/issues)
         and [discussions](https://github.com/odysseus-dev/odysseus/discussions) first.
-        The [roadmap](https://github.com/odysseus-dev/odysseus/blob/main/ROADMAP.md) is directional rather than a complete backlog.
-        Feature requests that duplicate an existing issue or accepted proposal may be closed as duplicates.
+        Feature requests that duplicate [ROADMAP.md](https://github.com/odysseus-dev/odysseus/blob/main/ROADMAP.md)
+        or an existing open issue will be closed as duplicates.
 
         If your idea needs community input before it becomes a concrete proposal,
         start a [discussion](https://github.com/odysseus-dev/odysseus/discussions/categories/ideas) instead.

README.md

@@ -55,7 +55,7 @@ A full hover-to-play tour lives on the landing page: [`docs/index.html`](docs/in
 
 ## Contributing
 
-Help is welcome. The best entry points are fresh-install testing, provider setup bugs, mobile/editor polish, documentation, and small focused refactors. See [CONTRIBUTING.md](CONTRIBUTING.md), the [public roadmap](ROADMAP.md), and the open [GitHub issues](https://github.com/odysseus-dev/odysseus/issues).
+Help is welcome. The best entry points are fresh-install testing, provider setup bugs, mobile/editor polish, docs, and small focused refactors. See [CONTRIBUTING.md](CONTRIBUTING.md) and [ROADMAP.md](ROADMAP.md).
 
 ## Security
 

ROADMAP.md

@@ -1,55 +1,87 @@
-# Roadmap
+# Roadmap / Help Wanted
 
-This document provides a high-level view of the areas Odysseus is currently improving.
+Odysseus is on a voyage, but not home yet. It works great for me (lol), but this ship is moving fast and feedback/help would be appreciated! (I don't know what I'm doing, help).
 
-It is directional rather than exhaustive. Priorities may change as the project evolves, defects are discovered, and maintainers learn more from implementation work and user feedback.
+If you see weird CSS, strange layout behavior, or a suspiciously murky corner of
+the codebase, you are probably right to stay away.
 
-For current implementation work, see the open [GitHub issues](https://github.com/odysseus-dev/odysseus/issues). Accepted behaviour should be documented in the repository alongside the code.
+## High Priority
 
-## Current priorities
+- SQUASH BUGS
+- Fresh install smoke tests on Linux, macOS, and Windows. Docker, native Python,
+  and WSL all need coverage.
 
-### Reliability and setup
+- Integration audit: do integrations even work? Confirm what works, what needs setup docs, and what should be removed or hidden. 
+- Cookbook reliability on other computers. This is probably the area most likely to need work across different machines, GPUs, drivers, shells, and Python environments.
+- Cookbook SGLang support across platforms. Make sure SGLang setup/serve works
+  predictably on Linux, Windows/WSL, macOS where possible, Docker, and common
+  NVIDIA/AMD hardware paths.
+- Deep Research model presets by hardware. Recommend approved model/parameter
+  profiles for small, medium, and large local setups so people with different
+  hardware can use Deep Research without guessing. Surface this either in Deep
+  Research settings or as a Cookbook scan/dropdown suggestion.
+- Cookbook model scan/download ranking. Prioritize newer architectures and
+  better hardware-fit models instead of scoring everything almost the same.
+  Ranking should account for architecture age, quant format, VRAM/RAM fit,
+  backend support, vision/mmproj requirements, and likely serve reliability.
+- Cookbook error feedback and logging. Failed downloads, dependency installs,
+  preflights, and serve jobs should show the actual command/output/error in the
+  UI, with copyable logs and clear next steps instead of just "crashed".
+- Agent prompt/context bloat. Agent mode is too heavy for smaller local models:
+  tool schemas, skills, memory, documents, and instructions can eat the context
+  before the user request really starts. We need slimmer prompts, better tool
+  selection, smaller default tool sets, and clearer guidance for models with
+  4k/8k/16k context windows.
+- Local model speculative decoding support. For Odysseus-tuned local models,
+  plan to ship or recommend a small same-tokenizer draft model when the serving
+  backend supports it. Early vLLM testing showed a generic `Qwen3-0.6B` draft
+  beside `Qwen3-8B` can materially reduce wall time, while an unsupported
+  DSpark conversion performed poorly. Treat this as a supported draft-model lane
+  first; keep MTP-specific packaging as future work only when the architecture
+  and runtime support are real. Judge this by time-to-success, tool correctness,
+  grammar, and unchanged target output, not tokens/sec alone.
+- Skill/tool prompt-injection audit. User-editable skills, notes, documents,
+  fetched pages, and memories should be treated as untrusted data. Keep testing
+  whether models follow malicious instructions from those surfaces.
+- Better degraded-state reporting for ChromaDB, SearXNG, email, ntfy, and provider probes.
+- Email performance audit. Fetching, searching, opening, deleting, and sending
+  email can feel slow, especially over IMAP/SMTP providers with high latency.
+  Need someone who knows mail performance to profile the current flow, identify
+  whether the bottleneck is IMAP folder select/fetch, cache invalidation,
+  attachment/body loading, SMTP handshakes, or frontend refresh behavior, then
+  propose safer caching/prefetch/batching without breaking multi-account state.
+- Provider setup/probing audit for Anthropic, Gemini, Groq, xAI, OpenRouter, OpenAI, and DeepSeek.
 
-- Improve fresh-install and smoke-test coverage across supported environments.
-- Make provider setup, probing, and failure states more predictable.
-- Improve Cookbook reliability across hardware, operating systems, drivers, shells, and serving backends.
-- Improve degraded-state reporting and recovery guidance when optional services are unavailable.
+## Refactor Targets
+- CSS cleanup. `static/style.css` basically Calypso's island atm.
+- Tour core helper. The onboarding tours have too much copy-pasted scaffolding; promote a shared `tour-core.js` helper before adding more tours.
+- Modal/window positioning cleanup. Some window controls have improved, but the
+  underlying popup/dropdown/fixed-position behavior is still too fragile.
+- Mobile media override discoverability. A lot of "CSS did not move" bugs are mobile `@media` overrides of the same selector; comments or linting around desktop/mobile paired rules would help.
+- Dead code pass for old routes, stale feature flags, and unused UI states.
 
-### Local model workflows
+## Frontend
 
-- Improve hardware-aware model recommendations and compatibility guidance.
-- Evaluate serving optimizations, including speculative decoding, through reproducible benchmarks.
-- Improve installation, preflight checks, logging, and error reporting for local model serving.
-- Reduce prompt and context overhead for smaller local models.
+- Expand the Editor for quicker, more robust everyday use. Better file/document
+  handling, smoother window behavior, clearer save/export flows, stronger image
+  editing affordances, and fewer brittle edge cases.
+- Better AI integration for Notes and Todos. Notes should be easier for the
+  agent to read, update, summarize, and turn into actions. Todos should be
+  assignable to an agent from the UI, possibly through a button, task action,
+  or dedicated skill/tool flow.
+- Mobile gallery/editor polish. Easier to launch/download inpaint model or any missing pieces.
+- Accessibility pass: keyboard navigation, focus states, contrast, reduced motion.
+- Improve empty states and error messages on fresh installs.
+- Tighten first-run setup, hints, and tours so they do not repeat or fight each other.
+- Vendor CDN assets eventually for a more fully self-hosted/offline mode.
 
-### Safety and resilience
+## Backend
 
-- Continue hardening tool execution, filesystem access, credentials, networking, and destructive operations.
-- Treat content from documents, notes, memories, skills, and fetched pages as potentially untrusted.
-- Improve security-focused regression coverage and operational guidance.
-- Review integrations that expand access to sensitive data or privileged operations.
+- More tests around endpoint probing and provider setup.
+- Better task scheduler defaults and visibility.
+- Backup/restore guide and helper flow for `data/`.
+- Security hardening around admin-only tools and clear docs for their risk.
 
-### Product usability
+## Not The Focus Right Now
 
-- Improve first-run setup, onboarding, hints, and tours.
-- Improve accessibility, keyboard navigation, focus behaviour, contrast, and reduced-motion support.
-- Improve empty states, error messages, and recovery paths.
-- Strengthen Notes, Todos, Editor, mobile, and everyday workspace flows.
-
-### Architecture and maintainability
-
-- Reduce duplication and technical debt through focused, reviewable refactors.
-- Improve subsystem documentation as behaviour and architecture become stable.
-- Remove stale code, obsolete feature flags, and unsupported integrations.
-- Keep implementation decisions grounded in current code and verified behaviour.
-
-## Tracking work
-
-Concrete implementation tasks, defects, proposals, and technical investigations are tracked in:
-
-- [GitHub Issues](https://github.com/odysseus-dev/odysseus/issues)
-- [Contributing Guide](CONTRIBUTING.md)
-
-Maintainers may use additional private coordination tools for ownership, planning, and unresolved decisions.
-
-This roadmap is not a complete backlog or a guarantee that a particular item will be delivered.
+I prob shouldnt add more themes.

THREAT_MODEL.md

@@ -68,14 +68,14 @@ External content that reaches the LLM is treated as untrusted via `src/prompt_se
 - `X-Content-Type-Options: nosniff` and `Referrer-Policy: no-referrer` everywhere.
 - **CSP:** nonce-based `script-src 'self' 'nonce-{nonce}' https://cdn.jsdelivr.net`. `style-src 'unsafe-inline'` is intentionally kept — `static/index.html` ships inline `<style>` blocks and JS modules set `style=""` attributes at runtime. Inline styles do not execute script so the risk is visual-only. Removing this requires templating the HTML files and auditing all JS-set style attributes.
 
-## Token-Supplied Model Endpoints
-
-Direct `/api/v1/chat` requests with a token-supplied `base_url` must use a public HTTP(S) endpoint. This restriction applies only to untrusted direct values; administrator-configured endpoints may intentionally use local or LAN URLs for private model providers.
-
 ## Known Gaps
 
 These are open, acknowledged, and contributor help is welcome:
 
 1. **No shell/filesystem sandbox.** The agent `bash` and `read_file`/`write_file` tools run as the app process user with no network egress filtering or filesystem confinement. A successful prompt-injection reaching a shell-enabled admin session can make outbound requests to internal services. See #1058 for the sandbox proposal.
 
-2. **Token scopes are coarse.** There is no way to grant a session a subset of the owning user's privileges. Companion/mobile tokens carry either `chat` or `admin` scope with no per-capability granularity.
+2. **SSRF via `/api/v1/chat` `base_url` parameter.** A chat-scoped API token can supply an arbitrary `base_url`; the server forwards the LLM request to that host without validating the scheme or address. PR #1039 fixes this.
+
+3. **`src/search/` partial consolidation.** `src.search.core` and `src.search.providers` correctly alias `services.search` via `sys.modules` replacement. `analytics`, `cache`, `content`, `query`, and `ranking` are still independent copies that can drift. The SSRF regression tests in `tests/test_webhook_ssrf_resilience.py` test `src.webhook_manager` directly (separate from search), so the safety net there is intact. See #1058.
+
+4. **Token scopes are coarse.** There is no way to grant a session a subset of the owning user's privileges. Companion/mobile tokens carry either `chat` or `admin` scope with no per-capability granularity.

discovery/README.md

@@ -1,34 +0,0 @@
-# Odysseus discovery maps
-
-Compact, code-grounded discovery maps of cross-cutting systems in the checked-in Odysseus codebase. They preserve investigation context and open factual questions; they are not canonical subsystem specifications, a feature certification, or a substitute for normal testing.
-
-> [!IMPORTANT]
-> Checked-in code is the source of truth for current behaviour. Mature subsystem specifications, where they exist, are the canonical documentation of accepted subsystem behaviour. Check code, tests, and configuration before reconciling a discovery finding. Discovery remains non-canonical.
-
-## Explore the maps
-
-| Document | Purpose |
-|---|---|
-| [Current system map](system-map.md) | Records evidence locations, confirmed local observations, and factual open questions about subsystem boundaries. |
-| [Safety boundaries](safety-boundaries.md) | Records evidence about broad authority, safeguards, confirmed risks or gaps, and unverified behaviour. |
-
-## Working rules
-
-- **Trace the code first.** Confirm the current path in source before recording a claim.
-- **Promote selectively.** When an owning mature specification exists, add a fact only when it is verified, useful, and not already represented there.
-- **Record missing ownership.** When no owning specification exists, retain the verified finding in discovery and record missing documentation ownership as a follow-up.
-- **Retain uncertainty here.** Keep unresolved questions and useful investigation context in discovery rather than treating them as canonical truth.
-- **Keep specifications current-state only.** Do not record intentions, design direction, refactor plans, decision history, priority, ownership, or sequencing here.
-- **Investigate with cause.** Do not exhaustively revalidate existing functionality without a report, visible failure, relevant change, or high-authority review need.
-- **Review authority carefully.** Give execution, data access, external tools, credentials, destructive operations, and unattended work focused review.
-- **Use stable locations.** Cite modules, routes, classes, and functions instead of fragile line ranges or generated evidence tables.
-
-## Reconciliation flow
-
-1. Start with the relevant map and trace the cited code.
-2. Classify the finding against current source evidence and an owning mature specification where one exists.
-3. Promote only verified, useful facts that are missing from an existing owning specification.
-4. When no owning specification exists, retain the verified finding here and record missing documentation ownership as a follow-up; otherwise retain unresolved context here and correct stale wording.
-
-> [!NOTE]
-> This package intentionally contains no generator, validator, maturity scale, feature database, or parallel work tracker. The [architecture runtime inventory](../specs/architecture-runtime-inventory.md) remains useful structural context, but is an explicitly draft snapshot.

discovery/safety-boundaries.md

@@ -1,142 +0,0 @@
-# Safety boundaries
-
-> [!IMPORTANT]
-> This non-canonical discovery map records code-grounded safeguards, confirmed risks or gaps, and unverified behaviour. Broad authority does not by itself establish a vulnerability. Verify the cited source before relying on a finding. No destructive test, external connection, or real credential was used for this map.
-
-## Navigate the boundaries
-
-- [Shell and subprocess execution](#shell-and-subprocess-execution)
-- [Filesystem access and workspace confinement](#filesystem-access-and-workspace-confinement)
-- [Agent-controlled tool dispatch](#agent-controlled-tool-dispatch)
-- [MCP and external tool servers](#mcp-and-external-tool-servers)
-- [Outbound network requests and URL validation](#outbound-network-requests-and-url-validation)
-- [Secrets, credentials, and vault sessions](#secrets-credentials-and-vault-sessions)
-- [Authentication and privileged administration](#authentication-and-privileged-administration)
-- [Deletion, wipe, backup, and restore](#deletion-wipe-backup-and-restore)
-- [Background jobs and unattended task execution](#background-jobs-and-unattended-task-execution)
-
-## Shell and subprocess execution
-
-- **Boundary:** Shell routes, agent `bash` and `python` tools, local model serving, and detached background jobs.
-
-- **Available authority:** Commands run as the application process user and can create child processes.
-
-- **User-controlled inputs:** Direct shell requests, model-produced tool arguments, scheduled-task prompts, and model-serving configuration.
-
-- **Current safeguards:** Agent dispatch applies owner/admin checks and tool policy; process helpers use timeouts or bounded background-job lifecycle where implemented.
-
-- **Confirmed risks or gaps:** Intentional authority with a confirmed gap: the agent shell starts in its workspace but is not sandboxed to it, and has no egress sandbox. This is documented in source and the threat model; it is not a newly demonstrated bypass.
-
-- **Unverified behaviour:** Role-gate and disabled-tool outcomes, direct shell-route behaviour, and timeout, cancellation, and output handling for foreground and detached processes remain unverified.
-
-## Filesystem access and workspace confinement
-
-- **Boundary:** Agent read, write, patch, listing, glob, and grep tools.
-
-- **Available authority:** Read and modify files within active workspace confinement or fallback allowlisted roots.
-
-- **User-controlled inputs:** Tool paths, patches, file contents, search patterns, and workspace selection passed into the tool dispatcher.
-
-- **Current safeguards:** [`src/tool_execution.py`](../src/tool_execution.py) resolves paths, blocks sensitive subpaths, applies allowlist containment, and tightens paths to the active workspace when one is bound. File tools use those resolvers.
-
-- **Confirmed risks or gaps:** Intentional authority with safeguards. The file-tool policy does not sandbox the shell; treating a workspace as a whole-process containment boundary would be incorrect.
-
-- **Unverified behaviour:** Traversal, symlink, sensitive-name, absolute-path, and workspace-switch behaviour remains unverified.
-
-## Agent-controlled tool dispatch
-
-- **Boundary:** Model output becomes native or parsed tool calls and is dispatched by the agent loop.
-
-- **Available authority:** The authority of every enabled tool, including privileged built-ins and external tools.
-
-- **User-controlled inputs:** Chat content, attached/retrieved content that may influence the model, tool arguments, per-request tool selection, and policy toggles.
-
-- **Current safeguards:** [`src/tool_security.py`](../src/tool_security.py) blocks protected tools for non-admin users and fails closed for malformed tool names; [`src/tool_policy.py`](../src/tool_policy.py) supports disabled and guide-only policy; prompt-security helpers label untrusted context.
-
-- **Confirmed risks or gaps:** Credible risk requiring verification: aliases, legacy text tools, native function calls, and MCP-qualified names must all reach the same policy outcome. The code has specific alias handling for email/MCP names, which makes this a sensitive compatibility seam.
-
-- **Unverified behaviour:** The current policy outcomes for owner role, request mode, disabled state, native versus parsed invocation, qualified aliases, and external-content entry points remain unverified.
-
-## MCP and external tool servers
-
-- **Boundary:** Configured MCP servers and their tools are exposed to the agent through the MCP manager and routes.
-
-- **Available authority:** Depends on the server: external network access, local process access, messaging, or data mutation may be delegated outside the application.
-
-- **User-controlled inputs:** Server configuration, remote OAuth completion, tool arguments, and model-selected MCP calls.
-
-- **Current safeguards:** MCP routes are registered through [`routes/mcp_routes.py`](../routes/mcp_routes.py); MCP-qualified tools are denied to non-admin users by [`src/tool_security.py`](../src/tool_security.py). OAuth state and token persistence are handled in [`src/mcp_oauth.py`](../src/mcp_oauth.py).
-
-- **Confirmed risks or gaps:** Credible risk requiring verification: an MCP server authority is broader than the application can infer from its tool name. This map does not establish a trust or approval model for server installation and individual tool invocation.
-
-- **Unverified behaviour:** Server onboarding, credential storage, server-origin trust, OAuth callback deployment, tool disablement, and invocation audit behaviour remain unverified.
-
-## Outbound network requests and URL validation
-
-- **Boundary:** Search/content fetch, research, webhooks, skill import, provider endpoints, and other HTTP clients.
-
-- **Available authority:** The application can make outbound requests from its network position.
-
-- **User-controlled inputs:** Search/fetch URLs, imported skill URLs, webhook configuration, and some endpoint settings.
-
-- **Current safeguards:** [`src/url_security.py`](../src/url_security.py) validates untrusted public HTTP URLs and fails closed on unsuitable schemes or private addresses. [`services/search/content.py`](../services/search/content.py) resolves and rejects non-public hosts, pins resolved addresses for fetches, caps bodies, and limits redirects.
-
-- **Confirmed risks or gaps:** Intentional split: administrator-created model endpoints may target private providers, while untrusted URLs use public-address checks. That distinction is required for self-hosted deployments but needs explicit call-site review.
-
-- **Unverified behaviour:** The URL-source classification for outbound clients and the current handling of redirects and DNS changes remain unverified.
-
-## Secrets, credentials, and vault sessions
-
-- **Boundary:** Application-managed encrypted secrets, API keys, provider credentials, and Bitwarden/Vaultwarden CLI sessions.
-
-- **Available authority:** Credentials unlock remote providers and connected personal services.
-
-- **User-controlled inputs:** Administrative configuration, login/unlock requests, imported settings, and agent vault tool arguments.
-
-- **Current safeguards:** [`src/secret_storage.py`](../src/secret_storage.py) uses a locally stored Fernet key with restrictive permissions for supported database secrets. Vault routes require an administrator, avoid passing master passwords in command arguments, and set restrictive permissions on the vault-session file.
-
-- **Confirmed risks or gaps:** Confirmed current boundary: vault session data is persisted through the vault path, not through [`src/secret_storage.py`](../src/secret_storage.py). This is an unresolved question about current security semantics, not a confirmed exposure.
-
-- **Unverified behaviour:** Current encryption-at-rest, owner scope, rotation, lock/logout, backup/restore, and log/tool-result exposure behaviour remains unverified.
-
-## Authentication and privileged administration
-
-- **Boundary:** Session authentication, API tokens, privileged routes, and internal tool loopback.
-
-- **Available authority:** Administrative identity can access execution, settings, integrations, data deletion, and secrets.
-
-- **User-controlled inputs:** Login/signup data, session cookies, API tokens, authentication configuration, and requests to privileged routes.
-
-- **Current safeguards:** [`core/auth.py`](../core/auth.py), [`core/middleware.py`](../core/middleware.py), and route-level checks establish identity and administrator gates. [`app.py`](../app.py) warns when localhost bypass is configured; [`SECURITY.md`](../SECURITY.md) documents deployment requirements.
-
-- **Confirmed risks or gaps:** Intentional authority with safeguards. Security depends on deployments keeping authentication enabled and internal services private; this map does not audit reverse-proxy or environment configuration.
-
-- **Unverified behaviour:** Setup, anonymous, non-admin, admin, token, and internal-loopback behaviour, including privileged-route gate consistency, remains unverified.
-
-## Deletion, wipe, backup, and restore
-
-- **Boundary:** Administrative wipe, cleanup, backup import/export, and the backup restore command.
-
-- **Available authority:** Delete or replace user data and credentials.
-
-- **User-controlled inputs:** Administrative HTTP requests, cleanup choices, backup payloads, archive paths, and restore command options.
-
-- **Current safeguards:** Administrative wipe routes use the administrative boundary. Cleanup exposes a preview route before mutation. The documented backup tool requires explicit restore confirmation, stages the old data directory, and validates archive members before extraction.
-
-- **Confirmed risks or gaps:** Intentional destructive authority. Backup archives contain secrets by design, as documented in [`docs/backup-restore.md`](../docs/backup-restore.md); this is an operator confidentiality responsibility, not a code defect established here.
-
-- **Unverified behaviour:** Role-gate, confirmation, archive-rejection, staged-recovery, and owner-isolation behaviour remains unverified. No destructive runtime test was performed.
-
-## Background jobs and unattended task execution
-
-- **Boundary:** Scheduled tasks, background-job monitor, startup tasks, and notification/delivery work that continue without an active browser request.
-
-- **Available authority:** Scheduled agent work can obtain model access and, for eligible owners, shell and file tools; task output can interact with connected services.
-
-- **User-controlled inputs:** Stored task prompt, schedule, model/crew selection, enabled-tool configuration, output target, and prior persisted state.
-
-- **Current safeguards:** [`src/task_scheduler.py`](../src/task_scheduler.py) serializes execution, records task runs, associates work with an owner, and applies the agent owner-based tool gate. [`src/bg_jobs.py`](../src/bg_jobs.py) keeps bounded state and can terminate overlong subprocess jobs.
-
-- **Confirmed risks or gaps:** Credible risk requiring verification: authority is inherited and exercised later, so changes to roles, task configuration, and disabled tools must be checked at execution time rather than assumed from task creation.
-
-- **Unverified behaviour:** Creation, editing, role-change, scheduling, cancellation, restart-recovery, and execution behaviour remains unverified, including whether current policy is re-evaluated before privileged action.

discovery/system-map.md

@@ -1,139 +0,0 @@
-# Current system map
-
-> [!NOTE]
-> This non-canonical discovery map is an evidence guide, not an exhaustive feature catalog or runtime certification. Verify the cited source before relying on a finding. Each section records local implementation observations, evidence locations, confirmed current problems, and unresolved factual questions.
-
-## Navigate the system
-
-- [Startup and application composition](#startup-and-application-composition)
-- [Frontend shell and browser interaction](#frontend-shell-and-browser-interaction)
-- [Chat, sessions, and streaming](#chat-sessions-and-streaming)
-- [Agents, tools, and execution](#agents-tools-and-execution)
-- [Models, providers, and local serving](#models-providers-and-local-serving)
-- [Search and research](#search-and-research)
-- [Documents, retrieval, and personal knowledge](#documents-retrieval-and-personal-knowledge)
-- [Memory and skills](#memory-and-skills)
-- [Email, calendar, contacts, notes, and tasks](#email-calendar-contacts-notes-and-tasks)
-- [Media, speech, and image work](#media-speech-and-image-work)
-- [Authentication, secrets, and privileged administration](#authentication-secrets-and-privileged-administration)
-- [Persistence, background work, and operations](#persistence-background-work-and-operations)
-
-## Startup and application composition
-
-- **How it works:** [`app.py`](../app.py) creates the application, mounts static assets, constructs shared services, registers route factories, and owns lifespan startup and shutdown. [`src/app_initializer.py`](../src/app_initializer.py) prepares application state; [`core/`](../core/) provides persistence, authentication, middleware, sessions, and platform helpers.
-
-- **Evidence locations:** [`app.py`](../app.py); [`src/app_initializer.py`](../src/app_initializer.py); [`core/database.py`](../core/database.py); [`core/auth.py`](../core/auth.py); [`core/middleware.py`](../core/middleware.py); [`routes/`](../routes/).
-
-- **Known problems:** None recorded by this mapping.
-
-- **Open question:** Which component currently owns startup and shutdown for each long-lived service?
-
-## Frontend shell and browser interaction
-
-- **How it works:** [`static/index.html`](../static/index.html) is served by the root and SPA deep-link routes in [`app.py`](../app.py); [`static/app.js`](../static/app.js), [`static/style.css`](../static/style.css), and [`static/js/`](../static/js/) implement the client surface.
-
-- **Evidence locations:** [`static/index.html`](../static/index.html); [`static/app.js`](../static/app.js); [`static/js/`](../static/js/); [`static/style.css`](../static/style.css); [`app.py`](../app.py) deep-link handlers.
-
-- **Known problems:** The `/backgrounds` route in [`app.py`](../app.py) calls `serve_html_with_nonce` for `static/backgrounds.html`, but that file is absent from [`static/`](../static/). This is a confirmed broken prototype route, not evidence about the rest of the frontend.
-
-- **Open question:** Is `/backgrounds` currently an intentionally supported route or an obsolete prototype?
-
-## Chat, sessions, and streaming
-
-- **How it works:** [`routes/chat_routes.py`](../routes/chat_routes.py) and [`routes/chat_helpers.py`](../routes/chat_helpers.py) coordinate requests, session state, and SSE delivery. [`src/chat_handler.py`](../src/chat_handler.py), [`src/chat_processor.py`](../src/chat_processor.py), [`src/llm_core.py`](../src/llm_core.py), and [`src/session_actions.py`](../src/session_actions.py) provide message preparation, provider interaction, and session operations.
-
-- **Evidence locations:** [`routes/chat_routes.py`](../routes/chat_routes.py); [`routes/chat_helpers.py`](../routes/chat_helpers.py); [`routes/session_routes.py`](../routes/session_routes.py); [`src/chat_handler.py`](../src/chat_handler.py); [`src/chat_processor.py`](../src/chat_processor.py); [`src/llm_core.py`](../src/llm_core.py); [`core/session_manager.py`](../core/session_manager.py).
-
-- **Known problems:** [`src/agent_loop.py`](../src/agent_loop.py) annotates `_resolved_tool_event_name` with `Any` but imports no `Any` and does not enable postponed annotation evaluation. Python evaluates that annotation while importing the module, so this is an import-time defect at the checked baseline.
-
-- **Open question:** No end-to-end provider or browser streaming run was performed for this map.
-
-## Agents, tools, and execution
-
-- **How it works:** [`src/agent_loop.py`](../src/agent_loop.py) drives multi-round tool use. [`src/tool_execution.py`](../src/tool_execution.py) dispatches calls and binds workspace context. [`src/agent_tools/`](../src/agent_tools/) contains individual implementations; [`src/tool_security.py`](../src/tool_security.py) and [`src/tool_policy.py`](../src/tool_policy.py) apply role and request policies. Long-running command work is represented by [`src/bg_jobs.py`](../src/bg_jobs.py).
-
-- **Evidence locations:** [`src/agent_loop.py`](../src/agent_loop.py); [`src/tool_execution.py`](../src/tool_execution.py); [`src/agent_tools/`](../src/agent_tools/); [`src/tool_security.py`](../src/tool_security.py); [`src/tool_policy.py`](../src/tool_policy.py); [`src/tool_schemas.py`](../src/tool_schemas.py); [`src/bg_jobs.py`](../src/bg_jobs.py).
-
-- **Known problems:** The import-time annotation defect above blocks the main agent/tool path. The shell is intentionally not a filesystem or network sandbox; that is an authority boundary, not by itself a vulnerability claim.
-
-- **Open question:** Which native, legacy, and MCP-qualified invocation paths reach each policy gate?
-
-## Models, providers, and local serving
-
-- **How it works:** Model routes delegate to discovery, capabilities, endpoint resolution, and LLM core modules. Cookbook routes and hardware-fit services handle model lifecycle and local-serving support.
-
-- **Evidence locations:** [`routes/model_routes.py`](../routes/model_routes.py); [`src/model_discovery.py`](../src/model_discovery.py); [`src/model_capabilities.py`](../src/model_capabilities.py); [`src/endpoint_resolver.py`](../src/endpoint_resolver.py); [`src/llm_core.py`](../src/llm_core.py); [`routes/cookbook_routes.py`](../routes/cookbook_routes.py); [`src/cookbook_serve_lifecycle.py`](../src/cookbook_serve_lifecycle.py); [`services/hwfit/`](../services/hwfit/).
-
-- **Known problems:** None recorded by this mapping.
-
-- **Open question:** Which endpoint inputs are administrator-created and permitted to use private provider addresses?
-
-## Search and research
-
-- **How it works:** HTTP search routes use [`services/search/`](../services/search/); research is exposed through [`routes/research/`](../routes/research/) and implemented in [`services/research/`](../services/research/), [`src/deep_research.py`](../src/deep_research.py), and related helpers. [`src/search/`](../src/search/) remains an import-compatibility layer for callers not yet moved to `services.search`.
-
-- **Evidence locations:** [`routes/search_routes.py`](../routes/search_routes.py); [`services/search/`](../services/search/); [`routes/research/research_routes.py`](../routes/research/research_routes.py); [`services/research/`](../services/research/); [`src/deep_research.py`](../src/deep_research.py); [`src/search/`](../src/search/).
-
-- **Known problems:** None recorded by this mapping.
-
-- **Open question:** No live provider request was made; provider configuration and network access remain unverified.
-
-## Documents, retrieval, and personal knowledge
-
-- **How it works:** Document routes coordinate upload handling, document processing, and editor actions. Personal-document and RAG modules use Chroma and embedding clients. PDF viewing uses the optional-dependency loader in [`src/pdf_runtime.py`](../src/pdf_runtime.py); form extraction and filling live separately in [`src/pdf_forms.py`](../src/pdf_forms.py) and [`src/pdf_form_doc.py`](../src/pdf_form_doc.py).
-
-- **Evidence locations:** [`routes/document_routes.py`](../routes/document_routes.py); [`src/upload_handler.py`](../src/upload_handler.py); [`src/document_processor.py`](../src/document_processor.py); [`src/document_actions.py`](../src/document_actions.py); [`src/personal_docs.py`](../src/personal_docs.py); [`src/rag_manager.py`](../src/rag_manager.py); [`src/embeddings.py`](../src/embeddings.py); [`src/pdf_runtime.py`](../src/pdf_runtime.py); [`src/pdf_forms.py`](../src/pdf_forms.py); [`src/pdf_form_doc.py`](../src/pdf_form_doc.py).
-
-- **Known problems:** PDF viewing/runtime loading and PDF form processing are separate implementations. That separation is confirmed and intentional in the source; it is not a defect without a reported behavioural failure.
-
-- **Open question:** Optional PDF dependencies and representative uploaded documents were not exercised.
-
-## Memory and skills
-
-- **How it works:** Memory routes use [`services/memory/`](../services/memory/) and vector helpers. Skills are exposed through [`routes/skills_routes.py`](../routes/skills_routes.py), stored and managed in [`services/memory/skills.py`](../services/memory/skills.py), and may be imported through [`services/memory/skill_importer.py`](../services/memory/skill_importer.py).
-
-- **Evidence locations:** [`routes/memory/memory_routes.py`](../routes/memory/memory_routes.py); [`services/memory/`](../services/memory/); [`src/memory.py`](../src/memory.py); [`src/memory_vector.py`](../src/memory_vector.py); [`routes/skills_routes.py`](../routes/skills_routes.py); [`services/memory/skills.py`](../services/memory/skills.py); [`services/memory/skill_importer.py`](../services/memory/skill_importer.py).
-
-- **Known problems:** None recorded by this mapping.
-
-- **Open question:** Which imported skill content can reach execution-capable paths, and which validation occurs before that point?
-
-## Email, calendar, contacts, notes, and tasks
-
-- **How it works:** Dedicated route modules own email, CalDAV calendar, CardDAV contacts, notes, and tasks. Supporting modules include email helpers and pollers, CalDAV sync and writeback, and the task scheduler.
-
-- **Evidence locations:** [`routes/email_routes.py`](../routes/email_routes.py); [`routes/calendar_routes.py`](../routes/calendar_routes.py); [`routes/contacts/contacts_routes.py`](../routes/contacts/contacts_routes.py); [`routes/note/note_routes.py`](../routes/note/note_routes.py); [`routes/task_routes.py`](../routes/task_routes.py); [`routes/assistant_routes.py`](../routes/assistant_routes.py); [`src/caldav_sync.py`](../src/caldav_sync.py); [`src/caldav_writeback.py`](../src/caldav_writeback.py); [`src/task_scheduler.py`](../src/task_scheduler.py).
-
-- **Known problems:** None recorded by this mapping.
-
-- **Open question:** External account behaviour, writeback, and delivery require controlled credentials and are not runtime-validated here.
-
-## Media, speech, and image work
-
-- **How it works:** Gallery and image routes coordinate media features. Service modules own speech and media integrations; [`src/generated_images.py`](../src/generated_images.py) and [`src/visual_report.py`](../src/visual_report.py) support artifact handling and presentation.
-
-- **Evidence locations:** [`routes/gallery/gallery_routes.py`](../routes/gallery/gallery_routes.py); [`routes/stt_routes.py`](../routes/stt_routes.py); [`routes/tts_routes.py`](../routes/tts_routes.py); [`src/generated_images.py`](../src/generated_images.py); [`services/stt/`](../services/stt/); [`services/tts/`](../services/tts/); [`services/faces/`](../services/faces/); [`src/visual_report.py`](../src/visual_report.py).
-
-- **Known problems:** None recorded by this mapping.
-
-- **Open question:** Hardware- and provider-dependent media workflows were not exercised.
-
-## Authentication, secrets, and privileged administration
-
-- **How it works:** [`core/auth.py`](../core/auth.py) and [`core/middleware.py`](../core/middleware.py) provide identity and request gates. [`src/secret_storage.py`](../src/secret_storage.py) encrypts application-managed database secrets with a local Fernet key. Vault handling is separate: [`routes/vault_routes.py`](../routes/vault_routes.py) and [`src/tools/vault.py`](../src/tools/vault.py) invoke the Bitwarden CLI and persist its session data in the application data area.
-
-- **Evidence locations:** [`core/auth.py`](../core/auth.py); [`core/middleware.py`](../core/middleware.py); [`routes/auth_routes.py`](../routes/auth_routes.py); [`routes/api_token_routes.py`](../routes/api_token_routes.py); [`src/secret_storage.py`](../src/secret_storage.py); [`routes/vault_routes.py`](../routes/vault_routes.py); [`src/tools/vault.py`](../src/tools/vault.py); [`routes/admin_wipe/admin_wipe_routes.py`](../routes/admin_wipe/admin_wipe_routes.py).
-
-- **Known problems:** Vault-command handling and local application secret storage are distinct paths with different storage mechanisms. This is a source-confirmed boundary, not evidence that either path is compromised.
-
-- **Open question:** What are the current confidentiality, ownership, rotation, and backup semantics for vault session data?
-
-## Persistence, background work, and operations
-
-- **How it works:** SQLite models and persistence are centred in [`core/database.py`](../core/database.py); managers use application data paths. The scheduler and background-job monitor can continue work outside a live browser request. Operational routes cover cleanup, backup, and administrative wipe; the repository also provides a backup script and user documentation.
-
-- **Evidence locations:** [`core/database.py`](../core/database.py); [`src/runtime_paths.py`](../src/runtime_paths.py); [`src/task_scheduler.py`](../src/task_scheduler.py); [`src/bg_jobs.py`](../src/bg_jobs.py); [`src/bg_monitor.py`](../src/bg_monitor.py); [`routes/backup_routes.py`](../routes/backup_routes.py); [`routes/cleanup/cleanup_routes.py`](../routes/cleanup/cleanup_routes.py); [`routes/admin_wipe/admin_wipe_routes.py`](../routes/admin_wipe/admin_wipe_routes.py); [`scripts/odysseus-backup`](../scripts/odysseus-backup); [`docs/backup-restore.md`](../docs/backup-restore.md).
-
-- **Known problems:** None recorded by this mapping.
-
-- **Open question:** What current behaviour applies to background execution, cancellation, retries, and authority inheritance?

docs/setup.md

@@ -441,7 +441,7 @@ uv pip sync requirements.lock                          # reproduce it exactly la
 ### Outlook / Office 365 email
 Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
 and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
-passwords will fail. See [email-outlook.md](email-outlook.md) for the
+passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
 current limitation and the planned integration direction.
 
 ## Security Notes

routes/personal_routes.py

@@ -1,11 +1,13 @@
 # routes/personal_routes.py
 """Routes for personal documents management."""
+import asyncio
 import os
 import logging
 import shutil
 import uuid
 from typing import Any, Dict, List, Tuple
 from fastapi import APIRouter, HTTPException, Query, Request, UploadFile, File, Depends
+from fastapi.concurrency import run_in_threadpool
 from src.request_models import DirectoryRequest
 from core.constants import BASE_DIR, PERSONAL_DIR, PERSONAL_UPLOADS_DIR
 from src.rag_singleton import get_rag_manager
@@ -18,7 +20,6 @@ UPLOADS_DIR = PERSONAL_UPLOADS_DIR
 
 logger = logging.getLogger(__name__)
 
-
 def _personal_upload_dir_for_owner(owner: str | None, *, create: bool = True) -> str:
     """Return the per-owner upload directory used for direct RAG uploads."""
     owner_segment = secure_filename((owner or "local").strip())[:80] or "local"
@@ -141,6 +142,22 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
     """
     router = APIRouter(prefix="/api/personal")
 
+    # Serializes directory index jobs across requests. Indexing runs in the
+    # threadpool (#5558), so concurrent requests would otherwise run in parallel
+    # and race PersonalDocsManager's unsynchronized list mutations and file
+    # writes; before the threadpool move they serialized on the blocked event
+    # loop, so one-at-a-time is behavior parity.
+    #
+    # An asyncio.Lock acquired in the async handler BEFORE offloading: a waiting
+    # request parks on the event loop instead of pinning a threadpool worker (an
+    # earlier threading.Lock taken INSIDE the worker meant queued jobs held pool
+    # tokens while blocked, starving every other run_in_threadpool caller).
+    # add/remove/reload all take this lock, so their mutations never interleave.
+    # Per-router (not module-global) so each app binds it to its own event loop.
+    # Scope is the single process: multi-worker deployments would need a shared
+    # lock (out of scope for #5558).
+    _index_job_lock = asyncio.Lock()
+
     def _rag():
         """Get the current RAG manager, retrying init if needed."""
         return get_rag_manager()
@@ -172,8 +189,12 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
         return {"files": files, "directories": directories}
     
     @router.post("/reload")
-    def api_personal_reload(owner: str = Depends(require_user), _admin: None = Depends(require_admin)):
-        personal_docs_manager.refresh_index()
+    async def api_personal_reload(owner: str = Depends(require_user), _admin: None = Depends(require_admin)):
+        # refresh_index() re-extracts text across every tracked directory —
+        # blocking work. Take the shared job lock (so it cannot race an add /
+        # remove) and run it off the event loop.
+        async with _index_job_lock:
+            await run_in_threadpool(personal_docs_manager.refresh_index)
         return {"ok": True, "count": len(personal_docs_manager.index)}
     
     @router.post("/add_directory")
@@ -207,12 +228,26 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
             # Use the RAGManager to index the directory
             rag = _rag()
             if rag:
-                result = rag.index_personal_documents(directory, owner=owner)
-                
+                def _index_directory():
+                    result = rag.index_personal_documents(directory, owner=owner)
+                    if result["success"]:
+                        # Also update the personal_docs_manager to track this
+                        # directory. Kept inside the offloaded call: it triggers
+                        # refresh_index(), which re-extracts text across tracked
+                        # directories.
+                        personal_docs_manager.add_directory(directory, index=False)
+                    return result
+
+                # Indexing walks, embeds, and stores the whole tree — minutes
+                # on a real directory. The handler is async, so calling it
+                # inline runs it on the event loop and every other request
+                # queues behind it until it finishes (#5558). Serialize on the
+                # async job lock BEFORE offloading so a queued request parks on
+                # the loop instead of pinning a threadpool worker.
+                async with _index_job_lock:
+                    result = await run_in_threadpool(_index_directory)
+
                 if result["success"]:
-                    # Also update the personal_docs_manager to track this directory
-                    personal_docs_manager.add_directory(directory, index=False)
-                    
                     return {
                         "success": True,
                         "message": f"Successfully indexed {result['indexed_count']} chunks from {directory}",
@@ -251,17 +286,25 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
 
             logger.info(f"Removing directory from RAG: {directory}")
 
-            # Always remove from personal_docs_manager tracking
-            if hasattr(personal_docs_manager, 'remove_directory'):
-                personal_docs_manager.remove_directory(directory)
-
-            # Remove from RAG vector store (best-effort)
             rag = _rag()
-            if rag:
-                try:
-                    rag.remove_directory(directory)
-                except Exception as e:
-                    logger.warning(f"RAG removal failed for directory {directory}: {e}")
+
+            def _remove_directory():
+                # Always remove from personal_docs_manager tracking. This
+                # mutates the same unsynchronized list/index an add job touches
+                # and re-extracts text (refresh_index), so it is blocking work.
+                if hasattr(personal_docs_manager, 'remove_directory'):
+                    personal_docs_manager.remove_directory(directory)
+                # Remove from RAG vector store (best-effort).
+                if rag:
+                    try:
+                        rag.remove_directory(directory)
+                    except Exception as e:
+                        logger.warning(f"RAG removal failed for directory {directory}: {e}")
+
+            # Same job lock as add/reload so remove cannot interleave with an
+            # in-flight add; offloaded off the event loop.
+            async with _index_job_lock:
+                await run_in_threadpool(_remove_directory)
 
             return {
                 "success": True,

src/agent_loop.py

@@ -12,7 +12,7 @@ import json
 import re
 import time
 import logging
-from typing import AsyncGenerator, List, Dict, Optional, Set
+from typing import Any, AsyncGenerator, List, Dict, Optional, Set
 from urllib.parse import urlparse
 
 from src.llm_core import (

tests/test_add_directory_event_loop.py

@@ -0,0 +1,253 @@
+"""Regression guard for #5558 — POST /api/personal/add_directory must not run
+the indexing job on the event loop.
+
+The handler is ``async def`` but called ``rag.index_personal_documents``
+(os.walk + file reads + per-chunk embedding + Chroma inserts) inline, so
+FastAPI ran the whole job on the event loop and every other request queued
+behind it: indexing a real directory froze the UI and API for 25+ minutes.
+``personal_docs_manager.add_directory`` sits in the same blocking section — it
+triggers ``refresh_index()``, which re-extracts text across tracked dirs.
+
+These tests build the real router with fake managers and compare the thread
+the indexing work runs on against the event loop's thread.
+"""
+import asyncio
+import os
+import threading
+
+os.environ.setdefault("DATABASE_URL", "sqlite:///:memory:")
+
+import httpx
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+
+def _serialization_probe():
+    """Shared counter proving two critical sections never overlap."""
+    state = {"active": 0, "max_active": 0}
+    lock = threading.Lock()
+
+    def enter():
+        with lock:
+            state["active"] += 1
+            state["max_active"] = max(state["max_active"], state["active"])
+
+    def leave():
+        with lock:
+            state["active"] -= 1
+
+    return state, enter, leave
+
+
+# Concurrency tests are `async def` (pyproject asyncio_mode="auto") and drive the
+# ASGI app through httpx.ASGITransport + AsyncClient + asyncio.gather, NOT starlette
+# TestClient + ThreadPoolExecutor: the job lock is an asyncio.Lock acquired in the
+# async handler, and TestClient's portal-thread dispatch deadlocks against it (same
+# reason test_notes_fail_closed_auth.py uses ASGITransport). asyncio.gather runs both
+# requests on the test's own loop.
+def _async_client(app):
+    return httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://t")
+
+import routes.personal_routes as personal_routes
+from core.middleware import require_admin
+from src.auth_helpers import require_user
+
+
+class _FakeRag:
+    def __init__(self, record):
+        self._record = record
+
+    def index_personal_documents(self, directory, owner=None):
+        self._record["index_thread"] = threading.get_ident()
+        return {"success": True, "indexed_count": 3, "failed_count": 0}
+
+
+class _FakeDocsManager:
+    def __init__(self, record):
+        self._record = record
+        self.index = []
+
+    def add_directory(self, directory, *, index=True, owner=None):
+        self._record["bookkeeping_thread"] = threading.get_ident()
+        self._record["bookkeeping_index_flag"] = index
+
+
+def _build_app(tmp_path, monkeypatch, record):
+    monkeypatch.setattr(personal_routes, "PERSONAL_DIR", str(tmp_path))
+    monkeypatch.setattr(personal_routes, "get_rag_manager", lambda: _FakeRag(record))
+
+    app = FastAPI()
+    app.include_router(
+        personal_routes.setup_personal_routes(_FakeDocsManager(record), None, True)
+    )
+    app.dependency_overrides[require_user] = lambda: "tester"
+    app.dependency_overrides[require_admin] = lambda: None
+
+    @app.get("/loop-thread")
+    async def loop_thread_probe():
+        return {"thread": threading.get_ident()}
+
+    return app
+
+
+def test_indexing_runs_off_the_event_loop(tmp_path, monkeypatch):
+    record = {}
+    app = _build_app(tmp_path, monkeypatch, record)
+    target = tmp_path / "docs"
+    target.mkdir()
+
+    # Context-manager client: one portal/event loop serves both requests, so
+    # the probe and the POST are guaranteed to see the same loop thread.
+    with TestClient(app) as client:
+        loop_thread = client.get("/loop-thread").json()["thread"]
+        resp = client.post(
+            "/api/personal/add_directory", json={"directory": str(target)}
+        )
+
+    assert resp.status_code == 200
+    assert record["index_thread"] != loop_thread, (
+        "index_personal_documents ran on the event loop thread — every other "
+        "request queues behind the indexing job (#5558)"
+    )
+    assert record["bookkeeping_thread"] != loop_thread, (
+        "personal_docs_manager.add_directory (refresh_index) ran on the event "
+        "loop thread"
+    )
+
+
+def test_response_and_bookkeeping_unchanged(tmp_path, monkeypatch):
+    record = {}
+    app = _build_app(tmp_path, monkeypatch, record)
+    target = tmp_path / "docs"
+    target.mkdir()
+
+    client = TestClient(app)
+    resp = client.post("/api/personal/add_directory", json={"directory": str(target)})
+
+    assert resp.status_code == 200
+    body = resp.json()
+    assert body["success"] is True
+    assert body["indexed_count"] == 3
+    assert body["failed_count"] == 0
+    assert body["directory"] == os.path.realpath(str(target))
+    assert record["bookkeeping_index_flag"] is False
+
+
+async def test_concurrent_add_directory_requests_serialize_indexing(tmp_path, monkeypatch):
+    """Off-loop execution must not mean parallel index jobs: concurrent
+    requests would race PersonalDocsManager's unsynchronized list mutations
+    and file writes (save_directories/_save_excluded are plain open('w'))."""
+    import time
+
+    state, enter, leave = _serialization_probe()
+
+    def _slow_index(self, directory, owner=None):
+        enter(); time.sleep(0.2); leave()
+        return {"success": True, "indexed_count": 1, "failed_count": 0}
+
+    monkeypatch.setattr(_FakeRag, "index_personal_documents", _slow_index)
+
+    record = {}
+    app = _build_app(tmp_path, monkeypatch, record)
+    for name in ("docs_a", "docs_b"):
+        (tmp_path / name).mkdir()
+
+    async with _async_client(app) as ac:
+        results = await asyncio.gather(
+            ac.post("/api/personal/add_directory", json={"directory": str(tmp_path / "docs_a")}),
+            ac.post("/api/personal/add_directory", json={"directory": str(tmp_path / "docs_b")}),
+        )
+
+    assert all(r.status_code == 200 for r in results)
+    assert state["max_active"] == 1, (
+        f"{state['max_active']} index jobs ran in parallel — concurrent "
+        "add_directory requests must serialize"
+    )
+
+
+def test_failed_indexing_still_returns_500(tmp_path, monkeypatch):
+    record = {}
+    app = _build_app(tmp_path, monkeypatch, record)
+    target = tmp_path / "docs"
+    target.mkdir()
+
+    def _fail(directory, owner=None):
+        return {"success": False, "message": "boom"}
+
+    monkeypatch.setattr(_FakeRag, "index_personal_documents", staticmethod(_fail))
+
+    client = TestClient(app)
+    resp = client.post("/api/personal/add_directory", json={"directory": str(target)})
+    assert resp.status_code == 500
+    assert "boom" in resp.json()["detail"]
+
+
+async def test_add_and_remove_serialize(tmp_path, monkeypatch):
+    """#5634: remove must hold the SAME job lock as add. Otherwise a remove
+    running while an add job is in flight races PersonalDocsManager's
+    unsynchronized list/index mutations — the inconsistent state the PR's
+    'add/remove are serialized' guarantee claims to prevent."""
+    import time
+
+    state, enter, leave = _serialization_probe()
+
+    def _slow_index(self, directory, owner=None):
+        enter(); time.sleep(0.25); leave()
+        return {"success": True, "indexed_count": 1, "failed_count": 0}
+
+    def _slow_remove(self, directory):
+        enter(); time.sleep(0.25); leave()
+
+    monkeypatch.setattr(_FakeRag, "index_personal_documents", _slow_index)
+    monkeypatch.setattr(_FakeDocsManager, "remove_directory", _slow_remove, raising=False)
+
+    record = {}
+    app = _build_app(tmp_path, monkeypatch, record)
+    (tmp_path / "docs_a").mkdir()
+    (tmp_path / "docs_b").mkdir()
+
+    async with _async_client(app) as ac:
+        results = await asyncio.gather(
+            ac.post("/api/personal/add_directory", json={"directory": str(tmp_path / "docs_a")}),
+            ac.delete("/api/personal/remove_directory", params={"directory": str(tmp_path / "docs_b")}),
+        )
+
+    assert all(r.status_code == 200 for r in results)
+    assert state["max_active"] == 1, (
+        f"{state['max_active']} add/remove critical sections overlapped — "
+        "remove must hold the same index job lock as add"
+    )
+
+
+async def test_reload_serializes_with_add(tmp_path, monkeypatch):
+    """#5634: POST /reload rebuilds the index via refresh_index(); it must hold
+    the same job lock so it cannot race an in-flight add job."""
+    import time
+
+    state, enter, leave = _serialization_probe()
+
+    def _slow_index(self, directory, owner=None):
+        enter(); time.sleep(0.25); leave()
+        return {"success": True, "indexed_count": 1, "failed_count": 0}
+
+    def _slow_refresh(self):
+        enter(); time.sleep(0.25); leave()
+
+    monkeypatch.setattr(_FakeRag, "index_personal_documents", _slow_index)
+    monkeypatch.setattr(_FakeDocsManager, "refresh_index", _slow_refresh, raising=False)
+
+    record = {}
+    app = _build_app(tmp_path, monkeypatch, record)
+    (tmp_path / "docs_a").mkdir()
+
+    async with _async_client(app) as ac:
+        results = await asyncio.gather(
+            ac.post("/api/personal/add_directory", json={"directory": str(tmp_path / "docs_a")}),
+            ac.post("/api/personal/reload"),
+        )
+
+    assert all(r.status_code == 200 for r in results)
+    assert state["max_active"] == 1, (
+        f"{state['max_active']} add/reload critical sections overlapped — "
+        "reload must hold the same index job lock as add"
+    )

tests/test_provider_endpoints_url_building.py

@@ -1,8 +1,8 @@
 """Provider endpoint URL-building tests.
 
-Covers ``build_chat_url`` and ``build_models_url`` for the provider families
-exercised by this module: Anthropic, Gemini, Groq, xAI, OpenRouter, OpenAI,
-DeepSeek, and Ollama (local and cloud).
+Covers ``build_chat_url`` and ``build_models_url`` for every provider named in
+ROADMAP.md: Anthropic, Gemini, Groq, xAI, OpenRouter, OpenAI, DeepSeek, Ollama
+(local + cloud).
 """
 import pytest