perf(chat): batch live thinking rendering and bound timer updates (#5931)

* perf(chat): batch live thinking DOM updates

* test(chat): cover live thinking scheduler lifecycle

* fix(chat): guard background stop-state, restore live thinking text, drop source-text tests

- _closeOpenThinkingMarkup no longer overwrites currentAccumulated for
  backgrounded streams. It now mirrors the guard the delta path already uses
  (`if (!_isBg) currentAccumulated = accumulated`). Without it a backgrounded
  stream's text is written into the foreground session's stop-state, which
  abortCurrentRequest and detachCurrentStream then put in the wrong bubble.

- Split _extractLiveThinkingText into _liveThinkingText (strip every think tag)
  and _closedThinkingText (via extractThinkingBlocks). Slicing from the first
  <think> to the first </think> pinned the live box to "The" for the rest of the
  stream on the `<think>The</think>` + untagged-thinking pattern that the
  hasUnclosedThink detection deliberately keeps streaming through.

- The background transition now flushes with rich:true, so a stream that
  backgrounds mid-thinking isn't left as pre-wrap plain text permanently.

- Move the throttle to static/js/liveThinkingThrottle.js and import it. The
  .mjs suite imports the module instead of slicing it out of chat.js with
  vm.runInNewContext and marker comments.

- Replace the source-text assertions in tests/test_live_thinking_scheduler_js.py
  with behavioral coverage, per tests/TESTING_STANDARD.md. The .mjs suite grows
  from 3 to 6 cases.

- Collapse the duplicated tool_start/agent_step finalizers into one
  _endLiveThinkingSection().

* fix(chat): hoist thinking teardown out of the try block so catch can reach it

In an ES module a function declared inside `try { }` is scoped to that block,
and `catch` is a sibling scope rather than a nested one. _closeOpenThinkingMarkup
was declared inside the try and called from catch, so the call threw
ReferenceError and killed the rest of the error path: the stream never
finalized and the thinking block was never torn down.

Declare _closeOpenThinkingMarkup and a new _endThinkingOnTerminalPath next to
the existing _flushLiveThinking / _cancelLiveThinkingWork outer lets and assign
them inside the try, which is the pattern those two already use for exactly
this reason.

Verified against a live stream in a browser: before, clicking stop mid-thinking
logged "_closeOpenThinkingMarkup is not defined" and left no finalized thinking
section; after, the block collapses to "View thinking process" correctly.

* perf(chat): extract live thinking at commit cadence

* fix(chat): bound live thinking work

* test(chat): update stream invariant assertions

---------

Co-authored-by: Léo <leograndcontact@gmail.com>

.env.example

@@ -189,6 +189,7 @@ SEARXNG_INSTANCE=http://localhost:8080
 # ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES=26214400       # email compose attachment (25 MB)
 # ODYSSEUS_STT_MAX_AUDIO_BYTES=26214400                  # speech-to-text audio (25 MB)
 # ODYSSEUS_ICS_MAX_BYTES=10485760                        # calendar .ics import (10 MB)
+# ODYSSEUS_TTS_CACHE_MAX_BYTES=524288000                 # TTS cache (500 MB)
 
 # ============================================================
 # Host Docker access (explicit opt-in)

.github/scripts/check-issue-description.js

@@ -153,6 +153,16 @@ module.exports = async ({ github, context, core }) => {
     }
   }
 
+  const LABEL_BAD  = 'needs more info';
+  const LABEL_GOOD = 'ready for review';
+
+  // Closed issues are no longer awaiting review.
+  // This also prevents later edits to closed issues from restoring the label.
+  if (issue.state === 'closed') {
+    await dropLabel(LABEL_GOOD);
+    return;
+  }
+
   // ── Find existing bot comment to update in-place ──────────────────────────
   const MARKER = '<!-- issue-description-check -->';
   const { data: comments } = await github.rest.issues.listComments({
@@ -160,9 +170,6 @@ module.exports = async ({ github, context, core }) => {
   });
   const existing = comments.find(c => c.user.type === 'Bot' && c.body.includes(MARKER));
 
-  const LABEL_BAD  = 'needs more info';
-  const LABEL_GOOD = 'ready for review';
-
   if (failures.length === 0) {
     if (existing) {
       await github.rest.issues.deleteComment({ owner, repo, comment_id: existing.id });

.github/workflows/issue-description-check.yml

@@ -2,7 +2,7 @@ name: ci / issue description check
 
 on:
   issues:
-    types: [opened, edited, reopened]
+    types: [opened, edited, reopened, closed]
 
 permissions:
   issues: write

app.py

@@ -692,7 +692,7 @@ from routes.history.history_routes import setup_history_routes
 app.include_router(setup_history_routes(session_manager, upload_handler=upload_handler))
 
 # Search
-from routes.search_routes import setup_search_routes
+from routes.search.search_routes import setup_search_routes
 app.include_router(setup_search_routes(config))
 
 # Presets
@@ -739,7 +739,7 @@ app.include_router(setup_stt_routes(stt_service))
 logger.info("STT service initialized (provider managed via settings)")
 
 # Documents (artifacts/canvas)
-from routes.document_routes import setup_document_routes
+from routes.document.document_routes import setup_document_routes
 document_router = setup_document_routes(session_manager, upload_handler)
 app.include_router(document_router)
 
@@ -820,7 +820,7 @@ set_ai_rag_manager(rag_manager, personal_docs_mgr)
 logger.info("AI interaction tools initialized (session, memory, RAG, UI control)")
 
 # Webhooks
-from routes.webhook_routes import setup_webhook_routes
+from routes.webhook.webhook_routes import setup_webhook_routes
 app.include_router(setup_webhook_routes(webhook_manager, auth_manager, session_manager, api_key_manager))
 
 # API Tokens
@@ -852,7 +852,7 @@ app.include_router(setup_codex_routes(
 ))
 app.include_router(setup_claude_routes())
 
-from routes.vault_routes import setup_vault_routes
+from routes.vault.vault_routes import setup_vault_routes
 app.include_router(setup_vault_routes())
 
 # Contacts (CardDAV)

core/session_manager.py

@@ -194,7 +194,12 @@ class SessionManager:
             is_important=getattr(db_session, 'is_important', False) or False,
         )
 
-        session.message_count = getattr(db_session, 'message_count', len(history))
+        # The rows just loaded are the whole transcript, so they — not the
+        # denormalized sessions.message_count column — are the truth for this
+        # cached object. get_session's hydration gate compares against this
+        # number; seeding it from a drifted column would ask for a reload that
+        # can never close the gap.
+        session.message_count = len(history)
         return session
 
     # ------------------------------------------------------------------
@@ -398,30 +403,50 @@ class SessionManager:
     # ------------------------------------------------------------------
 
     def get_session(self, session_id: str) -> Session:
-        """Get a session by ID, loading from DB if needed.
+        """Get a session by ID, loading complete DB history when needed.
 
-        Sessions seeded by `load_sessions` start with empty history. The
-        first read here hydrates them with the message rows.
+        Sessions seeded by ``load_sessions`` start with empty history, and a
+        cached session can also become partially stale. Refresh metadata first,
+        then hydrate whenever the cached transcript is short of the stored rows.
+        Model-send routes enter through this method before building context,
+        while paginated display history reads SQLite directly.
+
+        The gate compares against ``sync_session_metadata``'s reconciled count
+        (the real ``chat_messages`` total), never the denormalized column, so a
+        hydrate always closes the gap and the next read is a cache hit.
         """
         if session_id not in self.sessions:
             self._load_session_from_db(session_id)
-        else:
-            cached = self.sessions[session_id]
-            # Lazy hydrate: metadata-only entries get their messages on first read.
-            if not cached.history and getattr(cached, "message_count", 0) > 0:
-                self._load_session_from_db(session_id)
 
         # Keep model/endpoint metadata fresh. Endpoint deletion can clear the
-        # DB row while a session object is still cached in RAM.
+        # DB row while a session object is still cached in RAM. Refreshing first
+        # also exposes the authoritative message count before completeness is
+        # checked.
         self.sync_session_metadata(session_id)
 
+        cached = self.sessions[session_id]
+        cached_count = len(cached.history or [])
+        stored_count = int(getattr(cached, "message_count", 0) or 0)
+        if cached_count < stored_count:
+            self._load_session_from_db(session_id)
+
         # Update last_accessed
         self._touch_session(session_id)
 
         return self.sessions[session_id]
 
     def sync_session_metadata(self, session_id: str) -> bool:
-        """Refresh non-message session fields from the DB into the cached object."""
+        """Refresh non-message session fields from the DB into the cached object.
+
+        ``message_count`` is reconciled against the real ``chat_messages`` rows
+        rather than copied from the denormalized ``sessions.message_count``
+        column. That column drifts in normal operation — ``_persist_message``
+        swallows a failed insert but ``add_message`` has already appended in
+        memory, so the next successful persist writes rows+1, and a persist for
+        an uncached session writes 0. Hydration keys off this number: a
+        drifted-high column would reload the whole transcript on every warm
+        read, and a drifted-low one would leave the model a truncated one.
+        """
         session = self.sessions.get(session_id)
         if session is None:
             return False
@@ -444,7 +469,11 @@ class SessionManager:
             session.archived = db_session.archived
             session.owner = getattr(db_session, "owner", None)
             session.is_important = getattr(db_session, "is_important", False) or False
-            session.message_count = getattr(db_session, "message_count", session.message_count) or 0
+            session.message_count = (
+                db.query(DbChatMessage)
+                .filter(DbChatMessage.session_id == session_id)
+                .count()
+            )
             return True
         except Exception as e:
             logger.error(f"Error syncing session metadata {session_id}: {e}")

docker-compose.gpu-amd.yml

@@ -67,6 +67,7 @@ services:
       - ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES=${ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES:-26214400}
       - ODYSSEUS_STT_MAX_AUDIO_BYTES=${ODYSSEUS_STT_MAX_AUDIO_BYTES:-26214400}
       - ODYSSEUS_ICS_MAX_BYTES=${ODYSSEUS_ICS_MAX_BYTES:-10485760}
+      - ODYSSEUS_TTS_CACHE_MAX_BYTES=${ODYSSEUS_TTS_CACHE_MAX_BYTES}
       - DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
       - GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
       - GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}

docker-compose.gpu-nvidia.yml

@@ -66,6 +66,7 @@ services:
       - ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES=${ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES:-26214400}
       - ODYSSEUS_STT_MAX_AUDIO_BYTES=${ODYSSEUS_STT_MAX_AUDIO_BYTES:-26214400}
       - ODYSSEUS_ICS_MAX_BYTES=${ODYSSEUS_ICS_MAX_BYTES:-10485760}
+      - ODYSSEUS_TTS_CACHE_MAX_BYTES=${ODYSSEUS_TTS_CACHE_MAX_BYTES}
       - DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
       - GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
       - GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}

docker-compose.yml

@@ -55,6 +55,7 @@ services:
       - ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES=${ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES:-26214400}
       - ODYSSEUS_STT_MAX_AUDIO_BYTES=${ODYSSEUS_STT_MAX_AUDIO_BYTES:-26214400}
       - ODYSSEUS_ICS_MAX_BYTES=${ODYSSEUS_ICS_MAX_BYTES:-10485760}
+      - ODYSSEUS_TTS_CACHE_MAX_BYTES=${ODYSSEUS_TTS_CACHE_MAX_BYTES}
       - DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
       - GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
       - GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}

mcp_servers/memory_server.py

@@ -17,6 +17,8 @@ from mcp.types import Tool, TextContent
 
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
 
+from src.memory import MemoryStoreUnreadable
+
 server = Server("memory")
 
 # Late-initialized managers (set during first tool call)
@@ -29,6 +31,10 @@ _OWNER_SCOPE_ERROR = (
     "Error: Memory MCP owner is not configured for an owner-scoped memory store. "
     "Set ODYSSEUS_MCP_MEMORY_OWNER for this server or use the owner-aware native memory tool."
 )
+_UNREADABLE_STORE_ERROR = (
+    "Error: Memory store is temporarily unreadable — nothing was saved. "
+    "Repair or restore memory.json, then retry."
+)
 
 
 def _configured_owner() -> str | None:
@@ -51,9 +57,21 @@ def _owner_scoped_store(entries: list[dict]) -> bool:
     return any(_entry_owner(entry) for entry in entries if isinstance(entry, dict))
 
 
-def _scope_entries() -> tuple[str | None, list[dict], list[dict], str | None]:
-    """Return configured owner, all entries, visible entries, and optional error."""
-    entries = _memory_manager.load_all()
+def _scope_entries(for_update: bool = False) -> tuple[str | None, list[dict], list[dict], str | None]:
+    """Return configured owner, all entries, visible entries, and optional error.
+
+    ``for_update=True`` is for read-modify-write callers. They save the ``all
+    entries`` list back, so an unreadable store must be reported as an error
+    instead of degrading to ``[]`` — otherwise the save writes their one new
+    entry over the whole store (issue #5673).
+    """
+    if for_update:
+        try:
+            entries = _memory_manager.load_all_for_update()
+        except MemoryStoreUnreadable as e:
+            return None, [], [], f"{_UNREADABLE_STORE_ERROR} ({e})"
+    else:
+        entries = _memory_manager.load_all()
     owner = _configured_owner()
     if owner is None and _owner_scoped_store(entries):
         return None, entries, [], _OWNER_SCOPE_ERROR
@@ -161,7 +179,7 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
         category = arguments.get("category", "fact")
         if not text:
             return _text_result("Error: Memory text cannot be empty")
-        owner, memories, _visible, scope_error = _scope_entries()
+        owner, memories, _visible, scope_error = _scope_entries(for_update=True)
         if scope_error:
             return _text_result(scope_error)
         entry = _memory_manager.add_entry(text, source="ai_agent", category=category, owner=owner)

requirements.txt

@@ -38,7 +38,10 @@ python-dateutil
 caldav
 cryptography
 bcrypt
-mcp
+# Built-in servers use the v1 low-level Server decorator API. MCP SDK v2 is a
+# breaking rewrite, so keep fresh installs on the maintained v1 line until the
+# servers are migrated together.
+mcp<2
 pyotp
 qrcode[pil]
 croniter

routes/backup_routes.py

@@ -6,6 +6,7 @@ from datetime import datetime
 
 from fastapi import APIRouter, HTTPException, Request, Response
 from core.middleware import require_admin
+from services.memory import MemoryStoreUnreadable
 from src.auth_helpers import get_current_user
 from src.settings import load_settings, save_settings, load_features, save_features
 
@@ -76,7 +77,15 @@ def setup_backup_routes(memory_manager, preset_manager, skills_manager) -> APIRo
 
         # ── Memories ──
         if "memories" in body and isinstance(body["memories"], list):
-            existing = memory_manager.load_all()
+            # Strict load: importing on top of an unreadable store would write
+            # only the incoming rows back and drop everything already saved.
+            try:
+                existing = memory_manager.load_all_for_update()
+            except MemoryStoreUnreadable as e:
+                logger.error("Refusing to import memories: %s", e)
+                raise HTTPException(
+                    503, "Memory store is temporarily unreadable — nothing was imported."
+                )
             # Dedup against THIS user's own memories only. Using every tenant's
             # rows (load_all) meant a memory whose text matched any other
             # user's was silently skipped, so the importing user lost their own

routes/document/__init__.py

@@ -0,0 +1,6 @@
+"""Document route domain package (slice 2m, #4082/#4071).
+
+Contains document_routes.py and document_helpers.py, migrated from the flat
+routes/ directory. Backward-compat shims at routes/document_routes.py and
+routes/document_helpers.py re-export from here.
+"""

routes/document/document_helpers.py

@@ -0,0 +1,243 @@
+"""document_helpers.py — Pydantic models, doc serializers, owner gating, file-locator helpers shared with document_routes.py."""
+
+"""Document routes — CRUD for living documents with version history."""
+
+import logging
+import os
+import re
+from typing import Any, Dict, Optional
+
+from fastapi import HTTPException, Request
+from pydantic import BaseModel
+
+from core.database import Document, DocumentVersion
+from core.database import Session as DbSession
+from src.auth_helpers import _auth_disabled
+from src.upload_handler import UploadHandler
+
+logger = logging.getLogger(__name__)
+
+
+# ---- Request schemas ----
+
+class DocumentCreate(BaseModel):
+    session_id: Optional[str] = None
+    title: str = "Untitled"
+    language: Optional[str] = None
+    content: str = ""
+
+class DocumentUpdate(BaseModel):
+    content: str
+    summary: Optional[str] = None
+    force_version: bool = False
+
+class DocumentPatch(BaseModel):
+    title: Optional[str] = None
+    language: Optional[str] = None
+    session_id: Optional[str] = None  # link/unlink document to a session
+
+
+# ---- Helpers ----
+
+def _doc_to_dict(doc: Document) -> Dict[str, Any]:
+    return {
+        "id": doc.id,
+        "session_id": doc.session_id,
+        "title": doc.title,
+        "language": doc.language,
+        "current_content": doc.current_content,
+        "version_count": doc.version_count,
+        "is_active": doc.is_active,
+        "archived": bool(getattr(doc, "archived", False)),
+        "created_at": (doc.created_at.isoformat() + "Z") if doc.created_at else None,
+        "updated_at": (doc.updated_at.isoformat() + "Z") if doc.updated_at else None,
+        # Source-email provenance (set when doc was created from an email
+        # attachment) — drives the "Send signed reply" menu item.
+        "source_email_uid":        getattr(doc, "source_email_uid", None),
+        "source_email_folder":     getattr(doc, "source_email_folder", None),
+        "source_email_account_id": getattr(doc, "source_email_account_id", None),
+        "source_email_message_id": getattr(doc, "source_email_message_id", None),
+    }
+
+def _version_to_dict(v: DocumentVersion) -> Dict[str, Any]:
+    return {
+        "id": v.id,
+        "document_id": v.document_id,
+        "version_number": v.version_number,
+        "content": v.content,
+        "summary": v.summary,
+        "source": v.source,
+        "created_at": v.created_at.isoformat() if v.created_at else None,
+    }
+
+
+def _verify_doc_owner(db, doc: Document, user: str):
+    """Verify `user` owns this document. Raise 404 if not.
+
+    Documents now carry their own `owner` column, so a doc whose session
+    was deleted (session_id → NULL) can still prove ownership and stay
+    openable / cloneable. We trust that column first and only fall back to
+    the session join for any not-yet-backfilled legacy row.
+    """
+    if user is None:
+        if _auth_disabled():
+            return  # Single-user / no-auth mode: allow access
+        raise HTTPException(403, "Authentication required")
+    if doc.owner is not None:
+        if doc.owner != user:
+            raise HTTPException(404, "Document not found")
+        return
+    # Legacy fallback: derive ownership from the linked session.
+    if not doc.session_id:
+        raise HTTPException(404, "Document not found")
+    session = db.query(DbSession).filter(DbSession.id == doc.session_id).first()
+    if not session or session.owner != user:
+        raise HTTPException(404, "Document not found")
+
+
+def _owner_session_filter(q, user):
+    """Restrict a documents query to those owned by `user`.
+
+    Documents now carry their own `owner` column (backfilled at boot from
+    the linked session, or assigned to the admin user for legacy/orphaned
+    docs). We filter on that directly rather than on a session join, so a
+    document whose session was deleted (session_id → NULL) still shows up
+    for its owner instead of silently vanishing from the Library + search.
+
+    The owner backfill runs in init_db before the app serves requests, so
+    by the time this filter is live there are no NULL-owner rows to leak;
+    we therefore match the owner strictly for authenticated callers."""
+    if not user:
+        if user == "" or _auth_disabled():
+            return q
+        return q.filter(False)
+    return q.filter(Document.owner == user)
+
+
+
+def _slug(name: str) -> str:
+    """Filesystem-friendly version of a document title.
+
+    Whitespace becomes underscores; other unsafe punctuation is dropped.
+    Preserves letters, digits, dot, hyphen, underscore. Idempotent.
+    """
+    import re as _re
+    s = (name or "").strip()
+    # Drop the trailing extension if the title happens to include one
+    s = _re.sub(r'\.pdf$', '', s, flags=_re.IGNORECASE)
+    s = _re.sub(r'\s+', '_', s)
+    s = _re.sub(r'[^A-Za-z0-9._-]', '', s)
+    s = _re.sub(r'_+', '_', s).strip('_')
+    return s or "form"
+
+
+# DPI scale for the interactive PDF view. ~150 DPI (2x of 72 PDF user-units).
+_PDF_RENDER_SCALE = 2.0
+
+
+def _upload_path_inside(upload_dir: str, path: str) -> bool:
+    base = os.path.realpath(upload_dir)
+    p = os.path.realpath(path)
+    try:
+        return os.path.commonpath([base, p]) == base
+    except Exception:
+        return False
+
+
+def _resolve_user_upload_path(
+    upload_handler: Any,
+    upload_id: str,
+    owner: Optional[str],
+    auth_manager=None,
+) -> Optional[str]:
+    """Resolve an upload id to a filesystem path the caller may read."""
+    if upload_handler is None:
+        return None
+    resolved = upload_handler.resolve_upload(
+        upload_id,
+        owner=owner,
+        auth_manager=auth_manager,
+    )
+    if not isinstance(resolved, dict) or not resolved:
+        return None
+    path = resolved.get("path")
+    upload_dir = getattr(upload_handler, "upload_dir", None)
+    if path and upload_dir and not _upload_path_inside(upload_dir, path):
+        logger.warning("Upload path outside upload directory: %s", path)
+        return None
+    return path
+
+
+def _locate_upload(
+    upload_dir: str,
+    file_id: str,
+    owner: Optional[str] = None,
+    auth_manager=None,
+    upload_handler: Any = None,
+):
+    """Find an upload by its filename ID via UploadHandler.resolve_upload."""
+    if upload_handler is None:
+        from src.upload_handler import UploadHandler
+
+        base_dir = os.path.dirname(os.path.abspath(upload_dir))
+        upload_handler = UploadHandler(base_dir, upload_dir)
+    return _resolve_user_upload_path(upload_handler, file_id, owner, auth_manager)
+
+
+def _assert_pdf_marker_upload_owned(
+    request: Request,
+    content: str,
+    user: Optional[str],
+    upload_handler: Any,
+) -> None:
+    """Reject document content whose pdf_source marker points at another user's upload."""
+    if upload_handler is None:
+        return
+    from src.pdf_form_doc import find_source_upload_id
+
+    upload_id = find_source_upload_id(content or "")
+    if not upload_id:
+        return
+    auth_manager = getattr(getattr(request.app, "state", None), "auth_manager", None)
+    if not _resolve_user_upload_path(upload_handler, upload_id, user, auth_manager):
+        raise HTTPException(
+            400,
+            "Document PDF marker references an upload you do not own",
+        )
+
+
+def _derive_title(content: str) -> str:
+    """Derive a title from document content."""
+    import re
+    if not isinstance(content, str):
+        return "Untitled"
+    text = content.strip()
+    if not text:
+        return "Untitled"
+
+    # Markdown header
+    md = re.match(r'^#{1,3}\s+(.+)', text, re.MULTILINE)
+    if md:
+        title = md.group(1).strip()
+        if len(title) > 50:
+            title = title[:48] + "…"
+        return title
+
+    # HTML heading
+    html = re.search(r'<h[1-3][^>]*>([^<]+)</h[1-3]>', text, re.IGNORECASE)
+    if html:
+        title = html.group(1).strip()
+        if len(title) > 50:
+            title = title[:48] + "…"
+        return title
+
+    # First non-empty line (if short enough)
+    for line in text.split('\n'):
+        line = line.strip()
+        if line and 2 <= len(line) <= 60:
+            title = re.sub(r'[:#*`]+$', '', line).strip()
+            if title and len(title) > 50:
+                title = title[:48] + "…"
+            return title or "Untitled"
+
+    return "Untitled"

routes/document_helpers.py

@@ -1,243 +1,14 @@
-"""document_helpers.py — Pydantic models, doc serializers, owner gating, file-locator helpers shared with document_routes.py."""
+"""Backward-compat shim — canonical location is routes/document/document_helpers.py.
 
-"""Document routes — CRUD for living documents with version history."""
+This module is replaced in ``sys.modules`` by the canonical module object so
+that ``import routes.document_helpers``, ``from routes.document_helpers import
+X``, and the ``sys.modules.pop("routes.document_helpers")`` + re-import
+pattern used by test_security_regressions.py all operate on the *same* object.
+Keeps existing import paths working after slice 2m (#4082/#4071).
+"""
 
-import logging
-import os
-import re
-from typing import Any, Dict, Optional
+import sys as _sys
 
-from fastapi import HTTPException, Request
-from pydantic import BaseModel
+from routes.document import document_helpers as _canonical  # noqa: F401
 
-from core.database import Document, DocumentVersion
-from core.database import Session as DbSession
-from src.auth_helpers import _auth_disabled
-from src.upload_handler import UploadHandler
-
-logger = logging.getLogger(__name__)
-
-
-# ---- Request schemas ----
-
-class DocumentCreate(BaseModel):
-    session_id: Optional[str] = None
-    title: str = "Untitled"
-    language: Optional[str] = None
-    content: str = ""
-
-class DocumentUpdate(BaseModel):
-    content: str
-    summary: Optional[str] = None
-    force_version: bool = False
-
-class DocumentPatch(BaseModel):
-    title: Optional[str] = None
-    language: Optional[str] = None
-    session_id: Optional[str] = None  # link/unlink document to a session
-
-
-# ---- Helpers ----
-
-def _doc_to_dict(doc: Document) -> Dict[str, Any]:
-    return {
-        "id": doc.id,
-        "session_id": doc.session_id,
-        "title": doc.title,
-        "language": doc.language,
-        "current_content": doc.current_content,
-        "version_count": doc.version_count,
-        "is_active": doc.is_active,
-        "archived": bool(getattr(doc, "archived", False)),
-        "created_at": (doc.created_at.isoformat() + "Z") if doc.created_at else None,
-        "updated_at": (doc.updated_at.isoformat() + "Z") if doc.updated_at else None,
-        # Source-email provenance (set when doc was created from an email
-        # attachment) — drives the "Send signed reply" menu item.
-        "source_email_uid":        getattr(doc, "source_email_uid", None),
-        "source_email_folder":     getattr(doc, "source_email_folder", None),
-        "source_email_account_id": getattr(doc, "source_email_account_id", None),
-        "source_email_message_id": getattr(doc, "source_email_message_id", None),
-    }
-
-def _version_to_dict(v: DocumentVersion) -> Dict[str, Any]:
-    return {
-        "id": v.id,
-        "document_id": v.document_id,
-        "version_number": v.version_number,
-        "content": v.content,
-        "summary": v.summary,
-        "source": v.source,
-        "created_at": v.created_at.isoformat() if v.created_at else None,
-    }
-
-
-def _verify_doc_owner(db, doc: Document, user: str):
-    """Verify `user` owns this document. Raise 404 if not.
-
-    Documents now carry their own `owner` column, so a doc whose session
-    was deleted (session_id → NULL) can still prove ownership and stay
-    openable / cloneable. We trust that column first and only fall back to
-    the session join for any not-yet-backfilled legacy row.
-    """
-    if user is None:
-        if _auth_disabled():
-            return  # Single-user / no-auth mode: allow access
-        raise HTTPException(403, "Authentication required")
-    if doc.owner is not None:
-        if doc.owner != user:
-            raise HTTPException(404, "Document not found")
-        return
-    # Legacy fallback: derive ownership from the linked session.
-    if not doc.session_id:
-        raise HTTPException(404, "Document not found")
-    session = db.query(DbSession).filter(DbSession.id == doc.session_id).first()
-    if not session or session.owner != user:
-        raise HTTPException(404, "Document not found")
-
-
-def _owner_session_filter(q, user):
-    """Restrict a documents query to those owned by `user`.
-
-    Documents now carry their own `owner` column (backfilled at boot from
-    the linked session, or assigned to the admin user for legacy/orphaned
-    docs). We filter on that directly rather than on a session join, so a
-    document whose session was deleted (session_id → NULL) still shows up
-    for its owner instead of silently vanishing from the Library + search.
-
-    The owner backfill runs in init_db before the app serves requests, so
-    by the time this filter is live there are no NULL-owner rows to leak;
-    we therefore match the owner strictly for authenticated callers."""
-    if not user:
-        if user == "" or _auth_disabled():
-            return q
-        return q.filter(False)
-    return q.filter(Document.owner == user)
-
-
-
-def _slug(name: str) -> str:
-    """Filesystem-friendly version of a document title.
-
-    Whitespace becomes underscores; other unsafe punctuation is dropped.
-    Preserves letters, digits, dot, hyphen, underscore. Idempotent.
-    """
-    import re as _re
-    s = (name or "").strip()
-    # Drop the trailing extension if the title happens to include one
-    s = _re.sub(r'\.pdf$', '', s, flags=_re.IGNORECASE)
-    s = _re.sub(r'\s+', '_', s)
-    s = _re.sub(r'[^A-Za-z0-9._-]', '', s)
-    s = _re.sub(r'_+', '_', s).strip('_')
-    return s or "form"
-
-
-# DPI scale for the interactive PDF view. ~150 DPI (2x of 72 PDF user-units).
-_PDF_RENDER_SCALE = 2.0
-
-
-def _upload_path_inside(upload_dir: str, path: str) -> bool:
-    base = os.path.realpath(upload_dir)
-    p = os.path.realpath(path)
-    try:
-        return os.path.commonpath([base, p]) == base
-    except Exception:
-        return False
-
-
-def _resolve_user_upload_path(
-    upload_handler: Any,
-    upload_id: str,
-    owner: Optional[str],
-    auth_manager=None,
-) -> Optional[str]:
-    """Resolve an upload id to a filesystem path the caller may read."""
-    if upload_handler is None:
-        return None
-    resolved = upload_handler.resolve_upload(
-        upload_id,
-        owner=owner,
-        auth_manager=auth_manager,
-    )
-    if not isinstance(resolved, dict) or not resolved:
-        return None
-    path = resolved.get("path")
-    upload_dir = getattr(upload_handler, "upload_dir", None)
-    if path and upload_dir and not _upload_path_inside(upload_dir, path):
-        logger.warning("Upload path outside upload directory: %s", path)
-        return None
-    return path
-
-
-def _locate_upload(
-    upload_dir: str,
-    file_id: str,
-    owner: Optional[str] = None,
-    auth_manager=None,
-    upload_handler: Any = None,
-):
-    """Find an upload by its filename ID via UploadHandler.resolve_upload."""
-    if upload_handler is None:
-        from src.upload_handler import UploadHandler
-
-        base_dir = os.path.dirname(os.path.abspath(upload_dir))
-        upload_handler = UploadHandler(base_dir, upload_dir)
-    return _resolve_user_upload_path(upload_handler, file_id, owner, auth_manager)
-
-
-def _assert_pdf_marker_upload_owned(
-    request: Request,
-    content: str,
-    user: Optional[str],
-    upload_handler: Any,
-) -> None:
-    """Reject document content whose pdf_source marker points at another user's upload."""
-    if upload_handler is None:
-        return
-    from src.pdf_form_doc import find_source_upload_id
-
-    upload_id = find_source_upload_id(content or "")
-    if not upload_id:
-        return
-    auth_manager = getattr(getattr(request.app, "state", None), "auth_manager", None)
-    if not _resolve_user_upload_path(upload_handler, upload_id, user, auth_manager):
-        raise HTTPException(
-            400,
-            "Document PDF marker references an upload you do not own",
-        )
-
-
-def _derive_title(content: str) -> str:
-    """Derive a title from document content."""
-    import re
-    if not isinstance(content, str):
-        return "Untitled"
-    text = content.strip()
-    if not text:
-        return "Untitled"
-
-    # Markdown header
-    md = re.match(r'^#{1,3}\s+(.+)', text, re.MULTILINE)
-    if md:
-        title = md.group(1).strip()
-        if len(title) > 50:
-            title = title[:48] + "…"
-        return title
-
-    # HTML heading
-    html = re.search(r'<h[1-3][^>]*>([^<]+)</h[1-3]>', text, re.IGNORECASE)
-    if html:
-        title = html.group(1).strip()
-        if len(title) > 50:
-            title = title[:48] + "…"
-        return title
-
-    # First non-empty line (if short enough)
-    for line in text.split('\n'):
-        line = line.strip()
-        if line and 2 <= len(line) <= 60:
-            title = re.sub(r'[:#*`]+$', '', line).strip()
-            if title and len(title) > 50:
-                title = title[:48] + "…"
-            return title or "Untitled"
-
-    return "Untitled"
+_sys.modules[__name__] = _canonical

routes/email_helpers.py

@@ -247,6 +247,7 @@ import re as _re_reply
 _REPLY_OPEN_RE = _re_reply.compile(r"<<<\s*(?:REPLY|SUMMARY|OUTPUT)\s*>>+", _re_reply.I)
 _REPLY_CLOSE_RE = _re_reply.compile(r"<<<\s*END\s*>>+", _re_reply.I)
 _REPLY_ROLE_MARKER_RE = _re_reply.compile(r"</?\|(?:assistant|assistan|user|system|tool)\|>?|</\|end\|>?", _re_reply.I)
+_SUMMARY_BULLET_RE = _re_reply.compile(r"^(?:[-*\u2022]\s+|\d+[.)]\s+)")
 
 
 def _extract_reply(text: str) -> str:
@@ -277,6 +278,125 @@ def _extract_reply(text: str) -> str:
     return _strip_think(t).strip()
 
 
+def _build_email_summary_messages(sender: str, subject: str, body_for_llm: str) -> list[dict[str, str]]:
+    return [
+        {
+            "role": "system",
+            "content": (
+                "You are an email summarizer. Format: 1-3 short bullet points "
+                "(use '- '). Cover: main point, action items, deadlines. If the "
+                "email has attachments (marked '--- ATTACHMENTS ---'), USE THEIR "
+                "CONTENTS - pull invoice totals, deadlines, key clauses, concrete "
+                "numbers/dates from PDFs/docs into the bullets. Be terse.\n\n"
+                "OUTPUT FORMAT: Put ONLY the bullet points between these exact "
+                "markers, each on its own line:\n"
+                "<<<SUMMARY>>>\n"
+                "- ...\n"
+                "<<<END>>>\n"
+                "Any reasoning must come BEFORE <<<SUMMARY>>> (ideally inside "
+                "<think>...</think>). Only the text between the markers is kept."
+            ),
+        },
+        {
+            "role": "user",
+            "content": (
+                f"From: {sender}\nSubject: {subject}\n\n{body_for_llm[:12000]}"
+                "\n\n---\n\nSummarize the email. Output the bullets between "
+                "<<<SUMMARY>>> and <<<END>>>."
+            ),
+        },
+    ]
+
+
+async def _generate_email_summary(
+    url: str,
+    model: str,
+    sender: str,
+    subject: str,
+    body_for_llm: str,
+    *,
+    headers: dict | None = None,
+    max_tokens: int = 8192,
+    timeout: int = 180,
+) -> str:
+    """Generate an interactive email summary through the shared LLM adapter."""
+    from src.llm_core import llm_call_async
+
+    raw = await llm_call_async(
+        url=url,
+        model=model,
+        messages=_build_email_summary_messages(sender, subject, body_for_llm),
+        temperature=0.3,
+        max_tokens=max_tokens,
+        headers=headers,
+        timeout=timeout,
+        workload="foreground",
+    )
+    return _normalize_email_summary(raw)
+
+
+async def _generate_scheduled_email_summary(
+    url: str,
+    model: str,
+    sender: str,
+    subject: str,
+    body_for_llm: str,
+    *,
+    headers: dict | None = None,
+    owner: str | None = None,
+    max_tokens: int = 8192,
+    timeout: int = 180,
+) -> str:
+    """Generate a scheduled summary through the background task candidate chain."""
+    from src.task_endpoint import task_llm_call_async
+
+    raw = await task_llm_call_async(
+        messages=_build_email_summary_messages(sender, subject, body_for_llm),
+        fallback_url=url,
+        fallback_model=model,
+        fallback_headers=headers,
+        owner=owner,
+        temperature=0.3,
+        max_tokens=max_tokens,
+        timeout=timeout,
+    )
+    return _normalize_email_summary(raw)
+
+
+def _normalize_email_summary(raw) -> str:
+    """Extract a stable cache/UI summary from provider output."""
+    raw_text = raw or ""
+    if _REPLY_OPEN_RE.search(raw_text):
+        summary = _extract_reply(raw_text)
+        if summary:
+            return summary
+
+    cleaned = _strip_think(raw_text).strip()
+    bullets = [
+        line.strip()
+        for line in cleaned.splitlines()
+        if _SUMMARY_BULLET_RE.match(line.strip())
+    ]
+    if bullets:
+        return "\n".join(bullets)
+    return cleaned.strip()
+
+
+EMAIL_SUMMARY_ERROR_CODE = "email_summary_unavailable"
+EMAIL_SUMMARY_ERROR_MESSAGE = "Failed to summarize"
+
+
+def _email_summary_failure_log_detail(exc: BaseException) -> str:
+    """Return useful provider-failure metadata without echoing exception text."""
+    detail = f"type={type(exc).__name__}"
+    status = getattr(exc, "status_code", None)
+    if status is None:
+        status = getattr(getattr(exc, "response", None), "status_code", None)
+    if isinstance(status, int):
+        detail += f" status={status}"
+    return detail
+
+
 def _apply_email_style_mechanics(text: str) -> str:
     """Enforce deterministic writing-style mechanics that models often miss."""
     if not text:

routes/email_pollers.py

@@ -40,6 +40,7 @@ from routes.email_helpers import (
     _pre_retrieve_context,
     _attach_compose_uploads, _cleanup_compose_uploads, _q,
     SCHEDULED_DB, _EMAIL_REPLY_SYS_PROMPT_BASE, _email_cache_owner_clause,
+    _generate_scheduled_email_summary, _email_summary_failure_log_detail,
 )
 
 logger = logging.getLogger(__name__)
@@ -653,6 +654,7 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
         no_msgid = 0
         examined = 0
         _summaries_created = 0
+        _summary_failed = 0
         _events_created = 0
         _replies_drafted = 0
         _reply_failed = 0
@@ -785,16 +787,17 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
 
                 if need_sum:
                     try:
-                        summary = await task_llm_call_async(
-                            messages=[
-                                {"role": "system", "content": "You are an email summarizer. Format: 1-3 short bullet points (use '- '). Cover: main point, action items, deadlines. If the email has attachments (marked '--- ATTACHMENTS ---'), USE THEIR CONTENTS — pull out invoice totals, deadlines, key clauses, any concrete numbers/dates in PDFs/docs, and reflect them in the bullets. Be terse.\n\nOUTPUT FORMAT: Put ONLY the bullet points between these exact markers, each on its own line:\n<<<SUMMARY>>>\n- ...\n<<<END>>>\nAny reasoning or planning must come BEFORE <<<SUMMARY>>> (ideally inside <think>...</think>). Only the text between the markers is kept."},
-                                {"role": "user", "content": f"From: {sender}\nSubject: {subject}\n\n{body_for_llm[:12000]}\n\n---\n\nSummarize the email. Output the bullets between <<<SUMMARY>>> and <<<END>>>."},
-                            ],
-                            fallback_url=url, fallback_model=model, fallback_headers=headers,
+                        summary = await _generate_scheduled_email_summary(
+                            url=url,
+                            model=model,
+                            sender=sender,
+                            subject=subject,
+                            body_for_llm=body_for_llm,
+                            headers=req_headers,
                             owner=account_owner or None,
-                            temperature=0.3, max_tokens=16384, timeout=240,
+                            max_tokens=16384,
+                            timeout=240,
                         )
-                        summary = _extract_reply((summary or "").strip())
                         if summary:
                             _c = _sql3.connect(SCHEDULED_DB)
                             _c.execute("""
@@ -808,10 +811,19 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
                             _summaries_created += 1
                             _uid_text = uid.decode() if isinstance(uid, bytes) else str(uid)
                             _detail_lines.append(f"summary · {_folder}#{_uid_text} · {subject or '(no subject)'} — {sender or '(unknown sender)'}")
+                        else:
+                            _summary_failed += 1
+                            _uid_text = uid.decode() if isinstance(uid, bytes) else str(uid)
+                            _detail_lines.append(f"summary empty · {_folder}#{_uid_text} · {subject or '(no subject)'} — {sender or '(unknown sender)'}")
                     except Exception as e:
+                        _summary_failed += 1
                         _uid_text = uid.decode() if isinstance(uid, bytes) else str(uid)
                         _detail_lines.append(f"summary failed · {_folder}#{_uid_text} · {subject or '(no subject)'} — {sender or '(unknown sender)'}")
-                        logger.warning(f"Auto-summary {uid} failed: {e}")
+                        logger.warning(
+                            "Auto-summary uid=%s failed %s",
+                            _uid_text,
+                            _email_summary_failure_log_detail(e),
+                        )
 
                 if need_reply:
                     await _emit_progress(progress_cb, f"Drafting reply {processed + 1}/{_max_process} · checked {examined}/{len(uid_list)}")
@@ -1320,6 +1332,8 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
             parts.append(f"processed {processed} new")
         if auto_sum:
             parts.append(f"summarized {_summaries_created}")
+            if _summary_failed:
+                parts.append(f"{_summary_failed} summary failed")
         if auto_reply_draft:
             parts.append(f"drafted {_replies_drafted} repl" + ("y" if _replies_drafted == 1 else "ies"))
             if _reply_failed:

routes/email_routes.py

@@ -57,7 +57,8 @@ from routes.email_helpers import (
     _extract_attachment_to_disk, _extract_html, _extract_text,
     _fetch_sender_thread_context, _pre_retrieve_context,
     _EMAIL_REPLY_SYS_PROMPT_BASE, _POOL_HOOKS,
-    _friendly_email_auth_error,
+    _friendly_email_auth_error, _email_summary_failure_log_detail,
+    _generate_email_summary, EMAIL_SUMMARY_ERROR_CODE, EMAIL_SUMMARY_ERROR_MESSAGE,
     SendEmailRequest, ExtractStyleRequest,
     ATTACHMENTS_DIR, COMPOSE_UPLOADS_DIR, SCHEDULED_DB,
     attachment_extract_dir, _email_cache_owner_clause, email_translation_body_hash,
@@ -2860,13 +2861,22 @@ def setup_email_routes():
                 return indexed_response
             return {"emails": [], "total": 0, "error": "Mail operation failed"}
 
-    def _read_email_sync(uid, folder, account_id, owner, mark_seen=True, full=False):
+    def _read_email_sync(uid, folder, account_id, owner, mark_seen=False, full=False):
         """Sync IMAP read — wrapped in to_thread by the async handler.
 
         The normal reader path fetches the headers plus a bounded body prefix.
         That avoids downloading multi-megabyte attachments just to open a
         message. Full-message fetch remains available for flows that need
         attachment metadata immediately, such as forwarding.
+
+        `mark_seen` defaults to False because it mutates provider state: it
+        selects the mailbox read-write and issues a STORE. Only a foreground
+        open should ask for it, and it has to ask explicitly.
+
+        A failed \\Seen transition is reported as `mark_seen_failed` on an
+        otherwise normal response, never as an error. The body has already been
+        fetched at that point, so refusing to return it would turn a cosmetic
+        flag failure into an unreadable message.
         """
         import time as _t
         _t0 = _t.monotonic()
@@ -2874,9 +2884,28 @@ def setup_email_routes():
         preview_bytes = 384 * 1024
         _t_select = 0.0
         _t_fetch = 0.0
+        mark_seen_failed = False
         try:
             with _imap(account_id, owner=owner) as conn:
-                conn.select(_q(folder), readonly=True)
+                # A foreground open owns both the body fetch and the \Seen
+                # transition. Keep them on one read-write IMAP selection so the
+                # route never schedules a second connection that can race the
+                # response. Prefetch/read-only callers retain BODY.PEEK and a
+                # read-only mailbox selection.
+                try:
+                    conn.select(_q(folder), readonly=not mark_seen)
+                except Exception as select_exc:
+                    if not mark_seen:
+                        raise
+                    # Read-only mailboxes (shared archives, some provider
+                    # folders) reject a read-write SELECT. Serve the message
+                    # read-only and report the flag failure.
+                    logger.warning(
+                        f"read-write SELECT rejected for {folder!r}; "
+                        f"serving read-only without \\Seen: {select_exc}"
+                    )
+                    conn.select(_q(folder), readonly=True)
+                    mark_seen_failed = True
                 _t_select = _t.monotonic() - _t0
                 fetch_query = "(BODY.PEEK[])" if full else f"(BODY.PEEK[HEADER] BODY.PEEK[TEXT]<0.{preview_bytes}>)"
                 status, msg_data = _imap_uid_fetch(conn, uid, fetch_query)
@@ -2902,22 +2931,44 @@ def setup_email_routes():
                         header_part = msg_data[0][1] or b""
                     raw = header_part + b"\r\n" + text_part
 
-            msg = email_mod.message_from_bytes(raw)
+                # Parse the fetched payload before mutating provider state. If
+                # the message is malformed enough that the reader cannot build
+                # a response, the caller gets an error while the message stays
+                # unread instead of receiving a false optimistic rollback.
+                msg = email_mod.message_from_bytes(raw)
 
-            subject = _decode_header(msg.get("Subject", "(no subject)"))
-            sender = _decode_header(msg.get("From", "unknown"))
-            to = _decode_header(msg.get("To", ""))
-            cc = _decode_header(msg.get("Cc", ""))
-            date_str = msg.get("Date", "")
-            message_id = msg.get("Message-ID", "")
-            in_reply_to = msg.get("In-Reply-To", "")
-            references = msg.get("References", "")
-            body = _extract_text(msg)
-            body_html = _extract_html(msg)
+                subject = _decode_header(msg.get("Subject", "(no subject)"))
+                sender = _decode_header(msg.get("From", "unknown"))
+                to = _decode_header(msg.get("To", ""))
+                cc = _decode_header(msg.get("Cc", ""))
+                date_str = msg.get("Date", "")
+                message_id = msg.get("Message-ID", "")
+                in_reply_to = msg.get("In-Reply-To", "")
+                references = msg.get("References", "")
+                body = _extract_text(msg)
+                body_html = _extract_html(msg)
+
+                sender_name, sender_addr = email.utils.parseaddr(sender)
+                parsed_date = email.utils.parsedate_to_datetime(date_str) if date_str else None
+                attachments = _list_attachments_from_msg(msg) if full else (_email_attachment_meta_cache_get(owner, account_id, folder, uid) or [])
+
+                if mark_seen and not mark_seen_failed:
+                    seen_status, _ = conn.uid("STORE", _uid_bytes(uid), "+FLAGS", "(\\Seen)")
+                    if seen_status != "OK":
+                        # Report, don't raise. The parsed body below is still a
+                        # valid response; only the flag claim is untrue.
+                        logger.warning(
+                            f"IMAP STORE \\Seen failed for UID {uid} in {folder!r}: {seen_status}"
+                        )
+                        mark_seen_failed = True
+
+            # Only record the local flag transition when the provider actually
+            # accepted it, so the index and list cache cannot drift ahead of
+            # the mailbox.
+            if mark_seen and not mark_seen_failed:
+                _email_index_update_flags(owner, account_id, folder, uid, "\\Seen", True)
+                _update_list_cache_seen(account_id, folder, uid, True)
 
-            sender_name, sender_addr = email.utils.parseaddr(sender)
-            parsed_date = email.utils.parsedate_to_datetime(date_str) if date_str else None
-            attachments = _list_attachments_from_msg(msg) if full else (_email_attachment_meta_cache_get(owner, account_id, folder, uid) or [])
             related_attachments = []
             if full and not _has_visible_attachments(msg):
                 related_attachments = _related_thread_attachments_sync(
@@ -3038,20 +3089,29 @@ def setup_email_routes():
                 "boundaries": cached_boundaries,
                 "thread_turns": cached_turns,
                 "sender_signature": cached_sender_sig,
+                # Per-request, not part of the message: the route strips this
+                # before caching so a one-off flag failure is never replayed to
+                # later readers.
+                "mark_seen_failed": mark_seen_failed,
             }
         except Exception as e:
             logger.error(f"Failed to read email {uid}: {e}")
             return {"error": "Mail operation failed"}
 
     def _mark_email_seen_sync(uid, folder, account_id, owner):
+        """Synchronously mark a cached email seen and report success."""
         try:
             with _imap(account_id, owner=owner) as conn:
-                conn.select(_q(folder))
-                conn.uid("STORE", _uid_bytes(uid), "+FLAGS", "\\Seen")
+                conn.select(_q(folder), readonly=False)
+                status, _ = conn.uid("STORE", _uid_bytes(uid), "+FLAGS", "(\\Seen)")
+                if status != "OK":
+                    return False
             _email_index_update_flags(owner, account_id, folder, uid, "\\Seen", True)
             _update_list_cache_seen(account_id, folder, uid, True)
+            return True
         except Exception as e:
-            logger.debug(f"mark-seen after cached read failed uid={uid}: {e}")
+            logger.warning(f"mark-seen after cached read failed uid={uid}: {e}")
+            return False
 
     @router.get("/read/{uid}")
     async def read_email_by_uid(
@@ -3077,32 +3137,32 @@ def setup_email_routes():
             if cached.get("attachment_version") != EMAIL_READ_ATTACHMENT_VERSION:
                 cached = None
         if cached is not None:
-            if mark_seen:
-                try:
-                    _asyncio.create_task(_asyncio.to_thread(_mark_email_seen_sync, uid, folder, account_id, owner))
-                except RuntimeError:
-                    pass
+            # A cache hit already holds a complete, valid message. Await the
+            # STORE so the response reports the real flag state, but never let
+            # a failed STORE withhold a body we are holding in memory.
+            if mark_seen and not await _asyncio.to_thread(
+                _mark_email_seen_sync, uid, folder, account_id, owner
+            ):
+                return {**cached, "mark_seen_failed": True}
             return cached
         if not full:
             persisted = _email_preview_cache_get(owner, account_id, folder, uid)
             if persisted and persisted.get("attachment_version") == EMAIL_READ_ATTACHMENT_VERSION:
                 _read_cache_put(ck, persisted)
-                if mark_seen:
-                    try:
-                        _asyncio.create_task(_asyncio.to_thread(_mark_email_seen_sync, uid, folder, account_id, owner))
-                    except RuntimeError:
-                        pass
+                if mark_seen and not await _asyncio.to_thread(
+                    _mark_email_seen_sync, uid, folder, account_id, owner
+                ):
+                    return {**persisted, "mark_seen_failed": True}
                 return persisted
         result = await _asyncio.to_thread(_read_email_sync, uid, folder, account_id, owner, mark_seen, full)
         if result and not result.get("error"):
-            _read_cache_put(ck, result)
+            # `mark_seen_failed` describes this request, not the message, so it
+            # must not enter either cache — a later reader would otherwise be
+            # told a STORE failed that it never issued.
+            cacheable = {k: v for k, v in result.items() if k != "mark_seen_failed"}
+            _read_cache_put(ck, cacheable)
             if not full:
-                _email_preview_cache_put(owner, account_id, folder, uid, result)
-            if mark_seen:
-                try:
-                    _asyncio.create_task(_asyncio.to_thread(_mark_email_seen_sync, uid, folder, account_id, owner))
-                except RuntimeError:
-                    pass
+                _email_preview_cache_put(owner, account_id, folder, uid, cacheable)
         return result
 
     def _schedule_recent_email_warm(emails: list, folder: str, account_id: str | None, owner: str):
@@ -4766,8 +4826,6 @@ def setup_email_routes():
         """Generate a quick AI summary of an email body."""
         try:
             from src.endpoint_resolver import resolve_endpoint
-            from src.llm_core import _uses_max_completion_tokens, _restricts_temperature
-            import requests as _req
 
             body = data.get("body", "")
             subject = data.get("subject", "")
@@ -4778,7 +4836,11 @@ def setup_email_routes():
             if account_id:
                 _assert_owns_account(account_id, owner)
             if not body:
-                return {"success": False, "error": "No body provided"}
+                return {
+                    "success": False,
+                    "error": "No body provided",
+                    "error_code": "email_summary_missing_body",
+                }
 
             # If we know which UID this is, fetch the raw message and pull
             # attachment text so the summary can reference invoice totals,
@@ -4807,53 +4869,43 @@ def setup_email_routes():
             if not url:
                 url, model, headers = resolve_endpoint("default", owner=owner)
             if not url or not model:
-                return {"success": False, "error": "No LLM endpoint configured"}
+                return {
+                    "success": False,
+                    "error": "No model configured for email summaries",
+                    "error_code": "email_summary_not_configured",
+                }
 
             req_headers = {"Content-Type": "application/json"}
             if headers:
                 req_headers.update(headers)
-            tok_key = "max_completion_tokens" if _uses_max_completion_tokens(model) else "max_tokens"
-            payload = {
-                "model": model,
-                "messages": [
-                    {"role": "system", "content": "You are an email summarizer. Format: 1-3 short bullet points (use '- '). Cover: main point, action items, deadlines. If the email has attachments (marked '--- ATTACHMENTS ---'), USE THEIR CONTENTS — pull invoice totals, deadlines, key clauses, concrete numbers/dates from PDFs/docs into the bullets. Be terse.\n\nOUTPUT FORMAT: Put ONLY the bullet points between these exact markers, each on its own line:\n<<<SUMMARY>>>\n- ...\n<<<END>>>\nAny reasoning must come BEFORE <<<SUMMARY>>> (ideally inside <think>...</think>). Only the text between the markers is kept."},
-                    {"role": "user", "content": f"From: {sender}\nSubject: {subject}\n\n{body_for_llm[:12000]}\n\n---\n\nSummarize the email. Output the bullets between <<<SUMMARY>>> and <<<END>>>."},
-                ],
-                tok_key: 8192,
-                "temperature": 0.3,
-                "stream": False,
-            }
-            # Reasoning models (o1/o3/o4/gpt-5) reject an explicit temperature.
-            if _restricts_temperature(model):
-                payload.pop("temperature", None)
-            resp = await asyncio.to_thread(
-                _req.post, url, json=payload, headers=req_headers, timeout=180
-            )
-            if not resp.ok:
-                return {"success": False, "error": f"LLM HTTP {resp.status_code}"}
-            rdata = resp.json()
-            msg = (rdata.get("choices") or [{}])[0].get("message", {})
-            content = (msg.get("content") or "").strip()
-            content = _extract_reply(content)
+            try:
+                content = await _generate_email_summary(
+                    url=url,
+                    model=model,
+                    sender=sender,
+                    subject=subject,
+                    body_for_llm=body_for_llm,
+                    headers=req_headers,
+                    max_tokens=8192,
+                    timeout=180,
+                )
+            except Exception as e:
+                logger.warning(
+                    "Email summary LLM call failed %s",
+                    _email_summary_failure_log_detail(e),
+                )
+                return {
+                    "success": False,
+                    "error": EMAIL_SUMMARY_ERROR_MESSAGE,
+                    "error_code": EMAIL_SUMMARY_ERROR_CODE,
+                }
 
             if not content:
-                # Model put everything in reasoning_content — extract bullet points
-                rc = (msg.get("reasoning_content") or "").strip()
-                # Find bullet-point style output (lines starting with -, •, *, or numbered)
-                bullet_lines = []
-                for line in rc.split("\n"):
-                    stripped = line.strip()
-                    if re.match(r"^[-•*]\s+|^\d+[.)]\s+", stripped):
-                        bullet_lines.append(stripped)
-                if bullet_lines:
-                    content = "\n".join(bullet_lines)
-                else:
-                    # Last resort: take the last paragraph
-                    paragraphs = [p.strip() for p in rc.split("\n\n") if p.strip()]
-                    content = paragraphs[-1] if paragraphs else rc[:500]
-
-            if not content:
-                return {"success": False, "error": "Empty response from model"}
+                return {
+                    "success": False,
+                    "error": "The model returned an empty summary",
+                    "error_code": "email_summary_empty",
+                }
 
             # Cache the summary if we have a message_id
             mid = data.get("message_id", "")
@@ -4876,8 +4928,15 @@ def setup_email_routes():
 
             return {"success": True, "summary": content, "model_used": model}
         except Exception as e:
-            logger.error(f"Failed to summarize: {e}")
-            return {"success": False, "error": "Mail operation failed"}
+            logger.error(
+                "Email summary route failed %s",
+                _email_summary_failure_log_detail(e),
+            )
+            return {
+                "success": False,
+                "error": EMAIL_SUMMARY_ERROR_MESSAGE,
+                "error_code": EMAIL_SUMMARY_ERROR_CODE,
+            }
 
     @router.post("/translate")
     async def translate_email(data: dict, owner: str = Depends(require_owner)):

routes/history/history_routes.py

@@ -137,44 +137,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
             entry["metadata"] = meta
         return entry
 
-    def _db_message_metadata(m: DbChatMessage) -> Dict[str, Any]:
-        meta = {}
-        if m.meta_data:
-            try:
-                meta = json.loads(m.meta_data) or {}
-            except (json.JSONDecodeError, ValueError):
-                meta = {}
-        if m.timestamp and "timestamp" not in meta:
-            meta["timestamp"] = m.timestamp.isoformat() + "Z"
-        return meta
-
-    def _hydrate_session_history_from_db(session_id: str, rows: list[DbChatMessage]) -> None:
-        """Rebuild in-memory context from raw DB rows after a history load.
-
-        The browser history endpoint can return paged/display-trimmed messages,
-        but the next model call reads ``session.history``. After a restart or a
-        stale in-memory session, selecting an old chat through the paged endpoint
-        used to show the transcript while the model only saw fresh context.
-        """
-        if not rows:
-            return
-        try:
-            session = session_manager.get_session(session_id)
-        except KeyError:
-            return
-        session.history = [
-            ChatMessage(role=m.role, content=m.content, metadata=_db_message_metadata(m) or None)
-            for m in rows
-        ]
-        session.message_count = len(session.history)
-
-    def _session_needs_db_history_hydration(session_id: str, total: int) -> bool:
-        try:
-            session = session_manager.get_session(session_id)
-        except KeyError:
-            return False
-        return len(session.history or []) < int(total or 0)
-
     @router.get("/api/history/{session_id}")
     async def get_session_history(
         request: Request,
@@ -198,6 +160,8 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
                 )
                 page_offset = int(offset) if offset is not None else max(total - page_limit, 0)
                 page_offset = max(0, min(page_offset, total))
+                # Keep display pagination page-scoped. ``get_session`` is the
+                # full model-context hydration seam and must not be entered here.
                 rows = (
                     db.query(DbChatMessage)
                     .filter(DbChatMessage.session_id == session_id)
@@ -206,14 +170,6 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
                     .limit(page_limit)
                     .all()
                 )
-                if _session_needs_db_history_hydration(session_id, total):
-                    full_rows = (
-                        db.query(DbChatMessage)
-                        .filter(DbChatMessage.session_id == session_id)
-                        .order_by(DbChatMessage.timestamp)
-                        .all()
-                    )
-                    _hydrate_session_history_from_db(session_id, full_rows)
                 history_dict = [
                     entry for entry in (_db_history_entry(m) for m in rows)
                     if not (entry.get("metadata") or {}).get("hidden")
@@ -258,7 +214,10 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
                     entry["metadata"] = msg["metadata"]
                 history_dict.append(entry)
 
-        # Fallback: load from DB if in-memory is empty
+        # Fallback: load from DB if in-memory renders empty. Display only —
+        # get_session above is the hydration seam, so nothing here writes back
+        # into session.history — rebuilding it from raw rows would overwrite
+        # parsed multimodal content and the _db_id edit/delete keys it just set.
         if not history_dict:
             db = SessionLocal()
             try:
@@ -268,17 +227,10 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
                     .order_by(DbChatMessage.timestamp)
                     .all()
                 )
-                db_history = []
-                for m in db_messages:
-                    db_history.append(_db_history_entry(m))
-                if db_history:
-                    # Rebuild in-memory history from the full set so hidden
-                    # messages (e.g. compaction summaries) are kept for AI context.
-                    _hydrate_session_history_from_db(session_id, db_messages)
                 # Response excludes hidden messages, matching the in-memory path.
                 history_dict = [
-                    m for m in db_history
-                    if not (m.get("metadata") or {}).get("hidden")
+                    entry for entry in (_db_history_entry(m) for m in db_messages)
+                    if not (entry.get("metadata") or {}).get("hidden")
                 ]
             except Exception as e:
                 logger.error(f"DB fallback failed for {session_id}: {e}")
@@ -645,8 +597,14 @@ def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
             body = await request.json()
             keep_count = body.get("keep_count", 0)
 
-            # Get the source session
-            source = session_manager.sessions.get(session_id)
+            # Get the source session. keep_count indexes into source.history,
+            # so this must go through get_session — reading the cache directly
+            # forks an empty transcript out of a metadata-only session after a
+            # restart (display pagination no longer hydrates it).
+            try:
+                source = session_manager.get_session(session_id)
+            except KeyError:
+                raise HTTPException(404, "Session not found")
             if not source:
                 raise HTTPException(404, "Session not found")
 

routes/memory/memory_routes.py

@@ -21,7 +21,7 @@ def _strip_list_prefix(text: str) -> str:
         return text
     return _LIST_PREFIX_RE.sub("", text, count=1).strip()
 
-from services.memory import MemoryManager
+from services.memory import MemoryManager, MemoryStoreUnreadable
 from core.session_manager import SessionManager
 from src.request_models import MemoryAddRequest
 from core.database import SessionLocal
@@ -35,6 +35,22 @@ from src.upload_limits import read_upload_limited, MEMORY_IMPORT_MAX_BYTES
 logger = logging.getLogger(__name__)
 
 
+def _load_for_update(memory_manager) -> List[Dict[str, Any]]:
+    """Load the whole store for a read-modify-write cycle.
+
+    A transient read failure must not look like an empty store: the caller
+    would append to ``[]`` and save that back, atomically destroying every
+    existing memory (issue #5673). Surface it as a 503 and change nothing.
+    """
+    try:
+        return memory_manager.load_all_for_update()
+    except MemoryStoreUnreadable as e:
+        logger.error("Refusing to rewrite the memory store: %s", e)
+        raise HTTPException(
+            503, "Memory store is temporarily unreadable — no changes were made."
+        )
+
+
 def setup_memory_routes(memory_manager: MemoryManager, session_manager: SessionManager, memory_vector=None):
     """Set up memory-related routes."""
     router = APIRouter(prefix="/api/memory", tags=["memory"])
@@ -116,7 +132,7 @@ def setup_memory_routes(memory_manager: MemoryManager, session_manager: SessionM
         new_entry = memory_manager.add_entry(text, memory_data.source, memory_data.category, owner=user)
         if memory_data.session_id:
             new_entry["session_id"] = memory_data.session_id
-        all_mem = memory_manager.load_all()
+        all_mem = _load_for_update(memory_manager)
         all_mem.append(new_entry)
         memory_manager.save(all_mem)
         # Sync vector index
@@ -487,7 +503,7 @@ def setup_memory_routes(memory_manager: MemoryManager, session_manager: SessionM
     def pin_memory(request: Request, memory_id: str, pinned: bool = Form(True)):
         """Pin or unpin a memory. Pinned memories are always included in context."""
         user = _owner(request)
-        all_mem = memory_manager.load_all()
+        all_mem = _load_for_update(memory_manager)
         for i, memory in enumerate(all_mem):
             if memory["id"] == memory_id:
                 _verify_memory_owner(memory, user)
@@ -512,7 +528,7 @@ def setup_memory_routes(memory_manager: MemoryManager, session_manager: SessionM
     def update_memory(request: Request, memory_id: str, text: str = Form(...), category: str = Form(None)):
         """Update an existing memory item with new text and optional category."""
         user = _owner(request)
-        all_mem = memory_manager.load_all()
+        all_mem = _load_for_update(memory_manager)
         for i, memory in enumerate(all_mem):
             if memory["id"] == memory_id:
                 _verify_memory_owner(memory, user)
@@ -534,7 +550,7 @@ def setup_memory_routes(memory_manager: MemoryManager, session_manager: SessionM
     def delete_memory(request: Request, memory_id: str):
         """Delete a memory item by its ID."""
         user = _owner(request)
-        all_mem = memory_manager.load_all()
+        all_mem = _load_for_update(memory_manager)
 
         # Find and verify ownership before deleting
         target = next((m for m in all_mem if m["id"] == memory_id), None)

routes/search/__init__.py

@@ -0,0 +1,5 @@
+"""Search route domain package (slice 2j, #4082/#4071).
+
+Contains search_routes.py, migrated from the flat routes/ directory.
+Backward-compat shim at routes/search_routes.py re-exports from here.
+"""

routes/search/search_routes.py

@@ -0,0 +1,111 @@
+"""Search routes — /api/search/config GET, /api/search POST."""
+
+import logging
+from typing import Dict, Any
+
+from fastapi import APIRouter, Request
+
+import time
+
+from services.search import get_search_config, comprehensive_web_search, PROVIDER_INFO
+from services.search.core import _call_provider
+from services.search.providers import _get_provider_key, _get_search_instance
+
+logger = logging.getLogger(__name__)
+
+
+async def _request_values(request: Request) -> Dict[str, Any]:
+    """Accept JSON, form data, or query params for search endpoints.
+
+    The browser UI posts FormData, while the agent's generic app_api tool
+    posts JSON. FastAPI Form(...) rejects JSON with a 422 before our handler
+    runs, which made the model think SearXNG was broken.
+    """
+    values: Dict[str, Any] = dict(request.query_params)
+    content_type = (request.headers.get("content-type") or "").lower()
+    try:
+        if "application/json" in content_type:
+            body = await request.json()
+            if isinstance(body, dict):
+                values.update(body)
+        else:
+            form = await request.form()
+            values.update(dict(form))
+    except Exception:
+        pass
+    return values
+
+
+def setup_search_routes(config) -> APIRouter:
+    router = APIRouter(tags=["search"])
+
+    @router.get("/api/search/config")
+    async def get_search_settings() -> Dict[str, Any]:
+        return get_search_config()
+
+    @router.post("/api/search")
+    async def do_web_search(request: Request) -> Dict[str, Any]:
+        """Standalone web search — returns context string + source list.
+
+        Used by Compare mode to pre-search once and share results across panes.
+        """
+        values = await _request_values(request)
+        query = str(values.get("query") or values.get("q") or "").strip()
+        if not query:
+            return {"context": "", "sources": [], "error": "query is required"}
+        time_filter = values.get("time_filter") or values.get("freshness")
+        if time_filter is not None:
+            time_filter = str(time_filter).strip() or None
+        try:
+            context, sources = comprehensive_web_search(
+                query, return_sources=True, time_filter=time_filter,
+            )
+            return {"context": context, "sources": sources}
+        except Exception as e:
+            logger.error(f"Standalone web search failed: {e}")
+            return {"context": "", "sources": [], "error": str(e)}
+
+    @router.get("/api/search/providers")
+    async def list_search_providers():
+        """Return available search providers with config status."""
+        providers = []
+        for pid, (label, needs_key, needs_url) in PROVIDER_INFO.items():
+            if pid == "disabled":
+                continue
+            available = True
+            if needs_key and not _get_provider_key(pid):
+                available = False
+            if needs_url and pid == "searxng" and not _get_search_instance():
+                available = False
+            providers.append({
+                "id": pid,
+                "label": label,
+                "available": available,
+            })
+        return providers
+
+    @router.post("/api/search/query")
+    async def search_with_provider(request: Request) -> Dict[str, Any]:
+        """Search using a specific provider. Used by compare search mode."""
+        values = await _request_values(request)
+        query = str(values.get("query") or values.get("q") or "").strip()
+        provider = str(values.get("provider") or "").strip()
+        try:
+            count = int(values.get("count") or values.get("limit") or 10)
+        except Exception:
+            count = 10
+        if not query:
+            return {"results": [], "provider": provider, "error": "query is required"}
+        if provider not in PROVIDER_INFO or provider == "disabled":
+            return {"results": [], "provider": provider, "error": "Unknown provider"}
+        t0 = time.time()
+        try:
+            results = _call_provider(provider, query, min(count, 20))
+            elapsed = round(time.time() - t0, 2)
+            return {"results": results, "provider": provider, "time": elapsed}
+        except Exception as e:
+            elapsed = round(time.time() - t0, 2)
+            logger.error(f"Search provider {provider} failed: {e}")
+            return {"results": [], "provider": provider, "time": elapsed, "error": str(e)}
+
+    return router

routes/search_routes.py

@@ -1,111 +1,13 @@
-"""Search routes — /api/search/config GET, /api/search POST."""
+"""Backward-compat shim — canonical location is routes/search/search_routes.py.
 
-import logging
-from typing import Dict, Any
+This module is replaced in ``sys.modules`` by the canonical module object so
+that ``import routes.search_routes`` and ``from routes.search_routes import X``
+keep resolving to the canonical module. Keeps existing import paths working
+after slice 2j (#4082/#4071).
+"""
 
-from fastapi import APIRouter, Request
+import sys as _sys
 
-import time
+from routes.search import search_routes as _canonical  # noqa: F401
 
-from services.search import get_search_config, comprehensive_web_search, PROVIDER_INFO
-from services.search.core import _call_provider
-from services.search.providers import _get_provider_key, _get_search_instance
-
-logger = logging.getLogger(__name__)
-
-
-async def _request_values(request: Request) -> Dict[str, Any]:
-    """Accept JSON, form data, or query params for search endpoints.
-
-    The browser UI posts FormData, while the agent's generic app_api tool
-    posts JSON. FastAPI Form(...) rejects JSON with a 422 before our handler
-    runs, which made the model think SearXNG was broken.
-    """
-    values: Dict[str, Any] = dict(request.query_params)
-    content_type = (request.headers.get("content-type") or "").lower()
-    try:
-        if "application/json" in content_type:
-            body = await request.json()
-            if isinstance(body, dict):
-                values.update(body)
-        else:
-            form = await request.form()
-            values.update(dict(form))
-    except Exception:
-        pass
-    return values
-
-
-def setup_search_routes(config) -> APIRouter:
-    router = APIRouter(tags=["search"])
-
-    @router.get("/api/search/config")
-    async def get_search_settings() -> Dict[str, Any]:
-        return get_search_config()
-
-    @router.post("/api/search")
-    async def do_web_search(request: Request) -> Dict[str, Any]:
-        """Standalone web search — returns context string + source list.
-
-        Used by Compare mode to pre-search once and share results across panes.
-        """
-        values = await _request_values(request)
-        query = str(values.get("query") or values.get("q") or "").strip()
-        if not query:
-            return {"context": "", "sources": [], "error": "query is required"}
-        time_filter = values.get("time_filter") or values.get("freshness")
-        if time_filter is not None:
-            time_filter = str(time_filter).strip() or None
-        try:
-            context, sources = comprehensive_web_search(
-                query, return_sources=True, time_filter=time_filter,
-            )
-            return {"context": context, "sources": sources}
-        except Exception as e:
-            logger.error(f"Standalone web search failed: {e}")
-            return {"context": "", "sources": [], "error": str(e)}
-
-    @router.get("/api/search/providers")
-    async def list_search_providers():
-        """Return available search providers with config status."""
-        providers = []
-        for pid, (label, needs_key, needs_url) in PROVIDER_INFO.items():
-            if pid == "disabled":
-                continue
-            available = True
-            if needs_key and not _get_provider_key(pid):
-                available = False
-            if needs_url and pid == "searxng" and not _get_search_instance():
-                available = False
-            providers.append({
-                "id": pid,
-                "label": label,
-                "available": available,
-            })
-        return providers
-
-    @router.post("/api/search/query")
-    async def search_with_provider(request: Request) -> Dict[str, Any]:
-        """Search using a specific provider. Used by compare search mode."""
-        values = await _request_values(request)
-        query = str(values.get("query") or values.get("q") or "").strip()
-        provider = str(values.get("provider") or "").strip()
-        try:
-            count = int(values.get("count") or values.get("limit") or 10)
-        except Exception:
-            count = 10
-        if not query:
-            return {"results": [], "provider": provider, "error": "query is required"}
-        if provider not in PROVIDER_INFO or provider == "disabled":
-            return {"results": [], "provider": provider, "error": "Unknown provider"}
-        t0 = time.time()
-        try:
-            results = _call_provider(provider, query, min(count, 20))
-            elapsed = round(time.time() - t0, 2)
-            return {"results": results, "provider": provider, "time": elapsed}
-        except Exception as e:
-            elapsed = round(time.time() - t0, 2)
-            logger.error(f"Search provider {provider} failed: {e}")
-            return {"results": [], "provider": provider, "time": elapsed, "error": str(e)}
-
-    return router
+_sys.modules[__name__] = _canonical

routes/session_routes.py

@@ -801,15 +801,6 @@ def setup_session_routes(
         finally:
             db.close()
 
-    @router.get("/history/{sid}")
-    def get_history(request: Request, sid: str):
-        _verify_session_owner(request, sid)
-        try:
-            session = session_manager.get_session(sid)
-        except KeyError:
-            raise HTTPException(404, f"Session {sid} not found")
-        return {"history": [msg.to_dict() for msg in session.history]}
-    
     @router.get("/session/{sid}/export")
     def export_session(request: Request, sid: str, fmt: str = "md", filename: str = ""):
         """Export conversation history as a downloadable file.

routes/skills_routes.py

@@ -1409,7 +1409,7 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
 
         # Prefer the configured DEFAULT (→ Utility) model — not the current chat
         # session's model. Fall back to the caller's session model only if unset.
-        url, model, headers = resolve_endpoint("default", owner=user)
+        url, model, headers = resolve_endpoint("utility", owner=user)
         if not url or not model:
             url = url or ((body.get("endpoint_url") or "").strip() or None)
             model = model or ((body.get("model") or "").strip() or None)

routes/vault/__init__.py

@@ -0,0 +1,5 @@
+"""Vault route domain package (slice 2k, #4082/#4071).
+
+Contains vault_routes.py, migrated from the flat routes/ directory.
+Backward-compat shim at routes/vault_routes.py re-exports from here.
+"""

routes/vault/vault_routes.py

@@ -0,0 +1,242 @@
+"""
+vault_routes.py
+
+Vaultwarden / Bitwarden CLI integration — config and unlock endpoints.
+Stores the BW_SESSION key in data/vault.json with restrictive permissions.
+"""
+
+import json
+import logging
+import os
+import shutil
+import asyncio
+from pathlib import Path
+from datetime import datetime
+from fastapi import APIRouter, Request
+from pydantic import BaseModel
+
+from core.middleware import require_admin
+from core.platform_compat import IS_WINDOWS, safe_chmod, which_tool
+from src.constants import VAULT_FILE as _VAULT_FILE
+
+logger = logging.getLogger(__name__)
+
+VAULT_FILE = Path(_VAULT_FILE)
+
+
+def _find_bw() -> str:
+    """Locate the bw binary, checking PATH and common npm-global locations.
+
+    On Windows the Bitwarden CLI shim is `bw.cmd`/`bw.exe`, resolved by
+    which_tool via PATHEXT.
+    """
+    p = which_tool("bw")
+    if p:
+        return p
+    if IS_WINDOWS:
+        appdata = os.environ.get("APPDATA", os.path.expanduser("~"))
+        for candidate in (
+            os.path.join(appdata, "npm", "bw.cmd"),
+            os.path.join(appdata, "npm", "bw.exe"),
+        ):
+            if os.path.isfile(candidate):
+                return candidate
+        return "bw"
+    home = os.path.expanduser("~")
+    for candidate in (
+        f"{home}/.npm-global/bin/bw",
+        f"{home}/.nvm/versions/node/*/bin/bw",
+        "/usr/local/bin/bw",
+        "/opt/homebrew/bin/bw",
+    ):
+        if "*" in candidate:
+            import glob
+            for m in glob.glob(candidate):
+                if os.path.isfile(m) and os.access(m, os.X_OK):
+                    return m
+        elif os.path.isfile(candidate) and os.access(candidate, os.X_OK):
+            return candidate
+    return "bw"  # fall back to PATH lookup (will FileNotFoundError, handled below)
+
+
+def _load_config() -> dict:
+    if VAULT_FILE.exists():
+        try:
+            data = json.loads(VAULT_FILE.read_text(encoding="utf-8"))
+            return data if isinstance(data, dict) else {}
+        except Exception:
+            pass
+    return {}
+
+
+def _save_config(cfg: dict):
+    VAULT_FILE.parent.mkdir(parents=True, exist_ok=True)
+    VAULT_FILE.write_text(json.dumps(cfg, indent=2), encoding="utf-8")
+    # POSIX: restrict the BW_SESSION store to 0o600. Windows: no-op (profile dir
+    # is ACL-restricted already).
+    safe_chmod(str(VAULT_FILE), 0o600)
+
+
+async def _run_bw(args: list, session: str = None, input_text: str = None,
+                  bw_password: str = None) -> tuple:
+    env = {}
+    env.update(os.environ)
+    if session:
+        env["BW_SESSION"] = session
+    # Secrets must never be passed as argv — process arguments are world-readable
+    # via `ps` / `/proc/<pid>/cmdline` to any local user. Keep --passwordenv
+    # support for bw commands that need it; unlock/login callers should prefer
+    # stdin so the master password is not left in the child environment either.
+    if bw_password is not None:
+        env["BW_PASSWORD"] = bw_password
+    bw_path = _find_bw()
+    try:
+        proc = await asyncio.create_subprocess_exec(
+            bw_path, *args,
+            stdin=asyncio.subprocess.PIPE if input_text else None,
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+            env=env,
+        )
+    except FileNotFoundError:
+        return "", "bw CLI not installed (install `nodejs-bitwarden-cli` or `bitwarden-cli`)", 127
+    except Exception as e:
+        return "", f"Failed to launch bw: {e}", 1
+    try:
+        stdout, stderr = await proc.communicate(input=input_text.encode() if input_text else None)
+    except Exception as e:
+        return "", f"bw subprocess error: {e}", 1
+    return stdout.decode(errors="replace").strip(), stderr.decode(errors="replace").strip(), proc.returncode
+
+
+class VaultConfig(BaseModel):
+    server_url: str = ""
+    email: str = ""
+
+
+class VaultUnlockRequest(BaseModel):
+    master_password: str
+
+
+class VaultLoginRequest(BaseModel):
+    email: str
+    master_password: str
+
+
+def setup_vault_routes():
+    router = APIRouter(prefix="/api/vault", tags=["vault"])
+
+    @router.get("/config")
+    async def get_config(request: Request):
+        """Return vault config (no sensitive fields)."""
+        require_admin(request)
+        cfg = _load_config()
+        return {
+            "server_url": cfg.get("server_url", ""),
+            "email": cfg.get("email", ""),
+            "unlocked": bool(cfg.get("session")),
+            "unlocked_at": cfg.get("unlocked_at", ""),
+            "bw_installed": await _check_bw_installed(),
+        }
+
+    @router.post("/config")
+    async def save_config(req: VaultConfig, request: Request):
+        """Save vault URL + email. Runs 'bw config server' to point at Vaultwarden."""
+        require_admin(request)
+        cfg = _load_config()
+        cfg["server_url"] = req.server_url.strip().rstrip("/")
+        cfg["email"] = req.email.strip()
+
+        if cfg["server_url"]:
+            _, stderr, rc = await _run_bw(["config", "server", cfg["server_url"]])
+            if rc != 0:
+                return {"ok": False, "error": f"bw config failed: {stderr[:300]}"}
+
+        _save_config(cfg)
+        return {"ok": True}
+
+    @router.post("/login")
+    async def login(req: VaultLoginRequest, request: Request):
+        """Log in to Vaultwarden (required once per account)."""
+        require_admin(request)
+        cfg = _load_config()
+        # Update email
+        cfg["email"] = req.email
+        _save_config(cfg)
+
+        stdout, stderr, rc = await _run_bw(
+            ["login", req.email, "--raw"],
+            input_text=req.master_password + "\n",
+        )
+        if rc != 0:
+            # Already logged in is OK
+            if "already logged in" in stderr.lower():
+                return {"ok": True, "already": True}
+            return {"ok": False, "error": f"Login failed: {stderr[:300]}"}
+        # bw login --raw prints session key on success (when 2FA disabled)
+        if stdout:
+            cfg["session"] = stdout
+            cfg["unlocked_at"] = datetime.utcnow().isoformat()
+            _save_config(cfg)
+        return {"ok": True}
+
+    @router.post("/unlock")
+    async def unlock(req: VaultUnlockRequest, request: Request):
+        """Unlock the vault and save the session key."""
+        require_admin(request)
+        # Pass the master password on stdin, not argv. argv is visible through
+        # `ps` / /proc/<pid>/cmdline; stdin also avoids leaving the secret in
+        # the child process environment.
+        stdout, stderr, rc = await _run_bw(
+            ["unlock", "--raw"],
+            input_text=req.master_password + "\n",
+        )
+        if rc != 0:
+            return {"ok": False, "error": f"Unlock failed: {stderr[:300]}"}
+        session = stdout.strip()
+        if not session:
+            return {"ok": False, "error": "bw returned empty session"}
+        cfg = _load_config()
+        cfg["session"] = session
+        cfg["unlocked_at"] = datetime.utcnow().isoformat()
+        _save_config(cfg)
+        return {"ok": True, "message": "Vault unlocked"}
+
+    @router.post("/lock")
+    async def lock(request: Request):
+        """Lock the vault (clear session from config)."""
+        require_admin(request)
+        cfg = _load_config()
+        cfg.pop("session", None)
+        cfg.pop("unlocked_at", None)
+        _save_config(cfg)
+        # Also tell bw to lock
+        await _run_bw(["lock"])
+        return {"ok": True, "message": "Vault locked"}
+
+    @router.post("/logout")
+    async def logout(request: Request):
+        """Log out of the Bitwarden CLI completely."""
+        require_admin(request)
+        await _run_bw(["logout"])
+        cfg = _load_config()
+        cfg.pop("session", None)
+        cfg.pop("email", None)
+        cfg.pop("unlocked_at", None)
+        _save_config(cfg)
+        return {"ok": True}
+
+    return router
+
+
+async def _check_bw_installed() -> bool:
+    try:
+        proc = await asyncio.create_subprocess_exec(
+            _find_bw(), "--version",
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+        )
+        await proc.communicate()
+        return proc.returncode == 0
+    except Exception:
+        return False

routes/vault_routes.py

@@ -1,242 +1,14 @@
-"""
-vault_routes.py
+"""Backward-compat shim — canonical location is routes/vault/vault_routes.py.
 
-Vaultwarden / Bitwarden CLI integration — config and unlock endpoints.
-Stores the BW_SESSION key in data/vault.json with restrictive permissions.
+This module is replaced in ``sys.modules`` by the canonical module object so
+that ``import routes.vault_routes``, ``from routes.vault_routes import X``,
+and the ``import ... as vr`` + ``monkeypatch.setattr(vr, ...)`` pattern used
+by test_vault_password_not_in_argv.py all operate on the *same* object.
+Keeps existing import paths working after slice 2k (#4082/#4071).
 """
 
-import json
-import logging
-import os
-import shutil
-import asyncio
-from pathlib import Path
-from datetime import datetime
-from fastapi import APIRouter, Request
-from pydantic import BaseModel
+import sys as _sys
 
-from core.middleware import require_admin
-from core.platform_compat import IS_WINDOWS, safe_chmod, which_tool
-from src.constants import VAULT_FILE as _VAULT_FILE
+from routes.vault import vault_routes as _canonical  # noqa: F401
 
-logger = logging.getLogger(__name__)
-
-VAULT_FILE = Path(_VAULT_FILE)
-
-
-def _find_bw() -> str:
-    """Locate the bw binary, checking PATH and common npm-global locations.
-
-    On Windows the Bitwarden CLI shim is `bw.cmd`/`bw.exe`, resolved by
-    which_tool via PATHEXT.
-    """
-    p = which_tool("bw")
-    if p:
-        return p
-    if IS_WINDOWS:
-        appdata = os.environ.get("APPDATA", os.path.expanduser("~"))
-        for candidate in (
-            os.path.join(appdata, "npm", "bw.cmd"),
-            os.path.join(appdata, "npm", "bw.exe"),
-        ):
-            if os.path.isfile(candidate):
-                return candidate
-        return "bw"
-    home = os.path.expanduser("~")
-    for candidate in (
-        f"{home}/.npm-global/bin/bw",
-        f"{home}/.nvm/versions/node/*/bin/bw",
-        "/usr/local/bin/bw",
-        "/opt/homebrew/bin/bw",
-    ):
-        if "*" in candidate:
-            import glob
-            for m in glob.glob(candidate):
-                if os.path.isfile(m) and os.access(m, os.X_OK):
-                    return m
-        elif os.path.isfile(candidate) and os.access(candidate, os.X_OK):
-            return candidate
-    return "bw"  # fall back to PATH lookup (will FileNotFoundError, handled below)
-
-
-def _load_config() -> dict:
-    if VAULT_FILE.exists():
-        try:
-            data = json.loads(VAULT_FILE.read_text(encoding="utf-8"))
-            return data if isinstance(data, dict) else {}
-        except Exception:
-            pass
-    return {}
-
-
-def _save_config(cfg: dict):
-    VAULT_FILE.parent.mkdir(parents=True, exist_ok=True)
-    VAULT_FILE.write_text(json.dumps(cfg, indent=2), encoding="utf-8")
-    # POSIX: restrict the BW_SESSION store to 0o600. Windows: no-op (profile dir
-    # is ACL-restricted already).
-    safe_chmod(str(VAULT_FILE), 0o600)
-
-
-async def _run_bw(args: list, session: str = None, input_text: str = None,
-                  bw_password: str = None) -> tuple:
-    env = {}
-    env.update(os.environ)
-    if session:
-        env["BW_SESSION"] = session
-    # Secrets must never be passed as argv — process arguments are world-readable
-    # via `ps` / `/proc/<pid>/cmdline` to any local user. Keep --passwordenv
-    # support for bw commands that need it; unlock/login callers should prefer
-    # stdin so the master password is not left in the child environment either.
-    if bw_password is not None:
-        env["BW_PASSWORD"] = bw_password
-    bw_path = _find_bw()
-    try:
-        proc = await asyncio.create_subprocess_exec(
-            bw_path, *args,
-            stdin=asyncio.subprocess.PIPE if input_text else None,
-            stdout=asyncio.subprocess.PIPE,
-            stderr=asyncio.subprocess.PIPE,
-            env=env,
-        )
-    except FileNotFoundError:
-        return "", "bw CLI not installed (install `nodejs-bitwarden-cli` or `bitwarden-cli`)", 127
-    except Exception as e:
-        return "", f"Failed to launch bw: {e}", 1
-    try:
-        stdout, stderr = await proc.communicate(input=input_text.encode() if input_text else None)
-    except Exception as e:
-        return "", f"bw subprocess error: {e}", 1
-    return stdout.decode(errors="replace").strip(), stderr.decode(errors="replace").strip(), proc.returncode
-
-
-class VaultConfig(BaseModel):
-    server_url: str = ""
-    email: str = ""
-
-
-class VaultUnlockRequest(BaseModel):
-    master_password: str
-
-
-class VaultLoginRequest(BaseModel):
-    email: str
-    master_password: str
-
-
-def setup_vault_routes():
-    router = APIRouter(prefix="/api/vault", tags=["vault"])
-
-    @router.get("/config")
-    async def get_config(request: Request):
-        """Return vault config (no sensitive fields)."""
-        require_admin(request)
-        cfg = _load_config()
-        return {
-            "server_url": cfg.get("server_url", ""),
-            "email": cfg.get("email", ""),
-            "unlocked": bool(cfg.get("session")),
-            "unlocked_at": cfg.get("unlocked_at", ""),
-            "bw_installed": await _check_bw_installed(),
-        }
-
-    @router.post("/config")
-    async def save_config(req: VaultConfig, request: Request):
-        """Save vault URL + email. Runs 'bw config server' to point at Vaultwarden."""
-        require_admin(request)
-        cfg = _load_config()
-        cfg["server_url"] = req.server_url.strip().rstrip("/")
-        cfg["email"] = req.email.strip()
-
-        if cfg["server_url"]:
-            _, stderr, rc = await _run_bw(["config", "server", cfg["server_url"]])
-            if rc != 0:
-                return {"ok": False, "error": f"bw config failed: {stderr[:300]}"}
-
-        _save_config(cfg)
-        return {"ok": True}
-
-    @router.post("/login")
-    async def login(req: VaultLoginRequest, request: Request):
-        """Log in to Vaultwarden (required once per account)."""
-        require_admin(request)
-        cfg = _load_config()
-        # Update email
-        cfg["email"] = req.email
-        _save_config(cfg)
-
-        stdout, stderr, rc = await _run_bw(
-            ["login", req.email, "--raw"],
-            input_text=req.master_password + "\n",
-        )
-        if rc != 0:
-            # Already logged in is OK
-            if "already logged in" in stderr.lower():
-                return {"ok": True, "already": True}
-            return {"ok": False, "error": f"Login failed: {stderr[:300]}"}
-        # bw login --raw prints session key on success (when 2FA disabled)
-        if stdout:
-            cfg["session"] = stdout
-            cfg["unlocked_at"] = datetime.utcnow().isoformat()
-            _save_config(cfg)
-        return {"ok": True}
-
-    @router.post("/unlock")
-    async def unlock(req: VaultUnlockRequest, request: Request):
-        """Unlock the vault and save the session key."""
-        require_admin(request)
-        # Pass the master password on stdin, not argv. argv is visible through
-        # `ps` / /proc/<pid>/cmdline; stdin also avoids leaving the secret in
-        # the child process environment.
-        stdout, stderr, rc = await _run_bw(
-            ["unlock", "--raw"],
-            input_text=req.master_password + "\n",
-        )
-        if rc != 0:
-            return {"ok": False, "error": f"Unlock failed: {stderr[:300]}"}
-        session = stdout.strip()
-        if not session:
-            return {"ok": False, "error": "bw returned empty session"}
-        cfg = _load_config()
-        cfg["session"] = session
-        cfg["unlocked_at"] = datetime.utcnow().isoformat()
-        _save_config(cfg)
-        return {"ok": True, "message": "Vault unlocked"}
-
-    @router.post("/lock")
-    async def lock(request: Request):
-        """Lock the vault (clear session from config)."""
-        require_admin(request)
-        cfg = _load_config()
-        cfg.pop("session", None)
-        cfg.pop("unlocked_at", None)
-        _save_config(cfg)
-        # Also tell bw to lock
-        await _run_bw(["lock"])
-        return {"ok": True, "message": "Vault locked"}
-
-    @router.post("/logout")
-    async def logout(request: Request):
-        """Log out of the Bitwarden CLI completely."""
-        require_admin(request)
-        await _run_bw(["logout"])
-        cfg = _load_config()
-        cfg.pop("session", None)
-        cfg.pop("email", None)
-        cfg.pop("unlocked_at", None)
-        _save_config(cfg)
-        return {"ok": True}
-
-    return router
-
-
-async def _check_bw_installed() -> bool:
-    try:
-        proc = await asyncio.create_subprocess_exec(
-            _find_bw(), "--version",
-            stdout=asyncio.subprocess.PIPE,
-            stderr=asyncio.subprocess.PIPE,
-        )
-        await proc.communicate()
-        return proc.returncode == 0
-    except Exception:
-        return False
+_sys.modules[__name__] = _canonical

routes/webhook/__init__.py

@@ -0,0 +1,5 @@
+"""Webhook route domain package (slice 2l, #4082/#4071).
+
+Contains webhook_routes.py, migrated from the flat routes/ directory.
+Backward-compat shim at routes/webhook_routes.py re-exports from here.
+"""

routes/webhook/webhook_routes.py

@@ -0,0 +1,395 @@
+"""Webhook, API Token, and sync chat routes."""
+
+import uuid
+import logging
+from typing import Optional
+
+import httpx
+from fastapi import APIRouter, HTTPException, Request, Form
+from pydantic import BaseModel, Field
+
+from core.database import SessionLocal, Webhook, ModelEndpoint
+from src.auth_helpers import owner_filter
+from src.url_security import validate_public_http_url
+from src.webhook_manager import WebhookManager, validate_webhook_url, validate_events
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api", tags=["webhooks"])
+
+# Input limits
+MAX_NAME_LEN = 100
+MAX_URL_LEN = 2048
+MAX_SECRET_LEN = 256
+MAX_MESSAGE_LEN = 32_000
+
+
+from core.middleware import require_admin as _require_admin
+
+
+def _select_api_chat_fallback_endpoint(db, token_owner: Optional[str]):
+    """First enabled ModelEndpoint visible to token_owner — their own rows plus
+    legacy null-owner ("shared") rows. Owner-scoped: an unscoped .first() would
+    let a chat-scoped token fall back onto another user's private endpoint and
+    silently spend that owner's API key/quota. Prefer owner rows before shared
+    rows. Fails closed to null-owner rows only when token_owner is absent.
+    Does not validate base_url — admin-configured local/LAN endpoints remain allowed.
+    """
+    query = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True)  # noqa: E712
+    if token_owner:
+        query = owner_filter(query, ModelEndpoint, token_owner)
+        return query.order_by(ModelEndpoint.owner.desc(), ModelEndpoint.created_at).first()
+    return query.filter(ModelEndpoint.owner == None).order_by(ModelEndpoint.created_at).first()  # noqa: E711
+
+
+def _caller_owns_session(sess_owner, caller) -> bool:
+    """Strict session-ownership gate for the token-authenticated sync-chat
+    endpoint (`POST /api/v1/chat`).
+
+    Mirrors ``_verify_session_owner`` in session_routes.py and the null-owner
+    gates in notes/calendar/gallery: a caller may resume a session ONLY when
+    its owner matches them exactly. A null/empty session owner (legacy or
+    migrated rows) is deliberately NOT resumable by an arbitrary token — the
+    old ``sess_owner and sess_owner != caller`` form skipped the check whenever
+    ``sess_owner`` was falsy, so any chat-scoped token (e.g. a paired mobile
+    device) could resume such a session, inject a message, and read back its
+    history and reuse the owner's endpoint credentials. Fail closed: an
+    unresolvable caller also returns False.
+    """
+    if not caller:
+        return False
+    return sess_owner == caller
+
+
+def setup_webhook_routes(
+    webhook_manager: WebhookManager,
+    auth_manager,
+    session_manager=None,
+    api_key_manager=None,
+) -> APIRouter:
+
+    @router.get("/webhooks")
+    def list_webhooks(request: Request):
+        _require_admin(request)
+        db = SessionLocal()
+        try:
+            hooks = db.query(Webhook).all()
+            return [
+                {
+                    "id": w.id,
+                    "name": w.name,
+                    "url": w.url,
+                    "has_secret": bool(w.secret),
+                    "events": w.events.split(",") if w.events else [],
+                    "is_active": w.is_active,
+                    "last_triggered_at": w.last_triggered_at.isoformat() if w.last_triggered_at else None,
+                    "last_status_code": w.last_status_code,
+                    "last_error": w.last_error,
+                    "created_at": w.created_at.isoformat() if w.created_at else None,
+                }
+                for w in hooks
+            ]
+        finally:
+            db.close()
+
+    @router.post("/webhooks")
+    def create_webhook(
+        request: Request,
+        name: str = Form(""),
+        url: str = Form(""),
+        secret: str = Form(""),
+        events: str = Form(""),
+    ):
+        _require_admin(request)
+        name = name.strip()[:MAX_NAME_LEN]
+        if not name:
+            raise HTTPException(400, "Webhook name is required")
+        try:
+            url = validate_webhook_url(url)
+        except ValueError as e:
+            raise HTTPException(400, str(e))
+        try:
+            events = validate_events(events)
+        except ValueError as e:
+            raise HTTPException(400, str(e))
+
+        secret_val = secret.strip()[:MAX_SECRET_LEN] or None
+        # Encrypt the secret at rest using the same Fernet key as API keys
+        encrypted_secret = None
+        if secret_val and api_key_manager:
+            encrypted_secret = api_key_manager.encrypt_api_key(secret_val)
+        elif secret_val:
+            encrypted_secret = secret_val  # Fallback if no encryption available
+
+        webhook_id = str(uuid.uuid4())[:8]
+        db = SessionLocal()
+        try:
+            db.add(Webhook(
+                id=webhook_id,
+                name=name,
+                url=url,
+                secret=encrypted_secret,
+                events=events,
+                is_active=True,
+            ))
+            db.commit()
+        finally:
+            db.close()
+
+        return {"id": webhook_id, "name": name}
+
+    @router.post("/webhooks/{webhook_id}/test")
+    async def test_webhook(request: Request, webhook_id: str):
+        _require_admin(request)
+        db = SessionLocal()
+        try:
+            wh = db.query(Webhook).filter(Webhook.id == webhook_id).first()
+            if not wh:
+                raise HTTPException(404, "Webhook not found")
+            url, secret = wh.url, wh.secret
+        finally:
+            db.close()
+
+        await webhook_manager.deliver_test(webhook_id, url, secret)
+        return {"status": "sent"}
+
+    @router.patch("/webhooks/{webhook_id}")
+    def toggle_webhook(request: Request, webhook_id: str):
+        _require_admin(request)
+        db = SessionLocal()
+        try:
+            wh = db.query(Webhook).filter(Webhook.id == webhook_id).first()
+            if not wh:
+                raise HTTPException(404, "Webhook not found")
+            wh.is_active = not wh.is_active
+            db.commit()
+            return {"id": webhook_id, "is_active": wh.is_active}
+        finally:
+            db.close()
+
+    @router.delete("/webhooks/{webhook_id}")
+    def delete_webhook(request: Request, webhook_id: str):
+        _require_admin(request)
+        db = SessionLocal()
+        try:
+            deleted = db.query(Webhook).filter(Webhook.id == webhook_id).delete()
+            db.commit()
+            if not deleted:
+                raise HTTPException(404, "Webhook not found")
+        finally:
+            db.close()
+        return {"status": "deleted"}
+
+    # ================================================================
+    # Sync Chat Endpoint (for n8n / Make / Activepieces)
+    # ================================================================
+
+    # Known provider base URLs — auto-resolved from api_key prefix or model name
+    KNOWN_PROVIDERS = {
+        "deepseek": "https://api.deepseek.com/v1",
+        "openai": "https://api.openai.com/v1",
+        "mistral": "https://api.mistral.ai/v1",
+        "groq": "https://api.groq.com/openai/v1",
+        "together": "https://api.together.xyz/v1",
+        "openrouter": "https://openrouter.ai/api/v1",
+        "ollama": "https://ollama.com/api",
+        "opencode-zen": "https://opencode.ai/zen/v1",
+        "opencode-go": "https://opencode.ai/zen/go/v1",
+        "fireworks": "https://api.fireworks.ai/inference/v1",
+        "venice": "https://api.venice.ai/api/v1",
+        "kimi-code": "https://api.kimi.com/coding/v1",
+        "kimicode": "https://api.kimi.com/coding/v1",
+    }
+
+    # Model prefix → provider mapping for auto-detection
+    MODEL_PROVIDER_MAP = {
+        "deepseek": "deepseek",
+        "gpt-": "openai",
+        "o1": "openai",
+        "o3": "openai",
+        "o4": "openai",
+        "mistral": "mistral",
+        "llama": "groq",
+        "mixtral": "groq",
+        "kimi-for-coding": "kimi-code",
+        "kimi": "kimi-code",
+    }
+
+    def _resolve_base_url(model: Optional[str], provider: Optional[str]) -> Optional[str]:
+        """Try to auto-resolve a base URL from provider name or model prefix."""
+        if provider and provider.lower() in KNOWN_PROVIDERS:
+            return KNOWN_PROVIDERS[provider.lower()]
+        if model:
+            model_lower = model.lower()
+            for prefix, prov in MODEL_PROVIDER_MAP.items():
+                if model_lower.startswith(prefix):
+                    return KNOWN_PROVIDERS[prov]
+        return None
+
+    class SyncChatRequest(BaseModel):
+        message: str = Field(..., max_length=MAX_MESSAGE_LEN)
+        model: Optional[str] = Field(None, max_length=200)
+        session: Optional[str] = Field(None, max_length=100)
+        api_key: Optional[str] = Field(None, max_length=256)
+        base_url: Optional[str] = Field(None, max_length=MAX_URL_LEN)
+        provider: Optional[str] = Field(None, max_length=50)
+
+    @router.post("/v1/chat")
+    async def sync_chat(request: Request, body: SyncChatRequest):
+        if not getattr(request.state, "api_token", False):
+            raise HTTPException(403, "This endpoint requires an API token")
+        scopes = set(getattr(request.state, "api_token_scopes", []) or [])
+        if "chat" not in scopes:
+            raise HTTPException(403, "API token is not scoped for chat")
+        token_owner = getattr(request.state, "api_token_owner", None)
+
+        from core.models import ChatMessage
+        from src.llm_core import llm_call_async
+        from src.endpoint_resolver import build_chat_url, build_headers, build_models_url, normalize_base
+
+        message = body.message.strip()
+        if not message:
+            raise HTTPException(400, "Message is required")
+
+        session_id = body.session
+        sess = None
+
+        # --- Case 1: Resume an existing session ---
+        if session_id and session_manager:
+            try:
+                sess = session_manager.get_session(session_id)
+            except (KeyError, Exception):
+                raise HTTPException(404, "Session not found")
+            # SECURITY: verify the API-token's user owns this session — without
+            # this any token holder could resume any user's chat by passing its
+            # ID. The token's user is on request.state.user (set by API-token
+            # middleware); fall back to require_user if not present.
+            try:
+                from src.auth_helpers import get_current_user as _gcu
+                _tok_user = token_owner or getattr(request.state, "user", None) or _gcu(request)
+            except Exception:
+                _tok_user = None
+            # Strict ownership (see _caller_owns_session): fail closed so a
+            # null-owner / cross-owner session can't be resumed by an arbitrary
+            # chat-scoped token.
+            _sess_owner = getattr(sess, "owner", None)
+            if not _caller_owns_session(_sess_owner, _tok_user):
+                raise HTTPException(404, "Session not found")
+
+        # --- Case 2: Direct API key + model (no pre-configured endpoint needed) ---
+        if not sess and body.api_key:
+            api_key = body.api_key.strip()
+            model = body.model or "deepseek-chat"
+
+            # Validate only token-supplied direct base_url; auto-resolved known-provider
+            # URLs are not subject to extra local/LAN blocking beyond existing provider logic.
+            direct_base_url = body.base_url.strip().rstrip("/") if body.base_url else None
+            if direct_base_url:
+                try:
+                    base_url = validate_public_http_url(direct_base_url)
+                except ValueError as e:
+                    detail = str(e).replace("URL", "base_url", 1)
+                    raise HTTPException(400, detail)
+            else:
+                base_url = _resolve_base_url(model, body.provider)
+            if not base_url:
+                raise HTTPException(400,
+                    "Could not auto-detect provider. Pass base_url (e.g. 'https://api.deepseek.com/v1') "
+                    "or provider ('deepseek', 'openai', 'groq', etc.)")
+            base_url = normalize_base(base_url)
+            endpoint_url = build_chat_url(base_url)
+
+            if not session_manager:
+                raise HTTPException(500, "Session manager not available")
+
+            sid = str(uuid.uuid4())
+            sess = session_manager.create_session(
+                session_id=sid, name="API Chat", endpoint_url=endpoint_url,
+                model=model, owner=token_owner,
+            )
+            sess.headers = build_headers(api_key, base_url)
+            session_manager.save_sessions()
+            session_id = sid
+
+        # --- Case 3: Fall back to first configured ModelEndpoint ---
+        if not sess:
+            db = SessionLocal()
+            try:
+                ep = _select_api_chat_fallback_endpoint(db, token_owner)
+            finally:
+                db.close()
+
+            if not ep:
+                raise HTTPException(400,
+                    "No session, api_key, or configured endpoints. "
+                    "Pass api_key + model, or configure an endpoint in Admin.")
+
+            base_url = normalize_base(ep.base_url)
+            endpoint_url = build_chat_url(base_url)
+            model = body.model or "auto"
+            api_key = ep.api_key
+            if getattr(ep, "provider_auth_id", None):
+                try:
+                    from src.endpoint_resolver import resolve_endpoint_runtime
+                    base_url, api_key = resolve_endpoint_runtime(ep, owner=token_owner)
+                    endpoint_url = build_chat_url(base_url)
+                except Exception:
+                    raise HTTPException(500, "Could not resolve endpoint credentials")
+
+            if model == "auto":
+                try:
+                    async with httpx.AsyncClient(timeout=5) as client:
+                        models_url = build_models_url(base_url)
+                        hdrs = build_headers(api_key, base_url)
+                        if models_url:
+                            resp = await client.get(models_url, headers=hdrs)
+                            resp.raise_for_status()
+                            data = resp.json()
+                            items = data if isinstance(data, list) else (data.get("data") or [])
+                            ids = [m.get("id") for m in items if isinstance(m, dict) and m.get("id")]
+                            if not ids and isinstance(data, dict):
+                                ids = [
+                                    m.get("name") or m.get("model")
+                                    for m in (data.get("models") or [])
+                                    if m.get("name") or m.get("model")
+                                ]
+                        else:
+                            import json as _json
+                            ids = _json.loads(ep.cached_models or "[]")
+                        model = ids[0] if ids else "auto"
+                except Exception:
+                    raise HTTPException(500, "Could not discover models from endpoint")
+
+            if not session_manager:
+                raise HTTPException(500, "Session manager not available")
+
+            sid = str(uuid.uuid4())
+            sess = session_manager.create_session(
+                session_id=sid, name="API Chat", endpoint_url=endpoint_url,
+                model=model, owner=token_owner,
+            )
+            if api_key:
+                sess.headers = build_headers(api_key, base_url)
+                session_manager.save_sessions()
+            session_id = sid
+
+        # --- Send message and get response ---
+        sess.add_message(ChatMessage("user", message))
+
+        messages = [{"role": m.role, "content": m.content} for m in sess.history]
+
+        reply = await llm_call_async(
+            sess.endpoint_url, sess.model, messages,
+            headers=sess.headers, timeout=120,
+        )
+        sess.add_message(ChatMessage("assistant", reply))
+        session_manager.save_sessions()
+
+        webhook_manager.fire_and_forget("chat.completed", {
+            "session_id": session_id, "model": sess.model,
+            "user_message": message[:2000], "response": reply[:2000],
+        })
+
+        return {"response": reply, "session_id": session_id, "model": sess.model}
+
+    return router

routes/webhook_routes.py

@@ -1,395 +1,16 @@
-"""Webhook, API Token, and sync chat routes."""
+"""Backward-compat shim — canonical location is routes/webhook/webhook_routes.py.
 
-import uuid
-import logging
-from typing import Optional
+This module is replaced in ``sys.modules`` by the canonical module object so
+that ``import routes.webhook_routes``, ``from routes.webhook_routes import X``,
+``importlib.import_module("routes.webhook_routes")``, and the
+``__import__("routes.webhook_routes", fromlist=[...])`` + ``setattr(wh_mod,
+...)`` pattern used by test_null_owner_gates.py all operate on the *same*
+object. Keeps existing import paths working after slice 2l (#4082/#4071).
+Source-introspection tests read the canonical file by path.
+"""
 
-import httpx
-from fastapi import APIRouter, HTTPException, Request, Form
-from pydantic import BaseModel, Field
+import sys as _sys
 
-from core.database import SessionLocal, Webhook, ModelEndpoint
-from src.auth_helpers import owner_filter
-from src.url_security import validate_public_http_url
-from src.webhook_manager import WebhookManager, validate_webhook_url, validate_events
+from routes.webhook import webhook_routes as _canonical  # noqa: F401
 
-logger = logging.getLogger(__name__)
-
-router = APIRouter(prefix="/api", tags=["webhooks"])
-
-# Input limits
-MAX_NAME_LEN = 100
-MAX_URL_LEN = 2048
-MAX_SECRET_LEN = 256
-MAX_MESSAGE_LEN = 32_000
-
-
-from core.middleware import require_admin as _require_admin
-
-
-def _select_api_chat_fallback_endpoint(db, token_owner: Optional[str]):
-    """First enabled ModelEndpoint visible to token_owner — their own rows plus
-    legacy null-owner ("shared") rows. Owner-scoped: an unscoped .first() would
-    let a chat-scoped token fall back onto another user's private endpoint and
-    silently spend that owner's API key/quota. Prefer owner rows before shared
-    rows. Fails closed to null-owner rows only when token_owner is absent.
-    Does not validate base_url — admin-configured local/LAN endpoints remain allowed.
-    """
-    query = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True)  # noqa: E712
-    if token_owner:
-        query = owner_filter(query, ModelEndpoint, token_owner)
-        return query.order_by(ModelEndpoint.owner.desc(), ModelEndpoint.created_at).first()
-    return query.filter(ModelEndpoint.owner == None).order_by(ModelEndpoint.created_at).first()  # noqa: E711
-
-
-def _caller_owns_session(sess_owner, caller) -> bool:
-    """Strict session-ownership gate for the token-authenticated sync-chat
-    endpoint (`POST /api/v1/chat`).
-
-    Mirrors ``_verify_session_owner`` in session_routes.py and the null-owner
-    gates in notes/calendar/gallery: a caller may resume a session ONLY when
-    its owner matches them exactly. A null/empty session owner (legacy or
-    migrated rows) is deliberately NOT resumable by an arbitrary token — the
-    old ``sess_owner and sess_owner != caller`` form skipped the check whenever
-    ``sess_owner`` was falsy, so any chat-scoped token (e.g. a paired mobile
-    device) could resume such a session, inject a message, and read back its
-    history and reuse the owner's endpoint credentials. Fail closed: an
-    unresolvable caller also returns False.
-    """
-    if not caller:
-        return False
-    return sess_owner == caller
-
-
-def setup_webhook_routes(
-    webhook_manager: WebhookManager,
-    auth_manager,
-    session_manager=None,
-    api_key_manager=None,
-) -> APIRouter:
-
-    @router.get("/webhooks")
-    def list_webhooks(request: Request):
-        _require_admin(request)
-        db = SessionLocal()
-        try:
-            hooks = db.query(Webhook).all()
-            return [
-                {
-                    "id": w.id,
-                    "name": w.name,
-                    "url": w.url,
-                    "has_secret": bool(w.secret),
-                    "events": w.events.split(",") if w.events else [],
-                    "is_active": w.is_active,
-                    "last_triggered_at": w.last_triggered_at.isoformat() if w.last_triggered_at else None,
-                    "last_status_code": w.last_status_code,
-                    "last_error": w.last_error,
-                    "created_at": w.created_at.isoformat() if w.created_at else None,
-                }
-                for w in hooks
-            ]
-        finally:
-            db.close()
-
-    @router.post("/webhooks")
-    def create_webhook(
-        request: Request,
-        name: str = Form(""),
-        url: str = Form(""),
-        secret: str = Form(""),
-        events: str = Form(""),
-    ):
-        _require_admin(request)
-        name = name.strip()[:MAX_NAME_LEN]
-        if not name:
-            raise HTTPException(400, "Webhook name is required")
-        try:
-            url = validate_webhook_url(url)
-        except ValueError as e:
-            raise HTTPException(400, str(e))
-        try:
-            events = validate_events(events)
-        except ValueError as e:
-            raise HTTPException(400, str(e))
-
-        secret_val = secret.strip()[:MAX_SECRET_LEN] or None
-        # Encrypt the secret at rest using the same Fernet key as API keys
-        encrypted_secret = None
-        if secret_val and api_key_manager:
-            encrypted_secret = api_key_manager.encrypt_api_key(secret_val)
-        elif secret_val:
-            encrypted_secret = secret_val  # Fallback if no encryption available
-
-        webhook_id = str(uuid.uuid4())[:8]
-        db = SessionLocal()
-        try:
-            db.add(Webhook(
-                id=webhook_id,
-                name=name,
-                url=url,
-                secret=encrypted_secret,
-                events=events,
-                is_active=True,
-            ))
-            db.commit()
-        finally:
-            db.close()
-
-        return {"id": webhook_id, "name": name}
-
-    @router.post("/webhooks/{webhook_id}/test")
-    async def test_webhook(request: Request, webhook_id: str):
-        _require_admin(request)
-        db = SessionLocal()
-        try:
-            wh = db.query(Webhook).filter(Webhook.id == webhook_id).first()
-            if not wh:
-                raise HTTPException(404, "Webhook not found")
-            url, secret = wh.url, wh.secret
-        finally:
-            db.close()
-
-        await webhook_manager.deliver_test(webhook_id, url, secret)
-        return {"status": "sent"}
-
-    @router.patch("/webhooks/{webhook_id}")
-    def toggle_webhook(request: Request, webhook_id: str):
-        _require_admin(request)
-        db = SessionLocal()
-        try:
-            wh = db.query(Webhook).filter(Webhook.id == webhook_id).first()
-            if not wh:
-                raise HTTPException(404, "Webhook not found")
-            wh.is_active = not wh.is_active
-            db.commit()
-            return {"id": webhook_id, "is_active": wh.is_active}
-        finally:
-            db.close()
-
-    @router.delete("/webhooks/{webhook_id}")
-    def delete_webhook(request: Request, webhook_id: str):
-        _require_admin(request)
-        db = SessionLocal()
-        try:
-            deleted = db.query(Webhook).filter(Webhook.id == webhook_id).delete()
-            db.commit()
-            if not deleted:
-                raise HTTPException(404, "Webhook not found")
-        finally:
-            db.close()
-        return {"status": "deleted"}
-
-    # ================================================================
-    # Sync Chat Endpoint (for n8n / Make / Activepieces)
-    # ================================================================
-
-    # Known provider base URLs — auto-resolved from api_key prefix or model name
-    KNOWN_PROVIDERS = {
-        "deepseek": "https://api.deepseek.com/v1",
-        "openai": "https://api.openai.com/v1",
-        "mistral": "https://api.mistral.ai/v1",
-        "groq": "https://api.groq.com/openai/v1",
-        "together": "https://api.together.xyz/v1",
-        "openrouter": "https://openrouter.ai/api/v1",
-        "ollama": "https://ollama.com/api",
-        "opencode-zen": "https://opencode.ai/zen/v1",
-        "opencode-go": "https://opencode.ai/zen/go/v1",
-        "fireworks": "https://api.fireworks.ai/inference/v1",
-        "venice": "https://api.venice.ai/api/v1",
-        "kimi-code": "https://api.kimi.com/coding/v1",
-        "kimicode": "https://api.kimi.com/coding/v1",
-    }
-
-    # Model prefix → provider mapping for auto-detection
-    MODEL_PROVIDER_MAP = {
-        "deepseek": "deepseek",
-        "gpt-": "openai",
-        "o1": "openai",
-        "o3": "openai",
-        "o4": "openai",
-        "mistral": "mistral",
-        "llama": "groq",
-        "mixtral": "groq",
-        "kimi-for-coding": "kimi-code",
-        "kimi": "kimi-code",
-    }
-
-    def _resolve_base_url(model: Optional[str], provider: Optional[str]) -> Optional[str]:
-        """Try to auto-resolve a base URL from provider name or model prefix."""
-        if provider and provider.lower() in KNOWN_PROVIDERS:
-            return KNOWN_PROVIDERS[provider.lower()]
-        if model:
-            model_lower = model.lower()
-            for prefix, prov in MODEL_PROVIDER_MAP.items():
-                if model_lower.startswith(prefix):
-                    return KNOWN_PROVIDERS[prov]
-        return None
-
-    class SyncChatRequest(BaseModel):
-        message: str = Field(..., max_length=MAX_MESSAGE_LEN)
-        model: Optional[str] = Field(None, max_length=200)
-        session: Optional[str] = Field(None, max_length=100)
-        api_key: Optional[str] = Field(None, max_length=256)
-        base_url: Optional[str] = Field(None, max_length=MAX_URL_LEN)
-        provider: Optional[str] = Field(None, max_length=50)
-
-    @router.post("/v1/chat")
-    async def sync_chat(request: Request, body: SyncChatRequest):
-        if not getattr(request.state, "api_token", False):
-            raise HTTPException(403, "This endpoint requires an API token")
-        scopes = set(getattr(request.state, "api_token_scopes", []) or [])
-        if "chat" not in scopes:
-            raise HTTPException(403, "API token is not scoped for chat")
-        token_owner = getattr(request.state, "api_token_owner", None)
-
-        from core.models import ChatMessage
-        from src.llm_core import llm_call_async
-        from src.endpoint_resolver import build_chat_url, build_headers, build_models_url, normalize_base
-
-        message = body.message.strip()
-        if not message:
-            raise HTTPException(400, "Message is required")
-
-        session_id = body.session
-        sess = None
-
-        # --- Case 1: Resume an existing session ---
-        if session_id and session_manager:
-            try:
-                sess = session_manager.get_session(session_id)
-            except (KeyError, Exception):
-                raise HTTPException(404, "Session not found")
-            # SECURITY: verify the API-token's user owns this session — without
-            # this any token holder could resume any user's chat by passing its
-            # ID. The token's user is on request.state.user (set by API-token
-            # middleware); fall back to require_user if not present.
-            try:
-                from src.auth_helpers import get_current_user as _gcu
-                _tok_user = token_owner or getattr(request.state, "user", None) or _gcu(request)
-            except Exception:
-                _tok_user = None
-            # Strict ownership (see _caller_owns_session): fail closed so a
-            # null-owner / cross-owner session can't be resumed by an arbitrary
-            # chat-scoped token.
-            _sess_owner = getattr(sess, "owner", None)
-            if not _caller_owns_session(_sess_owner, _tok_user):
-                raise HTTPException(404, "Session not found")
-
-        # --- Case 2: Direct API key + model (no pre-configured endpoint needed) ---
-        if not sess and body.api_key:
-            api_key = body.api_key.strip()
-            model = body.model or "deepseek-chat"
-
-            # Validate only token-supplied direct base_url; auto-resolved known-provider
-            # URLs are not subject to extra local/LAN blocking beyond existing provider logic.
-            direct_base_url = body.base_url.strip().rstrip("/") if body.base_url else None
-            if direct_base_url:
-                try:
-                    base_url = validate_public_http_url(direct_base_url)
-                except ValueError as e:
-                    detail = str(e).replace("URL", "base_url", 1)
-                    raise HTTPException(400, detail)
-            else:
-                base_url = _resolve_base_url(model, body.provider)
-            if not base_url:
-                raise HTTPException(400,
-                    "Could not auto-detect provider. Pass base_url (e.g. 'https://api.deepseek.com/v1') "
-                    "or provider ('deepseek', 'openai', 'groq', etc.)")
-            base_url = normalize_base(base_url)
-            endpoint_url = build_chat_url(base_url)
-
-            if not session_manager:
-                raise HTTPException(500, "Session manager not available")
-
-            sid = str(uuid.uuid4())
-            sess = session_manager.create_session(
-                session_id=sid, name="API Chat", endpoint_url=endpoint_url,
-                model=model, owner=token_owner,
-            )
-            sess.headers = build_headers(api_key, base_url)
-            session_manager.save_sessions()
-            session_id = sid
-
-        # --- Case 3: Fall back to first configured ModelEndpoint ---
-        if not sess:
-            db = SessionLocal()
-            try:
-                ep = _select_api_chat_fallback_endpoint(db, token_owner)
-            finally:
-                db.close()
-
-            if not ep:
-                raise HTTPException(400,
-                    "No session, api_key, or configured endpoints. "
-                    "Pass api_key + model, or configure an endpoint in Admin.")
-
-            base_url = normalize_base(ep.base_url)
-            endpoint_url = build_chat_url(base_url)
-            model = body.model or "auto"
-            api_key = ep.api_key
-            if getattr(ep, "provider_auth_id", None):
-                try:
-                    from src.endpoint_resolver import resolve_endpoint_runtime
-                    base_url, api_key = resolve_endpoint_runtime(ep, owner=token_owner)
-                    endpoint_url = build_chat_url(base_url)
-                except Exception:
-                    raise HTTPException(500, "Could not resolve endpoint credentials")
-
-            if model == "auto":
-                try:
-                    async with httpx.AsyncClient(timeout=5) as client:
-                        models_url = build_models_url(base_url)
-                        hdrs = build_headers(api_key, base_url)
-                        if models_url:
-                            resp = await client.get(models_url, headers=hdrs)
-                            resp.raise_for_status()
-                            data = resp.json()
-                            items = data if isinstance(data, list) else (data.get("data") or [])
-                            ids = [m.get("id") for m in items if isinstance(m, dict) and m.get("id")]
-                            if not ids and isinstance(data, dict):
-                                ids = [
-                                    m.get("name") or m.get("model")
-                                    for m in (data.get("models") or [])
-                                    if m.get("name") or m.get("model")
-                                ]
-                        else:
-                            import json as _json
-                            ids = _json.loads(ep.cached_models or "[]")
-                        model = ids[0] if ids else "auto"
-                except Exception:
-                    raise HTTPException(500, "Could not discover models from endpoint")
-
-            if not session_manager:
-                raise HTTPException(500, "Session manager not available")
-
-            sid = str(uuid.uuid4())
-            sess = session_manager.create_session(
-                session_id=sid, name="API Chat", endpoint_url=endpoint_url,
-                model=model, owner=token_owner,
-            )
-            if api_key:
-                sess.headers = build_headers(api_key, base_url)
-                session_manager.save_sessions()
-            session_id = sid
-
-        # --- Send message and get response ---
-        sess.add_message(ChatMessage("user", message))
-
-        messages = [{"role": m.role, "content": m.content} for m in sess.history]
-
-        reply = await llm_call_async(
-            sess.endpoint_url, sess.model, messages,
-            headers=sess.headers, timeout=120,
-        )
-        sess.add_message(ChatMessage("assistant", reply))
-        session_manager.save_sessions()
-
-        webhook_manager.fire_and_forget("chat.completed", {
-            "session_id": session_id, "model": sess.model,
-            "user_message": message[:2000], "response": reply[:2000],
-        })
-
-        return {"response": reply, "session_id": session_id, "model": sess.model}
-
-    return router
+_sys.modules[__name__] = _canonical

services/memory/__init__.py

@@ -2,7 +2,7 @@
 """Memory service — persistent memory storage and retrieval."""
 
 from .service import MemoryService, Memory, MemorySearchResult
-from .memory import MemoryManager
+from .memory import MemoryManager, MemoryStoreUnreadable
 from .memory_vector import MemoryVectorStore
 
 __all__ = [
@@ -10,5 +10,6 @@ __all__ = [
     "Memory",
     "MemorySearchResult",
     "MemoryManager",
+    "MemoryStoreUnreadable",
     "MemoryVectorStore",
 ]

services/memory/memory.py

@@ -5,6 +5,16 @@ application runtime instantiates ``src.memory.MemoryManager``, so keeping a
 parallel implementation here risks silent drift between import paths.
 """
 
-from src.memory import MemoryManager, get_text_similarity, tokenize
+from src.memory import (
+    MemoryManager,
+    MemoryStoreUnreadable,
+    get_text_similarity,
+    tokenize,
+)
 
-__all__ = ["MemoryManager", "get_text_similarity", "tokenize"]
+__all__ = [
+    "MemoryManager",
+    "MemoryStoreUnreadable",
+    "get_text_similarity",
+    "tokenize",
+]

services/memory/memory_extractor.py

@@ -17,6 +17,8 @@ import os
 import re
 from typing import Optional
 
+from src.memory import MemoryStoreUnreadable
+
 logger = logging.getLogger(__name__)
 
 
@@ -387,7 +389,13 @@ async def extract_and_store(
         # Get owner from session
         _owner = getattr(session, 'owner', None)
 
-        existing = memory_manager.load_all()
+        # Strict load: this is a read-modify-write. Degrading to [] here would
+        # save only the newly extracted facts and drop the entire store.
+        try:
+            existing = memory_manager.load_all_for_update()
+        except MemoryStoreUnreadable as e:
+            logger.error("Skipping auto memory extraction, store unreadable: %s", e)
+            return
         added = 0
 
         for fact in facts:
@@ -626,7 +634,18 @@ async def audit_memories(
 
         # Merge audited entries back with other users' entries
         if owner:
-            all_entries = memory_manager.load_all()
+            # Strict load: the merge below reconstructs the whole file. If this
+            # degraded to [] we would save only this owner's audited slice and
+            # destroy every other tenant's memories.
+            try:
+                all_entries = memory_manager.load_all_for_update()
+            except MemoryStoreUnreadable as e:
+                logger.error("Aborting memory audit save, store unreadable: %s", e)
+                return {
+                    "before": before_count,
+                    "after": before_count,
+                    "error": "store_unreadable",
+                }
             audited_ids = {e["id"] for e in final_entries}
             other_entries = [e for e in all_entries if e.get("owner") != owner and (e.get("owner") is not None)]
             # Also keep legacy entries that weren't part of this audit

services/memory/skill_format.py

@@ -50,7 +50,7 @@ import json
 import logging
 import re
 from dataclasses import dataclass, field
-from datetime import datetime
+from datetime import datetime, timezone
 from typing import Any, Dict, List, Optional
 
 logger = logging.getLogger(__name__)
@@ -441,4 +441,4 @@ class Skill:
 
 
 def _now_iso() -> str:
-    return datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
+    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")

services/tts/tts_service.py

@@ -2,6 +2,7 @@
 """Multi-provider TTS service — dispatches to local Kokoro, OpenAI-compatible API, or browser."""
 
 import io
+import os
 import wave
 import logging
 import hashlib
@@ -42,6 +43,11 @@ class TTSService:
         self.cache_dir.mkdir(parents=True, exist_ok=True)
         self._kokoro = None  # lazy-init
         
+        try:
+            self.max_cache_bytes = int(os.getenv("ODYSSEUS_TTS_CACHE_MAX_BYTES", 500 * 1024 * 1024))
+        except ValueError:
+            self.max_cache_bytes = 500 * 1024 * 1024
+
     # ── Settings ──
 
     def _load_settings(self) -> dict:
@@ -89,6 +95,53 @@ class TTSService:
         ext = ".mp3" if (len(data) >= 3 and (data[:3] == b'ID3' or (data[0] == 0xff and (data[1] & 0xe0) == 0xe0))) else ".wav"
         (self.cache_dir / f"{key}{ext}").write_bytes(data)
 
+        self._enforce_cache_limit()
+
+    def _enforce_cache_limit(self):
+            """Evicts oldest files if the cache exceeds the configured byte limit."""
+            if self.max_cache_bytes <= 0:
+                return
+
+            try:
+                files = []
+                total_size = 0
+
+                # Safely scan files and sum sizes, ignoring files deleted mid-scan
+                for f in self.cache_dir.iterdir():
+                    try:
+                        if f.is_file() and f.suffix.lower() in (".mp3", ".wav"):
+                            files.append(f)
+                            total_size += f.stat().st_size
+                    except OSError:
+                        continue
+
+                if total_size > self.max_cache_bytes:
+                    logger.info(
+                        f"TTS cache ({total_size} bytes) exceeded limit ({self.max_cache_bytes} bytes). Evicting oldest files."
+                    )
+
+                    # Sort files by modification time (oldest first)
+                    try:
+                        files.sort(key=lambda f: f.stat().st_mtime)
+                    except OSError as e:
+                        logger.warning(f"Failed to sort cache files by mtime: {e}")
+
+                    # Trim down to 80% of max capacity
+                    target_size = self.max_cache_bytes * 0.8
+
+                    while files and total_size > target_size:
+                        f = files.pop(0)
+                        try:
+                            size = f.stat().st_size
+                            f.unlink()
+                            total_size -= size
+                        except OSError as e:
+                            logger.warning(f"Failed to evict cache file {f}: {e}")
+                            continue
+
+            except Exception as e:
+                logger.warning(f"Error enforcing TTS cache limit: {e}", exc_info=True)
+
     def clear_cache(self):
         count = 0
         for f in self.cache_dir.glob("*.*"):

src/agent_loop.py

@@ -12,7 +12,7 @@ import json
 import re
 import time
 import logging
-from typing import AsyncGenerator, List, Dict, Optional, Set
+from typing import Any, AsyncGenerator, List, Dict, Optional, Set
 from urllib.parse import urlparse
 
 from src.llm_core import (

src/ai_interaction.py

@@ -22,6 +22,7 @@ import time
 from typing import Any, Awaitable, Callable, Dict, Optional, Tuple
 
 from src.constants import GENERATED_IMAGES_DIR
+from src.memory import MemoryStoreUnreadable
 
 logger = logging.getLogger(__name__)
 
@@ -384,7 +385,15 @@ async def do_manage_memory(content: str, session_id: Optional[str] = None, owner
             return {"error": "Memory text cannot be empty"}
 
         entry = _memory_manager.add_entry(text, source="ai_agent", category=category, owner=owner)
-        memories = _memory_manager.load_all()
+        # Strict load: this is a read-modify-write, and it is the path an
+        # ordinary "remember that I prefer X" takes. Degrading to [] here would
+        # save just this one entry over a store we only failed to read,
+        # atomically destroying every memory in it (issue #5673).
+        try:
+            memories = _memory_manager.load_all_for_update()
+        except MemoryStoreUnreadable as e:
+            logger.error("Refusing to add memory, store unreadable: %s", e)
+            return {"error": "Memory store is temporarily unreadable — nothing was saved."}
         memories.append(entry)
         _memory_manager.save(memories)
 

src/integrations.py

@@ -1,11 +1,14 @@
+import ipaddress
 import json
 import os
+import time
 import uuid
 import logging
 import re
 from typing import Dict, List, Optional, Any
 from urllib.parse import urljoin, urlparse, urlunparse
 
+import httpcore
 import httpx
 from fastapi import HTTPException
 
@@ -354,6 +357,152 @@ def _find_integration(identifier: str) -> Optional[Dict[str, Any]]:
     return None
 
 
+# httpcore raises its own exception hierarchy; map the ones a simple request can
+# surface back to their httpx equivalents so the caller's `except httpx.*` blocks
+# below behave exactly as they did with the default transport.
+_HTTPCORE_TO_HTTPX_EXC = {
+    httpcore.ConnectError: httpx.ConnectError,
+    httpcore.ConnectTimeout: httpx.ConnectTimeout,
+    httpcore.NetworkError: httpx.NetworkError,
+    httpcore.PoolTimeout: httpx.PoolTimeout,
+    httpcore.ProtocolError: httpx.ProtocolError,
+    httpcore.ReadError: httpx.ReadError,
+    httpcore.ReadTimeout: httpx.ReadTimeout,
+    httpcore.RemoteProtocolError: httpx.RemoteProtocolError,
+    httpcore.TimeoutException: httpx.TimeoutException,
+    httpcore.WriteError: httpx.WriteError,
+    httpcore.WriteTimeout: httpx.WriteTimeout,
+}
+
+
+class _PinnedAsyncBackend(httpcore.AsyncNetworkBackend):
+    """Network backend that connects only to the pre-validated IPs, in order.
+
+    Every address here came out of the single SSRF resolution, so moving to the
+    next one after a connect failure is not re-resolution — it's ordinary
+    multi-address fallback restricted to the set the guard already approved.
+    httpcore takes TLS SNI and the ``Host`` header from the request URL rather
+    than the connect host, so pinning the socket destination leaves certificate
+    validation and vhost routing pointed at the original hostname.
+    """
+
+    def __init__(self, ips: List[ipaddress._BaseAddress]):
+        self._ips = [str(ip) for ip in ips]
+        self._real = httpcore.AnyIOBackend()
+
+    async def connect_tcp(self, host, port, timeout=None, local_address=None,
+                          socket_options=None):
+        # One shared connect budget: each attempt gets the time left until the
+        # original deadline, so N dead addresses can't stretch the connect
+        # phase to N * timeout.
+        deadline = None if timeout is None else time.monotonic() + timeout
+        last_exc: Optional[Exception] = None
+        for ip in self._ips:
+            remaining = None if deadline is None else max(0.0, deadline - time.monotonic())
+            try:
+                return await self._real.connect_tcp(
+                    ip, port, remaining, local_address, socket_options
+                )
+            except (httpcore.ConnectError, httpcore.ConnectTimeout) as exc:
+                last_exc = exc
+                if deadline is not None and time.monotonic() >= deadline:
+                    break
+        raise last_exc
+
+    async def connect_unix_socket(self, path, timeout=None, socket_options=None):
+        return await self._real.connect_unix_socket(path, timeout, socket_options)
+
+    async def sleep(self, seconds: float) -> None:
+        return await self._real.sleep(seconds)
+
+
+class _PinnedAsyncTransport(httpx.AsyncBaseTransport):
+    """httpx transport that pins the TCP connect to the pre-resolved IP(s).
+
+    Kept local, mirroring the per-module pinned transports web fetch and
+    webhook delivery already carry, rather than coupling api_call to the
+    webhook subsystem. The request URL passes through unchanged, so SNI and the
+    ``Host`` header stay the original hostname; only the socket destination is
+    pinned, which is what closes the rebinding window.
+    """
+
+    def __init__(self, ips: List[ipaddress._BaseAddress]):
+        self._pinned_ips = list(ips)
+        self._pool = httpcore.AsyncConnectionPool(
+            # Reuse the CA trust the default httpx client would build (certifi
+            # plus SSL_CERT_FILE / SSL_CERT_DIR when trust_env is set) so
+            # swapping in this transport doesn't quietly change which chains
+            # verify. ssl.create_default_context() would use system roots.
+            ssl_context=httpx.create_ssl_context(),
+            http1=True,
+            http2=False,
+            network_backend=_PinnedAsyncBackend(ips),
+        )
+
+    async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
+        core_req = httpcore.Request(
+            method=request.method,
+            url=httpcore.URL(
+                scheme=request.url.raw_scheme,
+                host=request.url.raw_host,
+                port=request.url.port,
+                target=request.url.raw_path,
+            ),
+            headers=request.headers.raw,
+            content=request.stream,
+            extensions=request.extensions,
+        )
+        try:
+            core_resp = await self._pool.handle_async_request(core_req)
+            content = b"".join([chunk async for chunk in core_resp.aiter_stream()])
+            await core_resp.aclose()
+        except Exception as exc:
+            mapped = _HTTPCORE_TO_HTTPX_EXC.get(type(exc))
+            if mapped is not None:
+                raise mapped(str(exc)) from exc
+            raise
+        return httpx.Response(
+            status_code=core_resp.status,
+            headers=core_resp.headers,
+            content=content,
+            extensions=core_resp.extensions,
+        )
+
+    async def aclose(self) -> None:
+        await self._pool.aclose()
+
+
+def _validated_ips(raw_ips: List[str]) -> List[ipaddress._BaseAddress]:
+    """Return every entry that parses as an IP address, de-duplicated, order
+    preserved.
+
+    check_outbound_url only reports ok when *all* of these classify as safe, so
+    the whole list is guard-approved and any of them is a legitimate connect
+    target. Skipping unparseable entries mirrors how the guard walks the same
+    resolver output.
+
+    De-duplication matters because the resolver is getaddrinfo(host, None) with
+    no socktype filter, so glibc reports the same address once per socktype
+    (SOCK_STREAM/SOCK_DGRAM/SOCK_RAW) — a single-homed host comes back three
+    times. Without this, the connect fallback would spend the shared deadline
+    retrying one dead address instead of moving on to a genuinely different one.
+    """
+    ips: List[ipaddress._BaseAddress] = []
+    seen = set()
+    for raw in raw_ips:
+        if not isinstance(raw, str):
+            continue
+        try:
+            ip = ipaddress.ip_address(raw.split("%")[0])  # strip IPv6 zone id
+        except ValueError:
+            continue
+        if ip in seen:
+            continue
+        seen.add(ip)
+        ips.append(ip)
+    return ips
+
+
 async def execute_api_call(
     integration_id: str,
     method: str,
@@ -409,13 +558,31 @@ async def execute_api_call(
     # loopback for locked-down deployments. Private stays allowed by default
     # because LAN integrations (Home Assistant, Miniflux, ntfy) are the
     # primary use case.
-    from src.url_safety import check_outbound_url
+    from src.url_safety import check_outbound_url, _default_resolver
     block_private = os.getenv(
         "INTEGRATION_API_BLOCK_PRIVATE_IPS", "false"
     ).lower() == "true"
-    ok, reason = check_outbound_url(url, block_private=block_private)
+    # Resolve the host exactly once and remember the IPs the guard validated so
+    # the request below can be pinned to them. check_outbound_url only reports
+    # (ok, reason); a plain httpx client re-resolves the host at connect time,
+    # which reopens a DNS-rebinding TOCTOU — a base_url host that answers with a
+    # public IP for the guard and then flips to 169.254.169.254 for the connect
+    # would reach cloud metadata with the integration's auth headers attached.
+    resolved_ips: List[str] = []
+
+    def _recording_resolver(host: str) -> List[str]:
+        ips = _default_resolver(host)
+        resolved_ips[:] = ips
+        return ips
+
+    ok, reason = check_outbound_url(
+        url, block_private=block_private, resolver=_recording_resolver
+    )
     if not ok:
         return {"error": f"URL rejected: {reason}", "exit_code": 1}
+    pinned_ips = _validated_ips(resolved_ips)
+    if not pinned_ips:
+        return {"error": "URL rejected: host did not resolve to a usable address", "exit_code": 1}
 
     method = method.upper()
 
@@ -455,7 +622,9 @@ async def execute_api_call(
             auth = httpx.BasicAuth(parts[0], parts[1])
 
     try:
-        async with httpx.AsyncClient(timeout=30.0) as client:
+        async with httpx.AsyncClient(
+            timeout=30.0, transport=_PinnedAsyncTransport(pinned_ips)
+        ) as client:
             response = await client.request(
                 method,
                 url,

src/llm_core.py

@@ -1237,15 +1237,27 @@ def _anthropic_rejects_temperature(model: str) -> bool:
         return False
     # `(?<![a-z])` anchors "opus" to a word boundary so a substring match like
     # `oct-opus`/`octopus-4-8` can't be read as Opus (it would otherwise strip
-    # temperature). Cap the minor at 1-2 digits and forbid a trailing digit so a
-    # dated id like `claude-opus-4-20250514` (Opus 4.0) parses as major-only (no
-    # minor match, kept) instead of reading the date `20250514` as a giant minor
-    # that would falsely test >= 4.7. Dated 4.7+ snapshots (`claude-opus-4-7-
-    # 20260201`) keep their explicit minor and are still matched.
-    match = re.search(r"(?<![a-z])opus[-_]?(\d+)[-_.](\d{1,2})(?!\d)", model.lower())
+    # temperature). Both version components are capped at 1-2 digits and forbid a
+    # trailing digit, so an 8-digit date can never be read as a version number:
+    # `claude-opus-4-20250514` (Opus 4.0) parses as major-only rather than reading
+    # `20250514` as a giant minor, and `claude-3-opus-20240229` (legacy Claude 3
+    # Opus, date directly after "opus-") fails to match at all rather than reading
+    # the date as a giant major. Dated 4.7+ snapshots (`claude-opus-4-7-20260201`)
+    # keep their explicit minor and are still matched.
+    #
+    # The minor is optional and a missing minor reads as `.0`, so major-only ids
+    # like `claude-opus-5` are correctly treated as >= 4.7 (issue #5753). Without
+    # this, every Opus 5 call kept `temperature` and failed with HTTP 400 — visible
+    # only on paths that pass a temperature, e.g. scheduled tasks inheriting
+    # `stream_agent_loop`'s 0.3 default, which returned empty responses.
+    match = re.search(
+        r"(?<![a-z])opus[-_]?(\d{1,2})(?!\d)(?:[-_.](\d{1,2})(?!\d))?", model.lower()
+    )
     if not match:
         return False
-    return (int(match.group(1)), int(match.group(2))) >= (4, 7)
+    major = int(match.group(1))
+    minor = int(match.group(2)) if match.group(2) else 0
+    return (major, minor) >= (4, 7)
 
 # Reasoning effort level sent to Mistral thinking-capable models. Mistral's
 # API accepts "high", "medium", "low", "none" — see

src/memory.py

@@ -10,6 +10,18 @@ from datetime import datetime
 
 logger = logging.getLogger(__name__)
 
+
+class MemoryStoreUnreadable(RuntimeError):
+    """memory.json exists on disk but could not be read or parsed.
+
+    "The contents are unknown" is categorically different from "there are no
+    memories". A read-modify-write caller that conflates the two appends to an
+    empty view and then persists it, destroying the whole store — the writes
+    are atomic, so the loss is durable. Raised by
+    :meth:`MemoryManager.load_all_for_update` so those callers fail closed.
+    """
+
+
 def tokenize(text: str) -> List[str]:
     """Simple tokenizer that splits on whitespace and removes punctuation."""
     return [word.strip('.,!?";') for word in text.split()]
@@ -110,21 +122,69 @@ class MemoryManager:
             with open(self.memory_file, 'w', encoding='utf-8') as f:
                 json.dump([], f, ensure_ascii=False, indent=2)
     
-    def load_all(self) -> List[Dict]:
-        """Load all memory entries from JSON file (unfiltered)."""
+    def _read_entries(self) -> List[Dict]:
+        """Parse the store, or raise :class:`MemoryStoreUnreadable`.
+
+        Returns ``[]`` only when the file genuinely does not exist. Every other
+        failure mode raises, so callers can tell "no memories" apart from
+        "couldn't read the memories".
+        """
         if not os.path.exists(self.memory_file):
             return []
 
         try:
             with open(self.memory_file, "r", encoding="utf-8") as f:
                 data = json.load(f)
-                if isinstance(data, list):
-                    return self._validate_entries(data)
-        except (json.JSONDecodeError, PermissionError) as e:
-            logger.error("Error loading memory.json: %s", e)
-            return self._migrate_from_legacy()
+        except OSError as e:
+            # PermissionError is an OSError (a scanner holding the file, a
+            # permissions problem, bad media).
+            raise MemoryStoreUnreadable(
+                f"cannot read {self.memory_file}: {e}"
+            ) from e
+        except json.JSONDecodeError as e:
+            # This is the branch that actually destroyed stores: the file reads
+            # back fine, so nothing stops the save that follows. A truncated
+            # memory.json is reachable because core/database.py rewrites it with
+            # a plain open(..,"w") + json.dump during migration.
+            #
+            # Preserved behaviour: a corrupt store still gets one shot at the
+            # pre-JSON memory.txt migration. Only raise when that finds nothing,
+            # so we never report "empty" for a store we simply failed to parse.
+            legacy = self._migrate_from_legacy()
+            if legacy:
+                return legacy
+            raise MemoryStoreUnreadable(
+                f"{self.memory_file} is not valid JSON: {e}"
+            ) from e
 
-        return []
+        if not isinstance(data, list):
+            raise MemoryStoreUnreadable(
+                f"{self.memory_file} is not a JSON array (got {type(data).__name__})"
+            )
+        return self._validate_entries(data)
+
+    def load_all(self) -> List[Dict]:
+        """Load all memory entries from JSON file (unfiltered).
+
+        Lenient by design: this feeds display, search, and context-injection
+        paths, so an unreadable store degrades to an empty list rather than
+        breaking chat. Never build a value from this that you intend to save
+        back — use :meth:`load_all_for_update` for that.
+        """
+        try:
+            return self._read_entries()
+        except MemoryStoreUnreadable as e:
+            logger.error("Error loading memory.json: %s", e)
+            return []
+
+    def load_all_for_update(self) -> List[Dict]:
+        """Load for a read-modify-write cycle.
+
+        Propagates :class:`MemoryStoreUnreadable` instead of degrading to ``[]``
+        so a caller can never append to an empty view and persist it over a
+        store that was only temporarily unreadable (issue #5673).
+        """
+        return self._read_entries()
 
     def load(self, owner: str = None) -> List[Dict]:
         """Load memory entries, optionally filtered by owner."""
@@ -135,7 +195,12 @@ class MemoryManager:
 
     def claim_ownerless(self, owner: str):
         """Assign all ownerless memory entries to the given owner."""
-        entries = self.load_all()
+        try:
+            entries = self.load_all_for_update()
+        except MemoryStoreUnreadable as e:
+            # Skip the sweep rather than rewrite the store from an unknown view.
+            logger.error("Skipping ownerless claim, memory store unreadable: %s", e)
+            return
         changed = False
         claimed = 0
         for entry in entries:
@@ -235,7 +300,12 @@ class MemoryManager:
         if not ids:
             return
         id_set = set(ids)
-        entries = self.load_all()
+        try:
+            entries = self.load_all_for_update()
+        except MemoryStoreUnreadable as e:
+            # Best-effort counter; never worth rewriting the store blind.
+            logger.error("Skipping uses bump, memory store unreadable: %s", e)
+            return
         changed = False
         for e in entries:
             if e.get("id") in id_set:

src/memory_provider.py

@@ -157,7 +157,11 @@ class NativeMemoryProvider(MemoryProvider):
         if metadata:
             entry["metadata"] = dict(metadata)
 
-        memories = self.memory_manager.load_all()
+        # Strict load: read-modify-write. `load_all` degrades an unreadable
+        # store to [], which would save this single entry over everything
+        # already stored (issue #5673). The provider API has no error channel,
+        # so MemoryStoreUnreadable propagates to the caller.
+        memories = self.memory_manager.load_all_for_update()
         memories.append(entry)
         self.memory_manager.save(memories)
 
@@ -223,7 +227,10 @@ class NativeMemoryProvider(MemoryProvider):
         ]
 
     async def delete(self, memory_id: str, *, owner: Optional[str] = None) -> bool:
-        memories = self.memory_manager.load_all()
+        # Strict load for the same reason: `remaining` is derived from this
+        # list and saved back, so it must never be built from a store we
+        # failed to read.
+        memories = self.memory_manager.load_all_for_update()
         remaining = []
         deleted_id = None
 

src/tool_parsing.py

@@ -187,8 +187,12 @@ _FUNCTION_MODEL_NAME_RE = re.compile(
 _FUNCTION_MODEL_PARAMS_OPEN_RE = re.compile(r"<parameters>\s*", re.IGNORECASE)
 _FUNCTION_MODEL_PARAMS_CLOSE_RE = re.compile(r"</parameters>", re.IGNORECASE)
 _QWEN_ROLE_MARKER_RE = re.compile(r"</?\|(?:assistant|assistan|user|system|tool)\|>?|</\|end\|>?", re.IGNORECASE)
+# At least one pipe is required around `end`. Both pipes used to be optional
+# (`\|?end\|?`), which also matched a bare `end` on its own line and deleted it
+# from ordinary prose and from Ruby/Lua/shell snippets that close blocks with
+# one; see #5547. `|end`, `end|`, `|end|` and `/|end|` still strip as before.
 _QWEN_BARE_MARKER_RE = re.compile(
-    r"(?:^|[\t\r\n ])(?:\|?end\|?|/?\|end\|)(?=[\t\r\n ]|$)|"
+    r"(?:^|[\t\r\n ])(?:/?\|end\||\|end|end\|)(?=[\t\r\n ]|$)|"
     r"(?:^|[\t\r\n ])assistan(?:t)?(?=[\t\r\n ]|$)",
     re.IGNORECASE,
 )

src/tools/system.py

@@ -46,7 +46,9 @@ async def do_manage_skills(content: str, owner: Optional[str] = None) -> Dict:
     except ValueError:
         return {"error": "Invalid JSON arguments", "exit_code": 1}
 
-    action = (args.get("action") or "").lower()
+    action = (args.get("action") or "").strip().lower()
+    if not action:
+        return {"error": "action is required (list|view|view_ref|add|edit|patch|publish|delete|search)", "exit_code": 1}
     from services.memory.skills import SkillsManager
     from services.memory.skill_format import Skill, slugify
     from src.constants import DATA_DIR
@@ -55,7 +57,7 @@ async def do_manage_skills(content: str, owner: Optional[str] = None) -> Dict:
     # Accept legacy `skill_id` as an alias for `name`.
     name = (args.get("name") or args.get("skill_id") or "").strip()
 
-    if action in ("list", "index", ""):
+    if action in ("list", "index"):
         all_skills = sm.load(owner=owner)
         if not all_skills:
             return {"results": "No skills yet. Create one with action='add'."}

static/app.js

@@ -10,14 +10,20 @@ import modelsModule from './js/models.js?v=20260715startupcalm2';
 import ragModule from './js/rag.js';
 import presetsModule from './js/presets.js';
 import searchModule from './js/search.js';
-import chatModule from './js/chat.js?v=20260722ctxheader4';
+import chatModule from './js/chat.js?v=20260801fix1';
 import compareModule from './js/compare/index.js?v=20260723compareicon2';
 import documentModule from './js/document.js?v=20260722emailfastindex1';
 import searchChatModule from './js/search-chat.js';
 import { makeWindowDraggable } from './js/windowDrag.js';
+import {
+  revealApplicationShellAfterPaint,
+  runDeferredRouteOpener,
+  deferRouteOpener,
+  settleSessionHydration
+} from './js/startupShell.js';
 import markdownModule from './js/markdown.js';
 import chatRenderer from './js/chatRenderer.js?v=20260722emailfastindex1';
-import sessionModule from './js/sessions.js?v=20260722ctxheader4';
+import sessionModule from './js/sessions.js';
 import memoryModule from './js/memory.js?v=20260722memoryloading1';
 import voiceRecorderModule from './js/voiceRecorder.js';
 import censorModule from './js/censor.js';
@@ -1217,12 +1223,13 @@ function initializeEventListeners() {
     '/library':  () => sessionModule && sessionModule.openLibrary && sessionModule.openLibrary(),
   };
   const _opener = _routeOpen[urlPath];
-  // Defer the opener — at this point in init, the modules whose handlers
-  // we trigger (#rail-new-session click handler, the email-section header
-  // click handler in emailInbox, sessionModule's loaded session list) are
-  // still being wired up further down in this same function. Stash the
-  // opener so it runs from sessionModule.loadSessions().finally() below.
-  if (_opener) window._odysseusRouteOpener = _opener;
+  // Defer the opener — at this point in init, the modules whose handlers we
+  // trigger (#rail-new-session click handler, the email-section header click
+  // handler in emailInbox, sessionModule) are still being wired up further
+  // down in this same function. startupShell decides when it can run: as soon
+  // as wiring completes, or — for the routes that read the session list —
+  // once /api/sessions has settled.
+  deferRouteOpener(urlPath, _opener);
 
   // Archive browser tool button
   const toolLibraryBtn = el('tool-library-btn');
@@ -1689,12 +1696,20 @@ function initializeEventListeners() {
   
   const newMemoryInput = el('new-memory-input');
   if (newMemoryInput) {
-    newMemoryInput.addEventListener('keypress', (e) => {
-      if (e.key === 'Enter') {
+    // keydown, not the deprecated keypress: keypress is not guaranteed to
+    // fire for Enter everywhere, which left the Add Memory form with no
+    // working submit path (#5828).
+    newMemoryInput.addEventListener('keydown', (e) => {
+      if (e.key === 'Enter' && !e.isComposing) {
+        e.preventDefault();
         memoryModule.addNewMemory();
       }
     });
   }
+  const newMemoryAddBtn = el('new-memory-add-btn');
+  if (newMemoryAddBtn) {
+    newMemoryAddBtn.addEventListener('click', () => memoryModule.addNewMemory());
+  }
 
 // Voice recording is handled by the dual-purpose send/mic button (see below)
 
@@ -3908,85 +3923,10 @@ function startOdysseusApp() {
   const messageInput = el('message');
   const modelPickerWrap = document.getElementById('model-picker-wrap');
 
-  function _readComposerPromptHistory() {
-    const chatBox = document.getElementById('chat-history');
-    if (!chatBox) return [];
-    return Array.from(chatBox.querySelectorAll('.msg-user'))
-      .reverse()
-      .map(msg => {
-        const body = msg.querySelector('.body');
-        return msg.dataset?.raw || (body ? body.textContent : '') || '';
-      })
-      .filter(Boolean);
-  }
-
-  if (messageInput && !messageInput._odysseusPromptRecallCapture) {
-    messageInput._odysseusPromptRecallCapture = true;
-    let recallHistory = [];
-    let recallIndex = -1;
-    let lastRecalled = '';
-    const norm = (v) => String(v || '').replace(/\r\n/g, '\n').trimEnd();
-    messageInput.addEventListener('input', () => {
-      if (norm(messageInput.value) === norm(lastRecalled)) return;
-      recallHistory = [];
-      recallIndex = -1;
-      lastRecalled = '';
-      try { delete messageInput.dataset.odysseusRecallIndex; } catch {}
-    }, true);
-    messageInput.addEventListener('keydown', (e) => {
-      if (e.key !== 'ArrowUp' && e.key !== 'ArrowDown') return;
-      if (e.shiftKey || e.altKey || e.ctrlKey || e.metaKey || e.isComposing) return;
-      if (window._ghostAutocomplete?.isActive?.()) return;
-      const fresh = _readComposerPromptHistory();
-      const history = fresh.length ? fresh : recallHistory;
-      if (!history.length) return;
-      const current = norm(messageInput.value);
-      let currentIndex = current ? history.findIndex(item => norm(item) === current) : -1;
-      if (current && currentIndex < 0 && current === norm(lastRecalled)) currentIndex = recallIndex;
-      if (current && currentIndex < 0) {
-        const markedIndex = Number(messageInput.dataset.odysseusRecallIndex);
-        if (Number.isInteger(markedIndex) && markedIndex >= 0 && markedIndex < history.length) {
-          currentIndex = markedIndex;
-        }
-      }
-      e.preventDefault();
-      e.stopPropagation();
-      e.stopImmediatePropagation();
-      if (e.key === 'ArrowDown') {
-        if (currentIndex < 0) return;
-        const nextIndex = currentIndex - 1;
-        if (nextIndex < 0) {
-          recallHistory = history;
-          recallIndex = -1;
-          lastRecalled = '';
-          try { delete messageInput.dataset.odysseusRecallIndex; } catch {}
-          messageInput.value = '';
-          try { messageInput.selectionStart = messageInput.selectionEnd = 0; } catch {}
-          try { uiModule.autoResize(messageInput); } catch {}
-          return;
-        }
-        const recalled = history[nextIndex];
-        recallHistory = history;
-        recallIndex = nextIndex;
-        lastRecalled = recalled;
-        try { messageInput.dataset.odysseusRecallIndex = String(nextIndex); } catch {}
-        messageInput.value = recalled;
-        try { messageInput.selectionStart = messageInput.selectionEnd = recalled.length; } catch {}
-        try { uiModule.autoResize(messageInput); } catch {}
-        return;
-      }
-      const nextIndex = currentIndex >= 0 ? Math.min(currentIndex + 1, history.length - 1) : 0;
-      const recalled = history[nextIndex];
-      if (!recalled) return;
-      recallHistory = history;
-      recallIndex = nextIndex;
-      lastRecalled = recalled;
-      try { messageInput.dataset.odysseusRecallIndex = String(nextIndex); } catch {}
-      messageInput.value = recalled;
-      try { messageInput.selectionStart = messageInput.selectionEnd = recalled.length; } catch {}
-      try { uiModule.autoResize(messageInput); } catch {}
-    }, true);
-  }
+  // ArrowUp/ArrowDown prompt recall on #message lives in
+  // static/js/composerArrowUpRecall.js (wired from chat.js). Do not re-add a
+  // copy here: two capture-phase listeners on the same textarea meant the one
+  // without the draft guard won and ate unsent multi-line prompts (#5862).
 
   const _sendIcon = '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M12 19V5M5 12l7-7 7 7"/></svg>';
   const _micIcon = '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 1a3 3 0 0 0-3 3v8a3 3 0 0 0 6 0V4a3 3 0 0 0-3-3z"/><path d="M19 10v2a7 7 0 0 1-14 0v-2"/><line x1="12" y1="19" x2="12" y2="23"/><line x1="8" y1="23" x2="16" y2="23"/></svg>';
@@ -4382,6 +4322,10 @@ function startOdysseusApp() {
   // Load initial data
   presetsModule.loadPresets(uiModule.showError);
 
+  // Core wiring is complete for this turn — reveal the shell independently of
+  // the session-list request.
+  revealApplicationShellAfterPaint();
+
   if (sessionModule) {
     sessionModule.initDependencies({
       API_BASE: API_BASE,
@@ -4393,21 +4337,19 @@ function startOdysseusApp() {
       scrollHistory: uiModule.scrollHistoryInstant
     });
 
-    // Load sessions first (critical path) — remove loader when done
-    sessionModule.loadSessions()
-      .catch(e => console.warn('loadSessions error:', e))
-      .finally(() => {
-        const loader = document.getElementById('app-loader');
-        if (loader) { loader.style.opacity = '0'; setTimeout(() => loader.remove(), 300); }
-        // Fire any URL route opener now that sessions + module wiring are
-        // ready. Deferred from up top of init for exactly this reason.
-        if (window._odysseusRouteOpener) {
-          try { window._odysseusRouteOpener(); } catch (_) {}
-          window._odysseusRouteOpener = null;
-        }
-      });
+    // sessionModule is now wired, so every route opener has the modules it
+    // drives. The ones that read no session data open here rather than
+    // queueing behind /api/sessions.
+    runDeferredRouteOpener();
+
+    // The shell is already usable at this point; session hydration is
+    // sidebar-local and settles on its own schedule.
+    settleSessionHydration(() => sessionModule.loadSessions());
   } else {
     console.error('Session module not loaded!');
+    // Nothing will hydrate. Settle immediately so the sidebar exposes the
+    // failure; session-dependent routes must remain unopened without data.
+    settleSessionHydration(null);
   }
 
   const runNonCriticalStartup = (fn, delay = 4000) => {

static/index.html

@@ -248,11 +248,11 @@
       }, { once: true });
     })();
   </script>
-  <link rel="stylesheet" href="/static/style.css?v=20260723tasksbulkfeedback1">
-  <link rel="modulepreload" href="/static/app.js?v=20260723tasksbulkfeedback1">
-  <link rel="modulepreload" href="/static/js/chat.js?v=20260722ctxheader4">
+  <link rel="stylesheet" href="/static/style.css?v=20260808startupshell1">
+  <link rel="modulepreload" href="/static/app.js?v=20260808startupshell1">
+  <link rel="modulepreload" href="/static/js/chat.js?v=20260801fix1">
   <link rel="modulepreload" href="/static/js/ui.js">
-  <link rel="modulepreload" href="/static/js/sessions.js?v=20260722ctxheader4">
+  <link rel="modulepreload" href="/static/js/sessions.js">
   <link rel="modulepreload" href="/static/js/markdown.js">
 </head>
 <body>
@@ -286,7 +286,13 @@
       if(!document.getElementById('app-loader')){clearInterval(iv);return}
       render();
     },150);
-    setTimeout(function(){var l=document.getElementById('app-loader');if(l){l.style.opacity='0';setTimeout(function(){l.remove()},300)}},5000);
+    // startupShell.js hides the loader as soon as the shell is wired; it calls
+    // back here to stop the wave because this interval is owned by this script.
+    window.__odysseusLoaderWaveStop=function(){clearInterval(iv)};
+    // Last-resort fallback for a boot that never reaches app.js at all. Must
+    // still REMOVE the node: sessions.js reads its presence as "startup in
+    // progress" and stops clearing the composer while it is around.
+    setTimeout(function(){var l=document.getElementById('app-loader');if(l){clearInterval(iv);l.style.opacity='0';setTimeout(function(){l.remove()},300)}},5000);
   })();
   </script>
   <!-- Memory Management Modal -->
@@ -365,6 +371,7 @@
                 <span class="skill-rich-ph"><span class="k">Add a memory</span> &mdash; e.g. 'I prefer concise replies' <svg class="k" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="vertical-align:-2px;margin-left:4px;" aria-hidden="true"><polyline points="9 10 4 15 9 20"/><path d="M20 4v7a4 4 0 0 1-4 4H4"/></svg></span>
               </div>
               <select id="new-memory-category" class="memory-edit-cat-select" aria-label="Memory category"></select>
+              <button type="button" id="new-memory-add-btn" class="theme-io-btn" title="Save this memory" style="flex:none;height:28px;font-size:12px;"><svg width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="vertical-align:-2px;margin-right:4px;" aria-hidden="true"><line x1="12" y1="5" x2="12" y2="19"/><line x1="5" y1="12" x2="19" y2="12"/></svg>Add</button>
             </div>
           </div>
           <div class="admin-card">
@@ -812,7 +819,13 @@
             <button class="session-bulk-btn" id="session-bulk-cancel" title="Cancel"><svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg></button>
           </div>
         </div>
-        <div id="session-list" role="listbox"></div>
+        <div id="session-list" role="listbox">
+          <!-- Sidebar-local bootstrap state. renderSessionList() replaces the
+               whole list on first hydration, so this row is transient. -->
+          <div id="session-list-loading" class="list-item session-list-bootstrap" role="option" aria-disabled="true" aria-live="polite" aria-atomic="true">
+            <span class="grow muted" data-session-list-status>Loading chats…</span>
+          </div>
+        </div>
       </div>
       <!-- Hidden dropdown for session actions -->
       <div id="session-actions-dropdown" class="dropdown hidden">
@@ -1005,7 +1018,7 @@
       var tips = mobile ? phone : desktop;
       var el = document.getElementById('welcome-tip');
       if (el) {
-        el.textContent = 'Pick a model if you want, or just type.';
+        el.textContent = tips[Math.floor(Math.random() * tips.length)];
       }
       fetch('/api/version').then(function(r){return r.json()}).then(function(d){
         if (d.version) window._appVersion = d.version;
@@ -2504,7 +2517,7 @@
 <script type="module" src="/static/js/ui.js"></script>
 <script type="module" src="/static/js/markdown.js"></script>
 <script type="module" src="/static/js/dragSort.js"></script>
-<script type="module" src="/static/js/sessions.js?v=20260722ctxheader4"></script>
+<script type="module" src="/static/js/sessions.js"></script>
 <script type="module" src="/static/js/memory.js?v=20260722memoryloading1"></script>
 <script type="module" src="/static/js/skills.js"></script>
 <script type="module" src="/static/js/tourHints.js"></script>
@@ -2522,7 +2535,7 @@
 <script type="module" src="/static/js/chatRenderer.js?v=20260722emailfastindex1"></script>
 <script type="module" src="/static/js/codeRunner.js"></script>
 <script type="module" src="/static/js/chatStream.js?v=20260722emailfastindex1"></script>
-  <script type="module" src="/static/js/chat.js?v=20260722ctxheader4"></script>
+  <script type="module" src="/static/js/chat.js?v=20260801fix1"></script>
 <script type="module" src="/static/js/cookbook.js"></script>
 <script src="/static/js/cookbookSchedule.js"></script>
 <script type="module" src="/static/js/search-chat.js"></script>
@@ -2530,7 +2543,7 @@
 <script type="module" src="/static/js/censor.js"></script>
 <script type="module" src="/static/js/settings.js?v=20260723compareicon1"></script>
 <script type="module" src="/static/js/assistant.js"></script>
-<script type="module" src="/static/app.js?v=20260723tasksbulkfeedback1"></script>  <!-- app.js must be LAST -->
+<script type="module" src="/static/app.js?v=20260808startupshell1"></script>  <!-- app.js must be LAST -->
   <script type="module" src="/static/js/init.js?v=20260715freshroot3"></script>
 <script type="module" src="/static/js/a11y.js"></script>
 <script nonce="{{CSP_NONCE}}">if('serviceWorker' in navigator){navigator.serviceWorker.register('/static/sw.js').catch(()=>{});}</script>

static/js/MODULE_SUMMARY.md

@@ -61,6 +61,7 @@ The largest and most central subsystem. Chat submission → backend SSE → prog
 | **`chatRenderer.js`** | Message DOM construction: `addMessage`, role labels, model route labels, color coding, footers, metrics, code blocks, sources boxes (`web`/`research`/`RAG`), findings box, images, report links, ask-user cards, welcome screen, and transcript utilities. |
 | **`streamingRenderer.js`** | Incremental streaming renderer used by `chat.js`. Freezes finalized DOM blocks and only re-renders the growing tail to avoid flicker and O(N²) re-parsing. |
 | **`streamingSegmenter.js`** | Splits a token stream into display units (text vs code fences) for `streamingRenderer.js`. |
+| **`liveThinkingThrottle.js`** | Trailing-edge coalescer for the live thinking block in `chat.js`: one DOM commit per 100 ms carrying the latest reasoning text, with `flush`/`cancel` for terminal and session-switch paths. |
 | **`slashCommands.js`** | Slash-command registry (`/help`, `/setup`, etc.), parsing, and dispatch handlers. Exported functions are consumed by `chat.js` and `slashAutocomplete.js`. |
 | **`slashAutocomplete.js`** | Composer autocomplete popup for `/` commands. |
 | **`composerArrowUpRecall.js`** | Recall last user message with `↑` on an empty composer. |

static/js/chatRenderer.js

@@ -478,7 +478,10 @@ const DSML_STRAY_RE = /<\s*\/?\s*[｜|]+\s*DSML\s*[｜|]+[^>]*>/gi;
 const DSML_INVOKE_RE = /<\s*[｜|]+\s*DSML\s*[｜|]+\s*invoke\b[^>]*>[\s\S]*?(?:<\s*\/\s*[｜|]+\s*DSML\s*[｜|]+\s*invoke\s*>|$)/gi;
 const RAW_OPENAI_TOOL_JSON_RE = /(?:\[\s*)?\{\s*"function"\s*:\s*\{[\s\S]*?\}\s*,\s*"id"\s*:\s*"[^"]*"\s*,\s*"type"\s*:\s*"function"\s*\}\s*\]?/gi;
 const QWEN_ROLE_MARKER_RE = /<\/?\|(?:assistant|assistan|user|system|tool)\|>?|<\/\|end\|>?/gi;
-const QWEN_BARE_MARKER_RE = /(?:^|[\t\r\n ])(?:\|?end\|?|\/?\|end\|)(?=[\t\r\n ]|$)|(?:^|[\t\r\n ])assistan(?:t)?(?=[\t\r\n ]|$)/gi;
+// Keep in sync with _QWEN_BARE_MARKER_RE in src/tool_parsing.py. At least one
+// pipe is required around `end`: with both optional (`\|?end\|?`) this also ate
+// a bare `end` on its own line, breaking Ruby/Lua/shell snippets (#5547).
+const QWEN_BARE_MARKER_RE = /(?:^|[\t\r\n ])(?:\/?\|end\||\|end|end\|)(?=[\t\r\n ]|$)|(?:^|[\t\r\n ])assistan(?:t)?(?=[\t\r\n ]|$)/gi;
 // Self-narration about tool results (model echoing stdout/exit_code)
 const TOOL_NARRATION_RE = /(?:The (?:result|output) shows?:?\s*)?-?\s*(?:stdout|stderr|exit_code):\s*.+/gi;
 

static/js/composerArrowUpRecall.js

@@ -143,9 +143,9 @@ export function wireArrowUpRecall(composer, getUserMessages, options = {}) {
       return;
     }
 
-    // ArrowUp owns prompt history in the chat composer. If the current text
-    // is not already a recalled prompt, start from newest instead of letting
-    // the browser move the caret inside the textarea.
+    // ArrowUp walks older prompts. An unmatched draft already returned above,
+    // so reaching here means the composer is empty or holds a recalled prompt
+    // — the caret-navigation case is never hijacked.
     const nextIndex = currentIndex >= 0 ? Math.min(currentIndex + 1, history.length - 1) : 0;
     const recalled = history[nextIndex];
     if (!recalled) {

static/js/emailInbox.js

@@ -149,6 +149,7 @@ let _loading = false;
 let _expanded = false;
 let _docModule = null;
 let _listSpinner = null;
+let _openEmailRequestSeq = 0;
 let _senderFilter = null;       // email address (lowercased) to filter by, or null
 let _senderFilterLabel = null;  // display label for the active filter chip
 let _showEmailTags = localStorage.getItem('odysseus.email.showTags') !== '0';
@@ -187,7 +188,7 @@ export function init(documentModule) {
       } catch (_) {}
       if (opts.compose) { _composeNew(); return; }
       if (opts.email) {
-        await _openEmail(opts.email, null, opts.emailData, opts.mode || 'reply', opts.noteHint || '');
+        await _openEmail(opts.email, null, opts.emailData, opts.mode || 'reply', opts.noteHint || '', '', opts.mailboxContext || null);
       }
     },
   });
@@ -751,7 +752,21 @@ function _createEmailItem(em) {
   return item;
 }
 
-async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', noteHint = '', prefilledBody = '') {
+async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', noteHint = '', prefilledBody = '', mailboxContext = null) {
+  const openRequestSeq = ++_openEmailRequestSeq;
+  const folderAtStart = mailboxContext?.messageFolder || _currentFolder;
+  const accountAtStart = mailboxContext?.accountId ?? (window.__odysseusActiveEmailAccount || '');
+  const accountQueryAtStart = accountAtStart ? `&account_id=${encodeURIComponent(accountAtStart)}` : '';
+  const mailboxContextIsCurrent = typeof mailboxContext?.isCurrent === 'function'
+    ? mailboxContext.isCurrent
+    : () => (
+        folderAtStart === _currentFolder &&
+        accountAtStart === (window.__odysseusActiveEmailAccount || '')
+      );
+  const isCurrentOpen = () => (
+    openRequestSeq === _openEmailRequestSeq &&
+    mailboxContextIsCurrent()
+  );
   const aiReplyMode = mode === 'ai-reply-fast' ? 'fast' : '';
   const wantsAiReply = mode === 'ai-reply' || !!aiReplyMode;
   // Body pre-fill from the agent's open_email_reply tool call takes the
@@ -780,9 +795,10 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
     let data = preloadedData;
     if (!data) {
       const fullQS = mode === 'forward' ? '&full=1' : '';
-      const res = await fetch(`${API_BASE}/api/email/read/${em.uid}?folder=${encodeURIComponent(_currentFolder)}${_acct()}${fullQS}`);
+      const res = await fetch(`${API_BASE}/api/email/read/${em.uid}?folder=${encodeURIComponent(folderAtStart)}${accountQueryAtStart}&mark_seen=true${fullQS}`);
       data = await res.json();
     }
+    if (!isCurrentOpen()) return;
     if (data.error) {
       console.error('Failed to read email:', data.error);
       return;
@@ -808,7 +824,7 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
       message_id: _fallback(data.message_id, em.message_id),
     };
     if (wantsAiReply) {
-      const activeReplyAccount = data.account_id || em.account_id || window.__odysseusActiveEmailAccount || '';
+      const activeReplyAccount = data.account_id || em.account_id || accountAtStart;
       if (data.cached_ai_reply && !noteHint && !activeReplyAccount) {
         aiSuggestedBody = _cleanAiReplyText(data.cached_ai_reply);
       } else {
@@ -834,7 +850,7 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
               session_id: currentSessionId,
               message_id: data.message_id || '',
               uid: String(em.uid || ''),
-              folder: _currentFolder,
+              folder: folderAtStart,
               account_id: activeReplyAccount,
               fast: true,
               user_hint: (noteHint || '').trim() || undefined,
@@ -842,6 +858,7 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
           });
           const result = await res.json();
           if (draftToastTimer) clearTimeout(draftToastTimer);
+          if (!isCurrentOpen()) return;
           if (result.success && result.reply) {
             aiSuggestedBody = _cleanAiReplyText(result.reply);
           } else {
@@ -855,6 +872,7 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
           }
         } catch (e) {
           if (draftToastTimer) clearTimeout(draftToastTimer);
+          if (!isCurrentOpen()) return;
           console.error('AI reply generation failed:', e);
           import('./ui.js').then(m => m.showError && m.showError('AI reply failed: ' + (e.message || e))).catch(() => {});
           return;
@@ -862,8 +880,12 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
       }
     }
 
-    em.is_read = true;
-    if (itemEl) itemEl.classList.remove('email-unread');
+    if (!isCurrentOpen()) return;
+    // Only claim the message is read when the provider accepted the \Seen
+    // transition. A failed STORE still opens the message; it just stays unread.
+    const markedSeen = !data.mark_seen_failed;
+    em.is_read = markedSeen;
+    if (itemEl) itemEl.classList.toggle('email-unread', !markedSeen);
 
     // Addresses to exclude from Reply All. Prefer the full set of configured
     // accounts (so a multi-account user's other mailboxes are excluded too),
@@ -911,7 +933,7 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
     if (mode !== 'forward' && data.message_id) content += `\nIn-Reply-To: ${data.message_id}`;
     if (mode !== 'forward' && data.message_id) content += `\nReferences: ${data.references ? data.references + ' ' + data.message_id : data.message_id}`;
     content += `\nX-Source-UID: ${em.uid}`;
-    content += `\nX-Source-Folder: ${_currentFolder}`;
+    content += `\nX-Source-Folder: ${folderAtStart}`;
     if (data.attachments && data.attachments.length > 0) {
       const attStr = data.attachments.map(a => `${a.index}:${a.filename}:${a.size}`).join('|');
       content += `\nX-Attachments: ${attStr}`;
@@ -980,21 +1002,27 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
       // and block Send on long threads.
       const reuseExisting = mode !== 'forward' && !!aiSuggestedBody;
       const existingDocId = (reuseExisting && _docModule.findEmailDocId)
-        ? _docModule.findEmailDocId(em.uid, _currentFolder)
+        ? _docModule.findEmailDocId(em.uid, folderAtStart)
         : null;
       if (existingDocId) {
         if (!_docModule.isPanelOpen()) _docModule.openPanel();
         await new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r)));
+        if (!isCurrentOpen()) return;
         await _docModule.loadDocument(existingDocId);
+        if (!isCurrentOpen()) return;
         if (typeof _docModule.ensureEmailDraftEnvelope === 'function') {
           await _docModule.ensureEmailDraftEnvelope(existingDocId, content);
+          if (!isCurrentOpen()) return;
         }
         if (aiSuggestedBody && typeof _docModule.replaceEmailReplyBody === 'function') {
           await _docModule.replaceEmailReplyBody(existingDocId, aiSuggestedBody, { force: false });
+          if (!isCurrentOpen()) return;
         }
         _bringEmailReplyDraftToFrontOnMobile();
       } else {
+        if (!isCurrentOpen()) return;
         let activeSid = await _createEmailChat(data, { forceNew: true });
+        if (!isCurrentOpen()) return;
         if (!activeSid) {
           console.error('reply: could not obtain a session_id');
           import('./ui.js').then(m => m.showError && m.showError('Could not start a reply chat.')).catch(() => {});
@@ -1012,13 +1040,20 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
           }),
         });
         let docRes = await createReplyDoc(activeSid);
+        if (!isCurrentOpen()) return;
         if (docRes.status === 404) {
           console.warn('[reply-debug] draft session rejected; retrying in a fresh email chat', activeSid);
+          if (!isCurrentOpen()) return;
           activeSid = await _createEmailChat(data, { forceNew: true });
-          if (activeSid) docRes = await createReplyDoc(activeSid);
+          if (!isCurrentOpen()) return;
+          if (activeSid) {
+            docRes = await createReplyDoc(activeSid);
+            if (!isCurrentOpen()) return;
+          }
         }
         if (!docRes.ok) {
           const errText = await docRes.text();
+          if (!isCurrentOpen()) return;
           console.error('[reply-debug] POST /api/document failed', docRes.status, errText);
           // uiModule isn't statically imported here — use the dynamic
           // import pattern the rest of this file uses. (Previously this
@@ -1028,10 +1063,12 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
           return;
         }
         const doc = await docRes.json();
+        if (!isCurrentOpen()) return;
         if (doc.id) {
           const wasOpen = _docModule.isPanelOpen();
           if (!wasOpen) _docModule.openPanel();
           await new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r)));
+          if (!isCurrentOpen()) return;
           // Use the doc dict from the POST directly — avoids a 404 race
           // when the GET fires before the new row is visible to the read
           // connection (or when caching is interfering). loadDocument's
@@ -1040,12 +1077,14 @@ async function _openEmail(em, itemEl, preloadedData = null, mode = 'reply', note
             _docModule.injectFreshDoc(doc);
           } else {
             await _docModule.loadDocument(doc.id);
+            if (!isCurrentOpen()) return;
           }
           _bringEmailReplyDraftToFrontOnMobile();
         }
       }
     }
   } catch (e) {
+    if (!isCurrentOpen()) return;
     console.error('Failed to open email:', e);
     // Surface the failure so a silent throw in the reply flow doesn't
     // look like "nothing happened". Dynamic import — uiModule isn't a

static/js/emailLibrary.js

@@ -13,7 +13,7 @@ import { makeWindowDraggable } from './windowDrag.js';
 import {
   _esc, _escLinkify, _extractName, _parseTurnMeta,
   _formatBubbleDate, _formatRecipients, _senderColor, _initials,
-  _sanitizeHtml,
+  _sanitizeHtml, _renderEmailSummaryError,
   _TALON_WROTE, _TALON_FROM, _TALON_SENT, _TALON_SUBJ, _TALON_TO,
   _TALON_ORIG_RE, _SIG_BLOAT_MIN_CHARS,
 } from './emailLibrary/utils.js';
@@ -30,6 +30,10 @@ import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
 const API_BASE = window.location.origin;
 let _emailUnreadChipClickWired = false;
 let _libLoadSeq = 0;
+let _emailMailboxGeneration = 0;
+let _emailCardOpenSeq = 0;
+let _emailReadMutationSeq = 0;
+const _emailReadMutations = new Map();
 let _libFolderSeq = 0;
 let _libSearchSeq = 0;
 let _libSearchHadResults = false;
@@ -837,14 +841,41 @@ document.addEventListener('keydown', (e) => {
   e.stopImmediatePropagation?.();
 }, true);
 
-function _syncEmailReadState(uid, isRead = true) {
+function _emailReadContextKey(context) {
+  return [context.accountId, context.folder, context.uid].map(value => String(value || '')).join('\u0000');
+}
+
+function _emailReadContextIsCurrent(context) {
+  if (!context) return true;
+  return (
+    String(state._libAccountId || '') === context.accountId &&
+    String(state._libFolder || 'INBOX') === context.libraryFolder &&
+    _emailMailboxGeneration === context.mailboxGeneration
+  );
+}
+
+function _emailMatchesReadContext(email, context) {
+  if (String(email?.uid || '') !== context.uid) return false;
+  const accountId = String(email?.account_id || context.accountId);
+  const folder = String(email?.folder || context.folder);
+  return accountId === context.accountId && folder === context.folder;
+}
+
+function _syncEmailReadState(uid, isRead = true, context = null) {
   if (uid == null) return;
   const uidStr = String(uid);
   const read = !!isRead;
-  const match = (state._libEmails || []).find(x => String(x.uid) === uidStr);
+  if (context && (!_emailReadContextIsCurrent(context) || uidStr !== context.uid)) return;
+  const match = (state._libEmails || []).find(x => (
+    context ? _emailMatchesReadContext(x, context) : String(x.uid) === uidStr
+  ));
   if (match) match.is_read = read;
 
   document.querySelectorAll('.doclib-card[data-uid="' + CSS.escape(uidStr) + '"]').forEach(card => {
+    if (context && (
+      String(card.dataset.emailAccount || '') !== context.accountId ||
+      String(card.dataset.emailFolder || '') !== context.folder
+    )) return;
     card.classList.toggle('email-card-unread', !read);
     const titleRow = card.querySelector('.email-card-titlerow');
     if (read) {
@@ -1762,11 +1793,18 @@ function _rememberedEmailAccountId() {
 // results and __scheduled__ are deliberately not cached.
 const _libListCache = new Map();
 const _LIB_CACHE_MAX = 24;
+const _LIB_INITIAL_PAGE_SIZE = 100;
 const _LIB_SESSION_CACHE_PREFIX = 'odysseus.email.list.';
 const _LIB_SESSION_CACHE_TTL_MS = 10 * 60 * 1000;
 const _LIB_LAST_ACCOUNT_KEY = 'odysseus.email.lastAccountId';
-let _libPrewarmTimer = null;
+const _LIB_PREWARM_COOLDOWN_MS = 5 * 60 * 1000;
+let _libPrewarmDelayTimer = null;
+let _libPrewarmIdleHandle = null;
 let _libPrewarmPromise = null;
+let _libPrewarmResolve = null;
+let _libPrewarmAbortController = null;
+let _libPrewarmDetachPriorityListeners = null;
+let _libPrewarmGeneration = 0;
 let _libLastPrewarmAt = 0;
 let _libUnreadPrewarmKey = '';
 let _libUnreadPrewarmAt = 0;
@@ -1908,6 +1946,7 @@ function _resetEmailListForFreshLoad({ useCache = true } = {}) {
   _exitEmailReaderModeForList();
   _resetBulkSelectionForContextChange();
   state._libOffset = 0;
+  _emailMailboxGeneration += 1;
   _libLoadSeq += 1;
   const ck = _libCacheKey();
   const cached = useCache ? _libCacheGet(ck) : null;
@@ -2076,162 +2115,319 @@ function _isChatInteractionBusy() {
   }
 }
 
-function _loadEmailsWhenChatIdle({ delay = 50, retries = 180, options = {} } = {}) {
-  const run = () => {
-    if (!state._libOpen || !document.getElementById('email-lib-modal')) return;
-    if (_isChatInteractionBusy() && retries > 0) {
-      setTimeout(() => _loadEmailsWhenChatIdle({ delay: 1000, retries: retries - 1, options }), 1000);
+function _canRunEmailPrewarm() {
+  if (state._libOpen || state._libLoading || _libSearchInFlight) return false;
+  if (document.visibilityState && document.visibilityState !== 'visible') return false;
+  return !_isChatInteractionBusy();
+}
+
+function _isEmailPrewarmTemporarilyBlocked() {
+  if (state._libOpen || state._libLoading || _libSearchInFlight) return false;
+  if (document.visibilityState && document.visibilityState !== 'visible') return false;
+  return _isChatInteractionBusy();
+}
+
+function _isEmailPrewarmCurrent(generation, signal) {
+  return generation === _libPrewarmGeneration
+    && !signal?.aborted
+    && _canRunEmailPrewarm();
+}
+
+function _settleEmailPrewarm(generation, value = false) {
+  if (generation !== _libPrewarmGeneration) return;
+  const resolve = _libPrewarmResolve;
+  const detachPriorityListeners = _libPrewarmDetachPriorityListeners;
+  _libPrewarmDelayTimer = null;
+  _libPrewarmIdleHandle = null;
+  _libPrewarmPromise = null;
+  _libPrewarmResolve = null;
+  _libPrewarmAbortController = null;
+  _libPrewarmDetachPriorityListeners = null;
+  detachPriorityListeners?.();
+  resolve?.(value);
+}
+
+function _cancelEmailPrewarm() {
+  const resolve = _libPrewarmResolve;
+  const detachPriorityListeners = _libPrewarmDetachPriorityListeners;
+  _libPrewarmGeneration += 1;
+  if (_libPrewarmDelayTimer !== null) {
+    clearTimeout(_libPrewarmDelayTimer);
+  }
+  if (_libPrewarmIdleHandle !== null && typeof window.cancelIdleCallback === 'function') {
+    try { window.cancelIdleCallback(_libPrewarmIdleHandle); } catch (_) {}
+  }
+  try { _libPrewarmAbortController?.abort(); } catch (_) {}
+  _libPrewarmDelayTimer = null;
+  _libPrewarmIdleHandle = null;
+  _libPrewarmPromise = null;
+  _libPrewarmResolve = null;
+  _libPrewarmAbortController = null;
+  _libPrewarmDetachPriorityListeners = null;
+  detachPriorityListeners?.();
+  resolve?.(false);
+}
+
+function _scheduleEmailPrewarm(task, { delay = 0 } = {}) {
+  if (_libPrewarmPromise) return _libPrewarmPromise;
+  // Do not disguise a timer as idle work. Browsers without the genuine idle
+  // callback simply skip this optional optimization and load on demand.
+  if (typeof window.requestIdleCallback !== 'function') return Promise.resolve(false);
+
+  const generation = ++_libPrewarmGeneration;
+  _libPrewarmPromise = new Promise(resolve => { _libPrewarmResolve = resolve; });
+  const promise = _libPrewarmPromise;
+  let attemptPending = false;
+  let retryRequested = false;
+
+  function clearScheduledAttempt() {
+    if (_libPrewarmDelayTimer !== null) clearTimeout(_libPrewarmDelayTimer);
+    if (_libPrewarmIdleHandle !== null && typeof window.cancelIdleCallback === 'function') {
+      try { window.cancelIdleCallback(_libPrewarmIdleHandle); } catch (_) {}
+    }
+    _libPrewarmDelayTimer = null;
+    _libPrewarmIdleHandle = null;
+  }
+
+  function scheduleIdleRetry(delay = 500) {
+    if (generation !== _libPrewarmGeneration) return;
+    retryRequested = true;
+    if (attemptPending || _libPrewarmDelayTimer !== null || _libPrewarmIdleHandle !== null) return;
+    if (document.visibilityState && document.visibilityState !== 'visible') return;
+    _libPrewarmDelayTimer = setTimeout(requestIdle, Math.max(50, Number(delay) || 500));
+  }
+
+  function handlePriorityChange() {
+    if (generation !== _libPrewarmGeneration) return;
+    if (_canRunEmailPrewarm()) {
+      scheduleIdleRetry(50);
       return;
     }
-    _loadEmails(options);
+
+    const priorityBlocked = _isChatInteractionBusy()
+      || (document.visibilityState && document.visibilityState !== 'visible');
+    if (!priorityBlocked) return;
+
+    retryRequested = true;
+    clearScheduledAttempt();
+    const controller = _libPrewarmAbortController;
+    _libPrewarmAbortController = null;
+    try { controller?.abort(); } catch (_) {}
+    // A hidden page waits for visibilitychange. Chat priority also retains the
+    // timer fallback for busy-until windows whose final transition has no event.
+    if (!document.visibilityState || document.visibilityState === 'visible') {
+      scheduleIdleRetry();
+    }
+  }
+
+  window.addEventListener('odysseus:chat-busy-change', handlePriorityChange);
+  document.addEventListener('visibilitychange', handlePriorityChange);
+  _libPrewarmDetachPriorityListeners = () => {
+    window.removeEventListener('odysseus:chat-busy-change', handlePriorityChange);
+    document.removeEventListener('visibilitychange', handlePriorityChange);
   };
-  setTimeout(run, Math.max(0, Number(delay) || 0));
+
+  function requestIdle() {
+    if (generation !== _libPrewarmGeneration) return;
+    _libPrewarmDelayTimer = null;
+    try {
+      _libPrewarmIdleHandle = window.requestIdleCallback((deadline) => {
+        if (generation !== _libPrewarmGeneration) return;
+        _libPrewarmIdleHandle = null;
+        const hasIdleBudget = Boolean(
+          deadline
+          && !deadline.didTimeout
+          && typeof deadline.timeRemaining === 'function'
+          && deadline.timeRemaining() > 0
+        );
+        if (!_canRunEmailPrewarm()) {
+          if (_isEmailPrewarmTemporarilyBlocked()) {
+            scheduleIdleRetry();
+          } else {
+            _settleEmailPrewarm(generation, false);
+          }
+          return;
+        }
+        if (!hasIdleBudget) {
+          scheduleIdleRetry();
+          return;
+        }
+        if (generation !== _libPrewarmGeneration) {
+          _settleEmailPrewarm(generation, false);
+          return;
+        }
+        const controller = new AbortController();
+        _libPrewarmAbortController = controller;
+        attemptPending = true;
+        retryRequested = false;
+        Promise.resolve()
+          .then(() => task({ signal: controller.signal, generation }))
+          .then(value => {
+            if (controller !== _libPrewarmAbortController || controller.signal.aborted) return;
+            _settleEmailPrewarm(generation, Boolean(value));
+          })
+          .catch(() => {
+            if (controller !== _libPrewarmAbortController || controller.signal.aborted) return;
+            _settleEmailPrewarm(generation, false);
+          })
+          .finally(() => {
+            attemptPending = false;
+            if (generation !== _libPrewarmGeneration) return;
+            if (retryRequested) scheduleIdleRetry();
+          });
+      });
+    } catch (_) {
+      _settleEmailPrewarm(generation, false);
+    }
+  }
+
+  const wait = Math.max(0, Number(delay) || 0);
+  if (wait > 0) _libPrewarmDelayTimer = setTimeout(requestIdle, wait);
+  else requestIdle();
+  return promise;
 }
 
 export function prewarmEmailLibrary({ delay = 2500 } = {}) {
-  if (_libPrewarmTimer || _libPrewarmPromise) return;
+  if (_libPrewarmPromise) return _libPrewarmPromise;
   const elapsed = Date.now() - _libLastPrewarmAt;
-  if (elapsed >= 0 && elapsed < 5 * 60 * 1000) return;
-  _libPrewarmTimer = setTimeout(() => {
-    _libPrewarmTimer = null;
-    _libPrewarmPromise = _prewarmEmailViews()
-      .catch(() => {})
-      .finally(() => { _libPrewarmPromise = null; });
-  }, Math.max(0, Number(delay) || 0));
+  if (elapsed >= 0 && elapsed < _LIB_PREWARM_COOLDOWN_MS) return Promise.resolve(false);
+  return _scheduleEmailPrewarm(_prewarmEmailViews, { delay });
 }
 
-async function _ensureEmailAccountsForPrewarm() {
+function _chooseEmailPrewarmAccountId(accounts) {
+  const enabled = Array.isArray(accounts) ? accounts.filter(a => a && a.enabled !== false) : [];
+  const remembered = _rememberedEmailAccountId();
+  const current = String(state._libAccountId || '').trim();
+  const chosen = enabled.find(a => String(a.id || '') === remembered)
+    || enabled.find(a => String(a.id || '') === current)
+    || enabled.find(a => a.is_default)
+    || enabled[0]
+    || null;
+  return String(chosen?.id || '').trim();
+}
+
+async function _ensureEmailAccountsForPrewarm({ signal, generation } = {}) {
+  if (!_isEmailPrewarmCurrent(generation, signal)) return null;
   const accountsFresh = _libAccountsLoadedAt && (Date.now() - _libAccountsLoadedAt) < _LIB_ACCOUNTS_TTL_MS;
-  if (Array.isArray(state._libAccounts) && state._libAccounts.length && accountsFresh) {
-    if (!state._libAccountId) {
-      const def = state._libAccounts.find(a => a.is_default) || state._libAccounts[0];
-      state._libAccountId = def?.id || null;
-      _publishActiveAccount();
-    }
-    return;
-  }
-  try {
-    const accountsRes = await fetch(`${API_BASE}/api/email/accounts`, { credentials: 'same-origin' });
-    if (!accountsRes.ok) return;
-    const accountsData = await accountsRes.json().catch(() => ({}));
-    if (Array.isArray(accountsData.accounts)) {
-      state._libAccounts = accountsData.accounts;
-      _libAccountsLoadedAt = Date.now();
-      if (!state._libAccountId && state._libAccounts.length) {
-        const def = state._libAccounts.find(a => a.is_default) || state._libAccounts[0];
-        state._libAccountId = def?.id || null;
-        _publishActiveAccount();
+  if (!(Array.isArray(state._libAccounts) && state._libAccounts.length && accountsFresh)) {
+    try {
+      const accountsRes = await fetch(`${API_BASE}/api/email/accounts`, {
+        credentials: 'same-origin',
+        signal,
+      });
+      if (!_isEmailPrewarmCurrent(generation, signal)) return null;
+      if (accountsRes.ok) {
+        const accountsData = await accountsRes.json().catch(() => ({}));
+        if (!_isEmailPrewarmCurrent(generation, signal)) return null;
+        if (Array.isArray(accountsData.accounts)) {
+          state._libAccounts = accountsData.accounts;
+          _libAccountsLoadedAt = Date.now();
+        }
       }
+    } catch (err) {
+      if (err?.name === 'AbortError') return null;
     }
-  } catch (_) {}
+  }
+
+  const accountId = _chooseEmailPrewarmAccountId(state._libAccounts);
+  if (!_isEmailPrewarmCurrent(generation, signal)) return null;
+  if (!accountId) return null;
+  if (accountId && state._libAccountId !== accountId) {
+    state._libAccountId = accountId;
+    _publishActiveAccount();
+  }
+  return accountId;
 }
 
-export async function prewarmUnreadEmails({ limit = 8, maxUid = 0 } = {}) {
-  if (state._libOpen) return;
-  await _ensureEmailAccountsForPrewarm();
-  if (state._libOpen) return;
-  const accountId = state._libAccountId || '';
+export function prewarmUnreadEmails({ limit = 8, maxUid = 0 } = {}) {
+  return _scheduleEmailPrewarm(
+    context => _prewarmUnreadEmailsNow({ limit, maxUid }, context),
+    { delay: 0 }
+  );
+}
+
+async function _prewarmUnreadEmailsNow({ limit = 8, maxUid = 0 } = {}, { signal, generation } = {}) {
+  if (!_isEmailPrewarmCurrent(generation, signal)) return false;
+  const accountId = await _ensureEmailAccountsForPrewarm({ signal, generation });
+  if (accountId === null || !_isEmailPrewarmCurrent(generation, signal)) return false;
   const n = Math.max(1, Math.min(20, Number(limit) || 8));
   const key = `${accountId}|${maxUid || 0}|${n}`;
-  if (_libUnreadPrewarmKey === key && (Date.now() - _libUnreadPrewarmAt) < 60 * 1000) return;
-  _libUnreadPrewarmKey = key;
-  _libUnreadPrewarmAt = Date.now();
+  if (_libUnreadPrewarmKey === key && (Date.now() - _libUnreadPrewarmAt) < 60 * 1000) return true;
   try {
     const folder = 'INBOX';
-	    const res = await fetch(emailApiUrl('/api/email/list', {
-	      folder,
-	      limit: n,
-	      offset: 0,
-	      filter: 'unread',
-	      account_id: accountId || undefined,
-	    }), { credentials: 'same-origin' });
-	    if (state._libOpen) return;
-	    if (!res.ok) return;
+    const res = await fetch(emailApiUrl('/api/email/list', {
+      folder,
+      limit: n,
+      offset: 0,
+      filter: 'unread',
+      account_id: accountId || undefined,
+    }), {
+      credentials: 'same-origin',
+      signal,
+    });
+    if (!_isEmailPrewarmCurrent(generation, signal) || !res.ok) return false;
     const data = await res.json().catch(() => null);
-    if (!data || data.error || !Array.isArray(data.emails) || !data.emails.length) return;
+    if (!_isEmailPrewarmCurrent(generation, signal)) return false;
+    if (!data || data.error || !Array.isArray(data.emails) || !data.emails.length) return false;
     const sync = data.sync || {};
     _libCachePut(_libCacheKeyFor(accountId, folder, 'unread', false), {
       emails: data.emails,
       total: data.total || data.emails.length,
       sync,
     });
-  } catch (_) {}
+    _libUnreadPrewarmKey = key;
+    _libUnreadPrewarmAt = Date.now();
+    return true;
+  } catch (_) {
+    return false;
+  }
 }
 
-function _sleep(ms) {
-  return new Promise(resolve => setTimeout(resolve, ms));
-}
-
-async function _prewarmEmailViews() {
-  if (state._libOpen) return;
-  _libLastPrewarmAt = Date.now();
+async function _prewarmEmailViews({ signal, generation } = {}) {
+  if (!_isEmailPrewarmCurrent(generation, signal)) return false;
   _setEmailSyncStatus({ warming: true });
   const folder = 'INBOX';
   const filter = 'all';
-
-  // The accounts request is cheap and warms the account strip for first open.
-  // Then folder/list requests warm both the client cache and the backend
-  // IMAP/read caches. Failure stays silent: no configured mail should not nag.
   try {
-    const accountsRes = await fetch(`${API_BASE}/api/email/accounts`, { credentials: 'same-origin' });
-    if (accountsRes.ok) {
-      const accountsData = await accountsRes.json().catch(() => ({}));
-      if (Array.isArray(accountsData.accounts)) {
-        state._libAccounts = accountsData.accounts;
-        _libAccountsLoadedAt = Date.now();
-      }
+    const accountId = await _ensureEmailAccountsForPrewarm({ signal, generation });
+    if (accountId === null || !_isEmailPrewarmCurrent(generation, signal)) return false;
+    const ck = _libCacheKeyFor(accountId, folder, filter, false);
+    if (_libCacheGet(ck)) {
+      _libLastPrewarmAt = Date.now();
+      return true;
     }
-  } catch (_) {}
 
-  const accounts = Array.isArray(state._libAccounts) ? state._libAccounts.filter(a => a && a.enabled !== false) : [];
-  const preferred = state._libAccountId
-    || (accounts.find(a => a.is_default)?.id)
-    || (accounts[0]?.id)
-    || '';
-  if (!state._libAccountId && preferred) {
-    state._libAccountId = preferred;
-    _publishActiveAccount();
-  }
-  const orderedAccountIds = [
-    preferred,
-    ...accounts.map(a => a.id).filter(id => id && id !== preferred),
-  ].filter((id, idx, arr) => arr.indexOf(id) === idx);
-  if (!orderedAccountIds.length) orderedAccountIds.push('');
-
-  try {
-    for (const accountId of orderedAccountIds.slice(0, 4)) {
-      if (state._libOpen) return;
-      const ck = _libCacheKeyFor(accountId, folder, filter, false);
-      if (_libCacheGet(ck)) continue;
-      await fetch(emailApiUrl('/api/email/folders', { account_id: accountId || undefined }), { credentials: 'same-origin' }).catch(() => null);
-      await fetch(emailApiUrl('/api/email/unread-state', { folder, account_id: accountId || undefined }), { credentials: 'same-origin' }).catch(() => null);
-      const res = await fetch(emailApiUrl('/api/email/list', {
-        folder,
-        limit: 100,
-        offset: 0,
-        filter,
-        account_id: accountId || undefined,
-      }), {
-        credentials: 'same-origin',
-      });
-      if (res.ok) {
-        const data = await res.json().catch(() => null);
-        if (data && !data.error) {
-          const sync = data.sync || {};
-          _libCachePut(ck, {
-            emails: data.emails || [],
-            total: data.total || 0,
-            sync,
-          });
-          _setEmailSyncStatus({
-            updatedAt: sync.updated_at || new Date().toISOString(),
-            source: sync.source || '',
-            warming: true,
-          });
-        }
-      }
-      await _sleep(900);
-    }
+    // One optional first-page request only. Folder metadata, unread state, and
+    // other accounts remain demand-driven so startup cannot fan out into IMAP.
+    const res = await fetch(emailApiUrl('/api/email/list', {
+      folder,
+      limit: _LIB_INITIAL_PAGE_SIZE,
+      offset: 0,
+      filter,
+      account_id: accountId || undefined,
+    }), {
+      credentials: 'same-origin',
+      signal,
+    });
+    if (!_isEmailPrewarmCurrent(generation, signal) || !res.ok) return false;
+    const data = await res.json().catch(() => null);
+    if (!_isEmailPrewarmCurrent(generation, signal)) return false;
+    if (!data || data.error || !Array.isArray(data.emails)) return false;
+    const sync = data.sync || {};
+    _libCachePut(ck, {
+      emails: data.emails,
+      total: data.total || 0,
+      sync,
+    });
+    _libLastPrewarmAt = Date.now();
+    _setEmailSyncStatus({
+      updatedAt: sync.updated_at || new Date().toISOString(),
+      source: sync.source || '',
+      warming: true,
+    });
+    return true;
+  } catch (_) {
+    return false;
   } finally {
     _setEmailSyncStatus({ warming: false });
   }
@@ -2286,16 +2482,34 @@ function _publishActiveAccount() {
 
 export function initEmailLibrary(config) {
   state._docModule = config.documentModule;
-  state._onEmailClick = config.onEmailClick;
+  const onEmailClick = config.onEmailClick;
+  state._onEmailClick = typeof onEmailClick === 'function' ? (options = {}) => {
+    const accountId = String(state._libAccountId || '');
+    const libraryFolder = String(state._libFolder || 'INBOX');
+    const messageFolder = String(options.email?.folder || libraryFolder);
+    const mailboxGeneration = _emailMailboxGeneration;
+    const mailboxContext = Object.freeze({
+      accountId,
+      libraryFolder,
+      messageFolder,
+      mailboxGeneration,
+      isCurrent: () => (
+        String(state._libAccountId || '') === accountId &&
+        String(state._libFolder || 'INBOX') === libraryFolder &&
+        _emailMailboxGeneration === mailboxGeneration
+      ),
+    });
+    return onEmailClick({ ...options, mailboxContext });
+  } : null;
 }
 
 export function isOpen() { return state._libOpen; }
 
 export function openEmailLibrary(opts = {}) {
-  if (_libPrewarmTimer) {
-    clearTimeout(_libPrewarmTimer);
-    _libPrewarmTimer = null;
-  }
+  // Foreground email always wins: cancel a delayed/idle callback and abort the
+  // one optional request if it has already started. Generation checks make a
+  // non-abortable response harmless if it races this transition.
+  _cancelEmailPrewarm();
   // Force-clean any stale state from previous attempts
   const existing = document.getElementById('email-lib-modal');
   if (existing) existing.remove();
@@ -2303,6 +2517,7 @@ export function openEmailLibrary(opts = {}) {
     document.removeEventListener('keydown', state._libEscHandler, true);
     state._libEscHandler = null;
   }
+  _emailMailboxGeneration += 1;
   state._libOpen = true;
   // On mobile the sidebar overlays content — close it so the email view isn't
   // opened behind it (same pattern as session-switch/delete).
@@ -2926,7 +3141,7 @@ export function openEmailLibrary(opts = {}) {
   }
   const fastAccountAtOpen = state._libAccountId || '';
   if (fastAccountAtOpen) {
-    _loadEmailsWhenChatIdle({ delay: 0 });
+    _loadEmails({ useCache: true });
   }
   // If we already know the previous/default account, paint that inbox first
   // from the durable index and validate accounts in parallel. Cold refreshes
@@ -2936,7 +3151,7 @@ export function openEmailLibrary(opts = {}) {
     _loadFolders();
     _loadEmailReminderBellVisibility();
     if (!fastAccountAtOpen || fastAccountAtOpen !== (state._libAccountId || '')) {
-      _loadEmailsWhenChatIdle();
+      _loadEmails({ useCache: true });
     }
   })();
 }
@@ -3121,6 +3336,7 @@ export async function openEmailLibrarySettings() {
 }
 
 export function closeEmailLibrary() {
+  _cancelEmailPrewarm();
   const modal = document.getElementById('email-lib-modal');
   if (modal) modal.remove();
   if (_libSyncTicker) {
@@ -4554,7 +4770,7 @@ async function _loadEmails({ force = false, useCache = true } = {}) {
         const ctrl = new AbortController();
         const timer = setTimeout(() => ctrl.abort(), 450);
         try {
-          const fastRes = await fetch(`${API_BASE}/api/email/list?folder=${encodeURIComponent(folderAtStart)}${accountQS}&limit=100&offset=${offsetAtStart}&filter=${filterAtStart}${attQS}&cached_only=1`, {
+          const fastRes = await fetch(`${API_BASE}/api/email/list?folder=${encodeURIComponent(folderAtStart)}${accountQS}&limit=${_LIB_INITIAL_PAGE_SIZE}&offset=${offsetAtStart}&filter=${filterAtStart}${attQS}&cached_only=1`, {
             signal: ctrl.signal,
           });
           const fastData = await fastRes.json().catch(() => null);
@@ -4581,7 +4797,7 @@ async function _loadEmails({ force = false, useCache = true } = {}) {
       // opens omit it so rapid close/reopen returns instantly; the
       // Refresh button passes `force: true` to add it back.
       const buster = force ? `&_=${Date.now()}` : '';
-      const res = await fetch(`${API_BASE}/api/email/list?folder=${encodeURIComponent(folderAtStart)}${accountQS}&limit=100&offset=${offsetAtStart}&filter=${filterAtStart}${attQS}${buster}`);
+      const res = await fetch(`${API_BASE}/api/email/list?folder=${encodeURIComponent(folderAtStart)}${accountQS}&limit=${_LIB_INITIAL_PAGE_SIZE}&offset=${offsetAtStart}&filter=${filterAtStart}${attQS}${buster}`);
       const data = await res.json();
       if (seq !== _libLoadSeq || accountAtStart !== (state._libAccountId || '')) return;
       if (data.error) throw new Error(data.error);
@@ -4836,6 +5052,8 @@ function _createCard(em) {
   else if (!em.is_read) cls += ' email-card-unread';
   card.className = cls;
   card.dataset.uid = String(em.uid);
+  card.dataset.emailAccount = String(em.account_id || state._libAccountId || '');
+  card.dataset.emailFolder = String(em.folder || state._libFolder || 'INBOX');
   if (state._selectMode && state._selectedUids.has(em.uid)) card.classList.add('selected');
 
   // Checkbox in select mode
@@ -5162,6 +5380,25 @@ async function _toggleCardPreview(card, em) {
   // currently-selected folder for normal inbox cards.
   const folderAtStart = (em && em.folder) || libraryFolderAtStart;
   const uidAtStart = String(em?.uid || card?.dataset?.uid || '');
+  const wasReadAtStart = !!em?.is_read;
+  const openGeneration = ++_emailCardOpenSeq;
+  const readContext = Object.freeze({
+    accountId: String(accountAtStart),
+    libraryFolder: String(libraryFolderAtStart),
+    folder: String(folderAtStart),
+    uid: uidAtStart,
+    mailboxGeneration: _emailMailboxGeneration,
+  });
+  const readContextKey = _emailReadContextKey(readContext);
+  const isCurrentOpen = () => (
+    openGeneration === _emailCardOpenSeq &&
+    _emailReadContextIsCurrent(readContext) &&
+    accountAtStart === (state._libAccountId || '') &&
+    libraryFolderAtStart === (state._libFolder || 'INBOX') &&
+    uidAtStart === String(card?.dataset?.uid || '') &&
+    card.isConnected &&
+    card.classList.contains('email-card-expanded')
+  );
   const grid = card.closest('.doclib-grid');
   const gridRect = grid?.getBoundingClientRect?.();
   const modal = document.getElementById('email-lib-modal');
@@ -5186,6 +5423,30 @@ async function _toggleCardPreview(card, em) {
     return;
   }
 
+  // Every authoritative open supersedes any older optimistic mutation for the
+  // same immutable mailbox identity. Carry the original unread state forward
+  // so a close/reopen followed by failure still rolls back exactly once, while
+  // a late failure from the superseded request cannot undo a newer success.
+  const previousMutation = _emailReadMutations.get(readContextKey);
+  const readMutation = {
+    generation: ++_emailReadMutationSeq,
+    rollbackUnread: !wasReadAtStart || !!previousMutation?.rollbackUnread,
+  };
+  _emailReadMutations.set(readContextKey, readMutation);
+  const restoreUnreadState = () => {
+    if (_emailReadMutations.get(readContextKey)?.generation !== readMutation.generation) return;
+    _emailReadMutations.delete(readContextKey);
+    if (readMutation.rollbackUnread) _syncEmailReadState(uidAtStart, false, readContext);
+  };
+  const commitReadState = () => {
+    // A successful STORE/mark_seen is authoritative for this immutable
+    // mailbox identity even when a newer open is still pending. Retire that
+    // newer rollback token too, otherwise its later failure could restore an
+    // unread state that no longer exists at the provider.
+    _emailReadMutations.delete(readContextKey);
+    _syncEmailReadState(uidAtStart, true, readContext);
+  };
+
   // Collapse any other expanded card
   if (grid) {
     grid.querySelectorAll('.email-card-expanded').forEach(c => {
@@ -5207,10 +5468,10 @@ async function _toggleCardPreview(card, em) {
   requestAnimationFrame(() => {
     try { card.scrollIntoView({ behavior: 'smooth', block: 'start' }); } catch (_) {}
   });
-  if (!em.is_read) {
-    _syncEmailReadState(em.uid, true);
-    fetch(`${API_BASE}/api/email/mark-read/${em.uid}?folder=${encodeURIComponent(folderAtStart)}${_acct()}`, { method: 'POST' })
-      .catch(err => console.error('Failed to mark email read:', err));
+  if (!wasReadAtStart) {
+    // Keep the current optimistic visual update, but let the read request below
+    // own the provider-side \Seen transition. A failure restores unread state.
+    _syncEmailReadState(uidAtStart, true, readContext);
   }
   // Class hook on the modal so the header-hide / padding rules work on
   // browsers without :has() support (Firefox mobile) — the :has() versions
@@ -5239,25 +5500,28 @@ async function _toggleCardPreview(card, em) {
     } catch (_) {}
   };
 
+  let authoritativeReadSucceeded = false;
   try {
-    const res = await fetch(`${API_BASE}/api/email/read/${em.uid}?folder=${encodeURIComponent(folderAtStart)}${_acct()}`);
+    const accountQueryAtStart = accountAtStart ? `&account_id=${encodeURIComponent(accountAtStart)}` : '';
+    const res = await fetch(`${API_BASE}/api/email/read/${encodeURIComponent(uidAtStart)}?folder=${encodeURIComponent(folderAtStart)}${accountQueryAtStart}&mark_seen=true`);
     if (!res.ok) throw new Error(`HTTP ${res.status}`);
     const data = await res.json();
-    if (
-      accountAtStart !== (state._libAccountId || '') ||
-      libraryFolderAtStart !== (state._libFolder || 'INBOX') ||
-      uidAtStart !== String(card?.dataset?.uid || '') ||
-      !card.isConnected ||
-      !card.classList.contains('email-card-expanded')
-    ) {
-      return;
-    }
     if (data.error) {
-      showFailedReader(`Failed to load email: ${data.error}`);
+      restoreUnreadState();
+      if (isCurrentOpen()) showFailedReader(`Failed to load email: ${data.error}`);
       return;
     }
-    // Mark as read locally
-    _syncEmailReadState(em.uid, true);
+    if (data.mark_seen_failed) {
+      // The body is authoritative even when the provider refused the \Seen
+      // transition. Render the message and roll the unread marker back so the
+      // list keeps telling the truth, rather than refusing to open a message
+      // we successfully read.
+      restoreUnreadState();
+    } else {
+      authoritativeReadSucceeded = true;
+      commitReadState();
+    }
+    if (!isCurrentOpen()) return;
     _stampReaderContext(reader, { ...em, ...data }, state._libFolder, state._libAccountId);
 
     // Build the attachments wrap using the shared helper so the signature-
@@ -5439,7 +5703,10 @@ async function _toggleCardPreview(card, em) {
     // Always stop bubbling so the card's click doesn't fire while reading.
     reader.addEventListener('click', (ev) => { ev.stopPropagation(); });
   } catch (e) {
-    showFailedReader(e?.message ? `Failed to load email: ${e.message}` : 'Failed to load email');
+    if (!authoritativeReadSucceeded) restoreUnreadState();
+    if (isCurrentOpen()) {
+      showFailedReader(e?.message ? `Failed to load email: ${e.message}` : 'Failed to load email');
+    }
   }
 }
 
@@ -7259,12 +7526,11 @@ async function _generateSummary(reader, data, btn) {
         if (label) label.textContent = 'Summary';
       }
     } else {
-      content.innerHTML = `<span style="color:var(--red)">${_esc(result.error || 'Failed to summarize')}</span>`;
-      panel.remove();
+      _renderEmailSummaryError(content, result);
     }
   } catch (e) {
     sp.destroy();
-    panel.remove();
+    _renderEmailSummaryError(content, null);
     if (uiModule) uiModule.showError?.('Failed to summarize');
   } finally {
     if (btn) btn.disabled = false;

static/js/emailLibrary/utils.js

@@ -30,6 +30,25 @@ export function _esc(text) {
   return div.innerHTML;
 }
 
+const _EMAIL_SUMMARY_ERROR_MESSAGES = Object.freeze({
+  email_summary_missing_body: 'No email body to summarize',
+  email_summary_not_configured: 'No model configured for email summaries',
+  email_summary_empty: 'The model returned an empty summary',
+  email_summary_unavailable: 'Failed to summarize',
+});
+
+export function _emailSummaryErrorMessage(result) {
+  const code = String(result?.error_code || '');
+  return _EMAIL_SUMMARY_ERROR_MESSAGES[code] || 'Failed to summarize';
+}
+
+export function _renderEmailSummaryError(container, result) {
+  const message = container.ownerDocument.createElement('span');
+  message.style.color = 'var(--red)';
+  message.textContent = _emailSummaryErrorMessage(result);
+  container.replaceChildren(message);
+}
+
 function _attrEsc(text) {
   return String(text ?? '')
     .replace(/"/g, '"')

static/js/liveThinkingThrottle.js

@@ -0,0 +1,206 @@
+// liveThinkingThrottle.js
+//
+// Pure trailing-edge coalescer for the live "thinking" block in chat.js.
+//
+// A reasoning stream delivers deltas far faster than a human can read them, and
+// the only thing that matters on screen is the LATEST cumulative text. Committing
+// every delta to the DOM makes the work grow with the length of the stream. This
+// throttle collapses a burst of updates into one commit per `delay` ms, always
+// carrying the most recent value.
+//
+// Timers are injected so the behaviour is testable without a browser or a clock:
+//
+//     const throttle = createLiveThinkingThrottle(commit, { prepare, schedule, cancel });
+//
+// Lifecycle contract, which the terminal paths in chat.js depend on:
+//
+//   update(value)  queue `value`; schedule a commit if one is not already pending
+//   flush()        commit any pending value NOW and drop the timer; returns whether
+//                  a commit happened, so a clean flush cannot duplicate a commit
+//   cancel()       drop the timer AND the pending value — nothing lands later
+//
+// `cancel()` is what stops a finished (or backgrounded) stream from mutating a
+// view the user has since navigated away to.
+
+export function stripLiveThinkingTags(text) {
+  return String(text ?? '').replace(
+    /<\/?(?:think(?:ing)?|thought)(?:\s+[^>]*)?>/gi,
+    '',
+  );
+}
+
+const THINKING_BOUNDARY_RE = /<\/?(?:(?:mm:)?think(?:ing)?|thought)(?:\s+[^>]*)?>|<\|channel>(?:thought|response)|<channel\|>/gi;
+const REPLY_PREFIX_SOURCE = "(?:Hey|Hi |Hi!|Hello|Sure|Yes|No |No,|Yo|OK|Here|Absolutely|Of course|Great|Alright|Thanks|Welcome|Good |I'm happy|I'd be)";
+const REPLY_LINE_RE = new RegExp('(?:^|\\n)\\s*' + REPLY_PREFIX_SOURCE, 'gi');
+const REPLY_INLINE_RE = new RegExp('[.!?]\\s*' + REPLY_PREFIX_SOURCE, 'gi');
+const REASONING_PREFIX_CANDIDATES = [
+  'thinking:', 'thinking process:', 'the user ', 'user wants', 'we need ',
+  'i need ', 'i should ', 'i will ', "i'll ", 'i am going ', 'let me think',
+  'let me look', 'let me see', 'let me check', 'let me read', 'let me review',
+  'let me analyze', 'let me parse', 'let me figure', 'let me draft', 'let me write',
+  'they are ', 'the question ', 'i can ',
+];
+
+const DISPLAY_FILTER_BOUNDARY_RE = /\[\/?TOOL_CALL\]|```(?:create_document|documen(?:t)?)(?:\s|$)|```[\w-]+[ \t]*[\[{]|<(?:[\w]+:)?(?:tool_call|function_call)>|<invoke\b|<\s*\/?\s*[｜|]+\s*DSML\s*[｜|]+|(?:\[\s*)?\{\s*"function"\s*:|<\/?\|(?:assistant|assistan|user|system|tool|end)\|?>|(?:^|[\r\n])\s*(?:stdout|stderr|exit_code):/i;
+
+function hasFreshMatch(text, regex, cursor, minStart = 0) {
+  regex.lastIndex = 0;
+  for (const match of text.matchAll(regex)) {
+    const end = match.index + match[0].length;
+    if (end > cursor && match.index >= minStart) return true;
+  }
+  return false;
+}
+
+// Incrementally decides when chat.js needs its compatibility-heavy cumulative
+// thinking analysis. The gate inspects only a short overlap plus the new text;
+// ordinary answer/reasoning deltas therefore stay O(delta) while split tags,
+// namespaced tags, non-tag reply boundaries, and false-close grace deadlines
+// still request the canonical full analysis.
+export function createThinkingAnalysisGate({
+  startsWithReasoningPrefix = () => false,
+  now = () => Date.now(),
+  overlap = 512,
+} = {}) {
+  let cursor = 0;
+  let prefixSettled = false;
+  let prefixProbe = '';
+
+  return {
+    shouldAnalyze(text, {
+      isThinking = false,
+      nonTagThinking = false,
+      recheckAt = 0,
+    } = {}) {
+      const fullText = String(text ?? '');
+      if (fullText.length < cursor) {
+        cursor = 0;
+        prefixSettled = false;
+        prefixProbe = '';
+      }
+      const previousCursor = cursor;
+      if (!prefixSettled && prefixProbe.length < overlap) {
+        // Build the initial probe from deltas so arbitrary leading whitespace
+        // cannot strand the gate in its undecided state. The retained state is
+        // bounded even if a provider emits a very large whitespace prefix.
+        prefixProbe = (prefixProbe + fullText.slice(previousCursor))
+          .trimStart()
+          .slice(0, overlap);
+      }
+      const scanStart = Math.max(0, previousCursor - overlap);
+      const freshText = fullText.slice(scanStart);
+      const relativeCursor = previousCursor - scanStart;
+      const hasBoundary = hasFreshMatch(freshText, THINKING_BOUNDARY_RE, relativeCursor);
+      const hasReplyBoundary = nonTagThinking && (
+        hasFreshMatch(freshText, REPLY_LINE_RE, relativeCursor)
+        || hasFreshMatch(freshText, REPLY_INLINE_RE, relativeCursor, Math.max(0, 20 - scanStart))
+      );
+      cursor = fullText.length;
+
+      if (hasBoundary || hasReplyBoundary) return true;
+      if (isThinking) return recheckAt > 0 && now() >= recheckAt;
+      if (prefixSettled) return false;
+
+      if (!prefixProbe) return false;
+      if (startsWithReasoningPrefix(prefixProbe)) {
+        prefixSettled = true;
+        return true;
+      }
+      const lowerProbe = prefixProbe.toLowerCase();
+      if (REASONING_PREFIX_CANDIDATES.some((candidate) => candidate.startsWith(lowerProbe))) {
+        return false;
+      }
+      prefixSettled = true;
+      return false;
+    },
+    reset() {
+      cursor = 0;
+      prefixSettled = false;
+      prefixProbe = '';
+    },
+  };
+}
+
+// Keep the common prose path append-only. At the first structured/tool
+// boundary, filter only the preceding visible prefix and hide the structured
+// tail until the authoritative terminal render.
+export function createIncrementalDisplayProjector(filter, { overlap = 512 } = {}) {
+  let projected = '';
+  let boundaryTail = '';
+  let rawLength = 0;
+  let structuredTailHidden = false;
+
+  return {
+    append(delta, fullText) {
+      const chunk = String(delta ?? '');
+      const raw = String(fullText ?? '');
+      if (raw.length < rawLength) this.reset();
+      const boundaryProbe = boundaryTail + chunk;
+      const boundaryMatch = !structuredTailHidden
+        ? DISPLAY_FILTER_BOUNDARY_RE.exec(boundaryProbe)
+        : null;
+      if (boundaryMatch) {
+        // Filter the visible prefix, not the incomplete marker itself: several
+        // compatibility regexes intentionally match only completed blocks.
+        const boundaryStart = Math.max(0, raw.length - boundaryProbe.length + boundaryMatch.index);
+        structuredTailHidden = true;
+        projected = String(filter(raw.slice(0, boundaryStart)) ?? '');
+      } else if (!structuredTailHidden) {
+        projected += chunk;
+      }
+      boundaryTail = (boundaryTail + chunk).slice(-overlap);
+      rawLength = raw.length;
+      return projected;
+    },
+    current() {
+      return projected;
+    },
+    reset() {
+      projected = '';
+      boundaryTail = '';
+      rawLength = 0;
+      structuredTailHidden = false;
+    },
+  };
+}
+
+export function createLiveThinkingThrottle(commit, {
+  delay = 100,
+  prepare = (value) => String(value ?? ''),
+  schedule = (callback, ms) => setTimeout(callback, ms),
+  cancel = (timer) => clearTimeout(timer),
+} = {}) {
+  let timer = null;
+  let latest = null;
+  let dirty = false;
+
+  const commitLatest = () => {
+    timer = null;
+    if (!dirty) return false;
+    dirty = false;
+    commit(prepare(latest));
+    return true;
+  };
+
+  return {
+    update(value) {
+      latest = value;
+      dirty = true;
+      if (timer === null) timer = schedule(commitLatest, delay);
+    },
+    flush() {
+      if (timer !== null) {
+        cancel(timer);
+        timer = null;
+      }
+      return commitLatest();
+    },
+    cancel() {
+      if (timer !== null) cancel(timer);
+      timer = null;
+      dirty = false;
+    },
+  };
+}
+
+export default createLiveThinkingThrottle;

static/js/markdown.js

@@ -758,30 +758,36 @@ export function mdToHtml(src, opts) {
   // Remove empty paragraphs
   s = s.replace(/<p><\/p>/g, '');
 
+  // Every restore below passes a function replacer rather than the block string
+  // itself. With a string replacement, `String.replace` reads `$&`, `` $` ``,
+  // `$'` and `$$` in the *replacement* as substitution patterns, so a restored
+  // block containing them is corrupted: `$&` re-inserts the placeholder, `` $` ``
+  // and `$'` splice in the surrounding document, and `$$` collapses to `$`. Those
+  // sequences are ordinary content in fenced code (`perl -pe 's/x/$& y/'`,
+  // `echo "$$USD"`). A function replacer inserts its return value verbatim.
+
   // CRITICAL: Restore allowed HTML blocks first
   allowedHtmlBlocks.forEach((block, index) => {
-    s = s.replace(`___ALLOWED_HTML_${index}___`, block);
+    s = s.replace(`___ALLOWED_HTML_${index}___`, () => block);
   });
 
   // Restore math blocks
   mathBlocks.forEach((block, index) => {
-    s = s.replace(`___MATH_BLOCK_${index}___`, block);
+    s = s.replace(`___MATH_BLOCK_${index}___`, () => block);
   });
 
   // Restore mermaid diagram blocks
   mermaidBlocks.forEach((block, index) => {
-    s = s.replace(`___MERMAID_BLOCK_${index}___`, block);
+    s = s.replace(`___MERMAID_BLOCK_${index}___`, () => block);
   });
 
   // CRITICAL: Restore code blocks at the end
   codeBlocks.forEach((block, index) => {
-    s = s.replace(`___CODE_BLOCK_${index}___`, block);
+    s = s.replace(`___CODE_BLOCK_${index}___`, () => block);
   });
 
   // Restore inline code spans last, so placeholders carried inside restored
-  // <a>/allowed-HTML blocks are resolved too. The function replacer keeps the
-  // escaped code literal — e.g. a shell snippet like `echo $1` is not treated
-  // as a regex back-reference.
+  // <a>/allowed-HTML blocks are resolved too.
   inlineCodeBlocks.forEach((block, index) => {
     s = s.replace(`___INLINE_CODE_${index}___`, () => block);
   });

static/js/sessions.js

@@ -1683,8 +1683,21 @@ export async function loadSessions() {
         url += `?active_incognito_id=${encodeURIComponent(currentSessionId)}`;
       }
       const res = await fetch(url);
+      if (!res.ok) {
+        let detail = '';
+        try {
+          const payload = await res.json();
+          detail = payload?.detail || payload?.error || '';
+        } catch (_) {}
+        const error = new Error(detail || `Session request failed (HTTP ${res.status})`);
+        error.status = res.status;
+        throw error;
+      }
       fetched = await res.json();
     }
+    if (!Array.isArray(fetched)) {
+      throw new Error('Session request returned an invalid response');
+    }
     sessions = _normalizeSessionsList(fetched);
     renderSessionList();
 
@@ -1807,9 +1820,15 @@ export async function loadSessions() {
         _autoCreateInProgress = false;
       }
     }
+    return true;
   } catch (error) {
     console.error('Error in loadSessions:', error);
-    uiModule.showError('Failed to load sessions: ' + error.message);
+    // app.js's global fetch wrapper owns expired-auth navigation. Avoid
+    // flashing a redundant session error while that 401 redirect is pending.
+    if (error?.status !== 401) {
+      uiModule.showError('Failed to load sessions: ' + error.message);
+    }
+    return false;
   }
 }
 
@@ -1847,6 +1866,10 @@ export async function selectSession(id, { keepSidebar = false, showLoading = tru
     const _isTransientChat = !!_meta && (_meta.folder === 'Assistant' || _meta.folder === 'Tasks');
     if (!_isTransientChat) {
       Storage.set('lastSessionId', id);
+      // Update URL hash without triggering hashchange handler
+      if (window.location.hash !== '#' + id) {
+        history.replaceState(null, '', '#' + id);
+      }
     }
     // Restore character preset for persistent chats
     try {
@@ -2313,6 +2336,7 @@ export async function materializePendingSession() {
     currentSessionId = payload.id;
     if (!isIncognito) {
       Storage.set('lastSessionId', payload.id);
+      history.replaceState(null, '', '#' + payload.id);
     }
 
     // Reload the sidebar in the background. Awaiting this used to block the first

static/js/settings.js

@@ -3031,12 +3031,14 @@ async function initEmailAccountsSettings() {
       const body = {
         name: el('eaf-name').value.trim() || el('eaf-from').value.trim(),
         from_address: el('eaf-from').value.trim(),
+        display_name: el('eaf-display-name').value.trim(),
         imap_host: el('eaf-imap-host').value.trim(),
         imap_port: parseInt(el('eaf-imap-port').value) || 993,
         imap_user: el('eaf-imap-user').value.trim(),
         imap_starttls: el('eaf-imap-starttls').checked,
         smtp_host: el('eaf-smtp-host').value.trim(),
         smtp_port: parseInt(el('eaf-smtp-port').value) || 587,
+        smtp_security: el('eaf-smtp-security').value,
         smtp_user: el('eaf-imap-user').value.trim(),
       };
       if (!body.name) { el('eaf-msg').textContent = 'Enter a Name or Email first'; el('eaf-msg').style.color = 'var(--red)'; return; }
@@ -5788,29 +5790,30 @@ export function close() {
   window.history.replaceState(null, '', clean);
   const success = sp.has('email_oauth_success');
   const errMsg = sp.get('email_oauth_error') || '';
-  // Open settings → integrations after the app has initialised.
-  function _tryOpen() {
-    if (window.settingsModule && typeof window.settingsModule.open === 'function') {
-      window.settingsModule.open('integrations');
-      // Brief toast-style banner.
-      const banner = document.createElement('div');
-      banner.textContent = success
-        ? '✓ Google account connected — email is ready'
-        : `Google OAuth failed: ${errMsg || 'unknown error'}`;
-      Object.assign(banner.style, {
-        position: 'fixed', bottom: '24px', left: '50%', transform: 'translateX(-50%)',
-        background: success ? 'var(--accent, #50fa7b)' : 'var(--red, #ff5555)',
-        color: '#000', padding: '8px 18px', borderRadius: '6px', fontSize: '12px',
-        fontWeight: '600', zIndex: '99999', pointerEvents: 'none',
-        boxShadow: '0 2px 12px rgba(0,0,0,0.3)',
-      });
-      document.body.appendChild(banner);
-      setTimeout(() => banner.remove(), 4000);
-    } else {
-      setTimeout(_tryOpen, 100);
-    }
+  // Open settings → integrations once the document is ready. This module owns
+  // the open() API, so it does not need to wait for a window-level alias.
+  function _showResult() {
+    open('integrations');
+    // Brief toast-style banner.
+    const banner = document.createElement('div');
+    banner.textContent = success
+      ? 'Google account connected — email is ready'
+      : `Google OAuth failed: ${errMsg || 'unknown error'}`;
+    Object.assign(banner.style, {
+      position: 'fixed', bottom: '24px', left: '50%', transform: 'translateX(-50%)',
+      background: success ? 'var(--accent, #50fa7b)' : 'var(--red, #ff5555)',
+      color: '#000', padding: '8px 18px', borderRadius: '6px', fontSize: '12px',
+      fontWeight: '600', zIndex: '99999', pointerEvents: 'none',
+      boxShadow: '0 2px 12px rgba(0,0,0,0.3)',
+    });
+    document.body.appendChild(banner);
+    setTimeout(() => banner.remove(), 4000);
+  }
+  if (document.readyState === 'loading') {
+    document.addEventListener('DOMContentLoaded', _showResult, { once: true });
+  } else {
+    _showResult();
   }
-  _tryOpen();
 })();
 
 const settingsModule = { open, close, initIntegrations, initUnifiedIntegrations, syncAdminVisibility, refreshAiModelEndpoints };

static/js/skills.js

@@ -83,11 +83,9 @@ export async function loadSkills(cascade = false) {
   // Play the domino-in entrance on this load (set when the tab is opened,
   // not for the silent re-loads after an edit/delete).
   if (cascade) _cascadeNext = true;
-  if (cascade && loaded && !_loadPromise && _playSkillsCascade()) {
-    _cascadeNext = false;
-    updateCount();
-    return;
-  }
+  // Always re-fetch when the tab is explicitly opened — the cascade
+  // animation is handled inside renderSkillsList() via _cascadeNext.
+  // Skipping the fetch here caused stale data on panel close/reopen (#5870).
   if (_loadPromise) return _loadPromise;
   _loadPromise = (async () => {
   try {

static/js/startupShell.js

@@ -0,0 +1,153 @@
+// Odysseus UI — startup shell sequencing
+// ES6 module — no application dependencies, DOM only.
+//
+// Revealing the application shell, retiring the boot loader, settling the
+// sidebar's own loading state, and firing a deferred URL route are separate
+// startup concerns that used to sit inline in app.js behind a single promise.
+// They live here so each step has one owner and so the whole contract can be
+// exercised directly (tests/test_startup_shell_js.py) without booting the app.
+
+const LOADER_ID = 'app-loader';
+const SESSION_BOOTSTRAP_ROW_ID = 'session-list-loading';
+
+// Route openers that read the hydrated session list. Everything else only
+// needs module wiring and must not wait on /api/sessions. `/email` spawns a
+// fresh chat, and that path falls back to the most recent session's model
+// (_createDirectChatFromPreferredModel in app.js) when there is no default
+// chat configured, so it genuinely needs the list.
+const ROUTES_NEEDING_SESSIONS = new Set(['/email']);
+
+let _routeOpener = null;
+let _routeOpenerNeedsSessions = false;
+
+function _loader() {
+  return document.getElementById(LOADER_ID);
+}
+
+/** Run `fn` after the next paint has committed (two animation frames). */
+export function afterNextPaint(fn) {
+  requestAnimationFrame(() => requestAnimationFrame(fn));
+}
+
+// The loader node stays in the DOM while sessions hydrate — sidebar-layout.js
+// and sessions.js both read its presence as a "still starting up" sentinel —
+// but it must stop covering, announcing, and animating over a usable shell.
+function _makeLoaderInert(loader) {
+  if (!loader || loader.dataset.shellRevealed === 'true') return;
+  loader.dataset.shellRevealed = 'true';
+  loader.setAttribute('aria-hidden', 'true');
+  loader.style.pointerEvents = 'none';
+  loader.style.opacity = '0';
+  // index.html's inline bootstrap animates the wave on a 150ms interval.
+  // Nothing of it is visible any more, so stop rendering into it.
+  try { window.__odysseusLoaderWaveStop?.(); } catch (_) {}
+}
+
+/**
+ * Hand the shell to the user once core wiring is done. Deferred by one paint
+ * so the first frame lands with the app already laid out.
+ */
+export function revealApplicationShellAfterPaint() {
+  const loader = _loader();
+  if (!loader || loader.dataset.shellRevealScheduled === 'true') return;
+  loader.dataset.shellRevealScheduled = 'true';
+  afterNextPaint(() => _makeLoaderInert(_loader()));
+}
+
+/** Retire the loader node for good. Safe to call after a reveal. */
+export function removeApplicationLoader() {
+  const loader = _loader();
+  if (!loader) return;
+  _makeLoaderInert(loader);
+  setTimeout(() => loader.remove(), 300);
+}
+
+/**
+ * Turn the sidebar's bootstrap row into a failure row. The write is delayed
+ * until the session renderer's frame has committed so a late success cannot
+ * leave stale failure text behind.
+ */
+export function markSessionListUnavailableIfStillBootstrapping() {
+  afterNextPaint(() => {
+    const row = document.getElementById(SESSION_BOOTSTRAP_ROW_ID);
+    if (!row) return;
+    const status = row.querySelector('[data-session-list-status]') || row;
+    status.textContent = 'Chats unavailable';
+  });
+}
+
+/** True when `path`'s route opener reads the hydrated session list. */
+export function routeNeedsSessionData(path) {
+  return ROUTES_NEEDING_SESSIONS.has(path);
+}
+
+/**
+ * Stash a URL route opener for later. At the point app.js resolves the route,
+ * the modules its handlers drive (the rail new-chat handler, the email
+ * section header handler, sessionModule) are still being wired further down
+ * the same init pass, so the opener cannot run inline.
+ */
+export function deferRouteOpener(path, opener) {
+  if (!opener) return;
+  _routeOpener = opener;
+  _routeOpenerNeedsSessions = routeNeedsSessionData(path);
+}
+
+/**
+ * Fire the deferred route opener if its data is ready. Called once when
+ * wiring completes and again after authoritative session hydration; a route
+ * that needs no session data takes the first call, one that does takes the
+ * second.
+ *
+ * @returns {boolean} whether an opener ran.
+ */
+export function runDeferredRouteOpener({ sessionsSettled = false } = {}) {
+  if (!_routeOpener) return false;
+  if (_routeOpenerNeedsSessions && !sessionsSettled) return false;
+  const opener = _routeOpener;
+  _routeOpener = null;
+  _routeOpenerNeedsSessions = false;
+  try { opener(); } catch (e) { console.warn('route opener failed:', e); }
+  return true;
+}
+
+/**
+ * Drive session hydration and everything that hangs off it settling: the
+ * sidebar's failure row, the loader node, and any session-dependent route.
+ *
+ * @param {(() => Promise<boolean>)|null} loadSessions — resolves true only
+ *   after the session list was authoritatively loaded and applied. Null means
+ *   the session module failed to load.
+ */
+export function settleSessionHydration(loadSessions) {
+  const settle = (succeeded) => {
+    if (!succeeded) {
+      markSessionListUnavailableIfStillBootstrapping();
+      // A later unrelated caller must not be able to release a stale startup
+      // opener against unknown session state.
+      _routeOpener = null;
+      _routeOpenerNeedsSessions = false;
+    }
+    removeApplicationLoader();
+    if (succeeded) runDeferredRouteOpener({ sessionsSettled: true });
+    return succeeded;
+  };
+  if (!loadSessions) {
+    return Promise.resolve(settle(false));
+  }
+  // Kick the request off synchronously — a microtask hop here would delay the
+  // fetch this whole change exists to get off the critical path.
+  let pending;
+  try {
+    pending = loadSessions();
+  } catch (e) {
+    console.warn('loadSessions error:', e);
+    return Promise.resolve(settle(false));
+  }
+  return Promise.resolve(pending)
+    .then(result => settle(result === true))
+    .catch(e => {
+      console.warn('loadSessions error:', e);
+      return settle(false);
+    });
+}

static/style.css

@@ -38015,6 +38015,12 @@ button.cal-add-btn.cal-add-btn-text.cal-add-btn-sm:hover .cal-add-label {
   outline-offset: 2px;
   border-radius: 5px;
 }
+/* Bootstrap row shown while the session list hydrates, and on load failure.
+   Reads as a normal list row but is not selectable. */
+.session-list-bootstrap {
+  cursor: default;
+  pointer-events: none;
+}
 #email-lib-grid .date-section-header {
   padding: 10px 5px 3px;
 }

tests/live_thinking_scheduler.test.mjs

@@ -0,0 +1,277 @@
+// Tests for the live-thinking throttle that bounds DOM work during long
+// reasoning streams (see static/js/liveThinkingThrottle.js).
+//
+// The throttle's contract is what the terminal paths in chat.js lean on:
+// a burst of deltas becomes ONE commit carrying the latest text; flush()
+// lands trailing text synchronously and cannot double-commit; cancel()
+// guarantees nothing lands after a stream is finished or backgrounded.
+//
+// Timers are injected, so this runs with no DOM and no real clock.
+import assert from 'node:assert/strict';
+import test from 'node:test';
+
+import {
+  createIncrementalDisplayProjector,
+  createLiveThinkingThrottle,
+  createThinkingAnalysisGate,
+  stripLiveThinkingTags,
+} from '../static/js/liveThinkingThrottle.js';
+
+function fakeTimers() {
+  let nextId = 1;
+  const callbacks = new Map();
+  const delays = [];
+  return {
+    schedule(callback, delay) {
+      const id = nextId++;
+      callbacks.set(id, callback);
+      delays.push(delay);
+      return id;
+    },
+    cancel(id) {
+      callbacks.delete(id);
+    },
+    run(id) {
+      const callback = callbacks.get(id);
+      assert.ok(callback, `missing timer ${id}`);
+      callbacks.delete(id);
+      callback();
+    },
+    pendingIds() {
+      return [...callbacks.keys()];
+    },
+    delays,
+  };
+}
+
+test('coalesces a burst and commits only the latest text after 100 ms', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  const throttle = createLiveThinkingThrottle((value) => commits.push(value), timers);
+
+  throttle.update('a');
+  throttle.update('ab');
+  throttle.update('abc');
+
+  assert.deepEqual(commits, []);
+  assert.deepEqual(timers.delays, [100], 'a burst must schedule exactly one commit');
+  const [timer] = timers.pendingIds();
+  timers.run(timer);
+  assert.deepEqual(commits, ['abc']);
+});
+
+test('commit count stays flat as the stream grows', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  const throttle = createLiveThinkingThrottle((value) => commits.push(value), timers);
+
+  // 500 deltas arriving inside one window is the regression this guards:
+  // the old code committed once per delta, so work grew with stream length.
+  let text = '';
+  for (let i = 0; i < 500; i++) {
+    text += 'token ';
+    throttle.update(text);
+  }
+  assert.deepEqual(commits, []);
+  assert.equal(timers.pendingIds().length, 1);
+  timers.run(timers.pendingIds()[0]);
+  assert.equal(commits.length, 1);
+  assert.equal(commits[0], text);
+});
+
+test('prepares a 200K cumulative stream only at scheduled commit cadence', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  let prepareCalls = 0;
+  let scannedCharacters = 0;
+  const throttle = createLiveThinkingThrottle((value) => commits.push(value), {
+    ...timers,
+    prepare(value) {
+      prepareCalls += 1;
+      scannedCharacters += value.length;
+      return stripLiveThinkingTags(value);
+    },
+  });
+
+  const delta = 'reasoning '.repeat(10); // 100 characters
+  let cumulative = '';
+  for (let i = 0; i < 2000; i++) {
+    cumulative += delta;
+    throttle.update(cumulative);
+  }
+
+  assert.equal(cumulative.length, 200_000);
+  assert.equal(prepareCalls, 0, 'cumulative extraction must not run per delta');
+  assert.equal(timers.pendingIds().length, 1);
+  timers.run(timers.pendingIds()[0]);
+  assert.equal(prepareCalls, 1);
+  assert.equal(scannedCharacters, 200_000);
+  assert.deepEqual(commits, [cumulative]);
+});
+
+test('ordinary answers and reasoning deltas do not request cumulative analysis', () => {
+  const startsReasoning = (text) => /^\s*thinking(?:\s+process)?\s*:/i.test(text);
+  const ordinaryGate = createThinkingAnalysisGate({ startsWithReasoningPrefix: startsReasoning });
+  let ordinary = '';
+  let ordinaryAnalyses = 0;
+  for (let i = 0; i < 2000; i++) {
+    ordinary += i === 0 ? 'Here is the answer. ' : 'answer '.repeat(10);
+    if (ordinaryGate.shouldAnalyze(ordinary)) ordinaryAnalyses += 1;
+  }
+  assert.equal(ordinaryAnalyses, 0);
+
+  const thinkingGate = createThinkingAnalysisGate({ startsWithReasoningPrefix: startsReasoning });
+  let thinking = 'Thin';
+  assert.equal(thinkingGate.shouldAnalyze(thinking), false);
+  thinking += 'king: inspect the problem';
+  assert.equal(thinkingGate.shouldAnalyze(thinking), true);
+  for (let i = 0; i < 2000; i++) {
+    thinking += ' reasoning'.repeat(10);
+    assert.equal(thinkingGate.shouldAnalyze(thinking, { isThinking: true, nonTagThinking: true }), false);
+  }
+  thinking += '\n\nHere is the answer';
+  assert.equal(thinkingGate.shouldAnalyze(thinking, { isThinking: true, nonTagThinking: true }), true);
+
+  const whitespaceGate = createThinkingAnalysisGate({ startsWithReasoningPrefix: startsReasoning });
+  let whitespaceThinking = ' '.repeat(250);
+  assert.equal(whitespaceGate.shouldAnalyze(whitespaceThinking), false);
+  whitespaceThinking += 'Thinking: bounded probe';
+  assert.equal(whitespaceGate.shouldAnalyze(whitespaceThinking), true);
+});
+
+test('split namespaced closes and false-close deadlines request analysis', () => {
+  let clock = 100;
+  const gate = createThinkingAnalysisGate({ now: () => clock });
+  let text = '<mm:think>x</mm:';
+  assert.equal(gate.shouldAnalyze(text, { isThinking: true }), true, 'fresh opening tag is analyzed');
+  text += 'think>answer';
+  assert.equal(gate.shouldAnalyze(text, { isThinking: true }), true, 'split namespaced close is analyzed');
+
+  text += ' still waiting';
+  assert.equal(gate.shouldAnalyze(text, { isThinking: true, recheckAt: 500 }), false);
+  clock = 500;
+  text += ' next delta';
+  assert.equal(gate.shouldAnalyze(text, { isThinking: true, recheckAt: 500 }), true);
+
+  const attributedGate = createThinkingAnalysisGate();
+  let attributed = `<think data-provider="${'x'.repeat(400)}"`;
+  assert.equal(attributedGate.shouldAnalyze(attributed), false);
+  attributed += '>reasoning';
+  assert.equal(attributedGate.shouldAnalyze(attributed), true, 'bounded carry preserves split tag attributes');
+});
+
+test('display projection is append-only and filters a structured tail once', () => {
+  let filterCalls = 0;
+  let filteredCharacters = 0;
+  const projector = createIncrementalDisplayProjector((text) => {
+    filterCalls += 1;
+    filteredCharacters += text.length;
+    return text.replace(/\[TOOL_CALL\][\s\S]*$/i, '');
+  });
+
+  let text = '';
+  for (let i = 0; i < 2000; i++) {
+    const delta = i === 0 ? 'Here is the answer. ' : 'ordinary text ';
+    text += delta;
+    assert.equal(projector.append(delta, text), text);
+  }
+  assert.equal(filterCalls, 0, 'ordinary deltas never run the cumulative filter');
+
+  text += '[TOOL_';
+  projector.append('[TOOL_', text);
+  text += 'CALL]{"name":"read"}';
+  const beforeToolPayload = projector.append('CALL]{"name":"read"}', text);
+  for (let i = 0; i < 2000; i++) {
+    const delta = 'payload ';
+    text += delta;
+    assert.equal(projector.append(delta, text), beforeToolPayload);
+  }
+  assert.equal(filterCalls, 1, 'structured payload filtering happens only at its boundary');
+  assert.ok(filteredCharacters < text.length, 'filter work is bounded by the first structured boundary');
+});
+
+test('literal escaped tags survive and malformed live tags retain trailing text', () => {
+  assert.equal(
+    stripLiveThinkingTags('&lt;think&gt;literal&lt;/think&gt;'),
+    '&lt;think&gt;literal&lt;/think&gt;',
+  );
+  assert.equal(
+    stripLiveThinkingTags('<think>first</think> middle <thinking mode="deep">trailing'),
+    'first middle trailing',
+  );
+  assert.equal(stripLiveThinkingTags('answer with 2 < 3 and 5 > 4'), 'answer with 2 < 3 and 5 > 4');
+});
+
+test('terminal flush prepares and commits the complete trailing cumulative text', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  const throttle = createLiveThinkingThrottle((value) => commits.push(value), {
+    ...timers,
+    prepare: stripLiveThinkingTags,
+  });
+
+  throttle.update('<think>reasoning without a closing tag');
+  assert.equal(throttle.flush(), true);
+  assert.deepEqual(commits, ['reasoning without a closing tag']);
+  assert.deepEqual(timers.pendingIds(), []);
+});
+
+test('independent throttles cannot commit cancelled text into another session', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  const first = createLiveThinkingThrottle((value) => commits.push(['first', value]), timers);
+  const second = createLiveThinkingThrottle((value) => commits.push(['second', value]), timers);
+
+  first.update('stale first-session text');
+  second.update('current second-session text');
+  first.cancel();
+  assert.equal(second.flush(), true);
+
+  assert.deepEqual(timers.pendingIds(), []);
+  assert.deepEqual(commits, [['second', 'current second-session text']]);
+});
+
+test('flush synchronously preserves trailing text and cancels the pending callback', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  const throttle = createLiveThinkingThrottle((value) => commits.push(value), timers);
+
+  throttle.update('trailing text');
+  assert.equal(throttle.flush(), true);
+  assert.deepEqual(commits, ['trailing text']);
+  assert.deepEqual(timers.pendingIds(), []);
+  assert.equal(throttle.flush(), false, 'clean flush must not duplicate the commit');
+});
+
+test('cancel discards pending work without a late DOM commit', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  const throttle = createLiveThinkingThrottle((value) => commits.push(value), timers);
+
+  throttle.update('stale session text');
+  throttle.cancel();
+  assert.deepEqual(timers.pendingIds(), []);
+  assert.deepEqual(commits, []);
+});
+
+test('a cancelled throttle accepts new work again', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  const throttle = createLiveThinkingThrottle((value) => commits.push(value), timers);
+
+  throttle.update('discarded');
+  throttle.cancel();
+  throttle.update('fresh');
+  assert.equal(throttle.flush(), true);
+  assert.deepEqual(commits, ['fresh']);
+});
+
+test('coerces nullish updates instead of committing undefined', () => {
+  const timers = fakeTimers();
+  const commits = [];
+  const throttle = createLiveThinkingThrottle((value) => commits.push(value), timers);
+
+  throttle.update(null);
+  throttle.flush();
+  assert.deepEqual(commits, ['']);
+});

tests/test_api_chat_security.py

@@ -76,7 +76,7 @@ def _load_webhook_routes_for_test(monkeypatch):
     module_name = "routes.webhook_routes_under_test"
     spec = importlib.util.spec_from_file_location(
         module_name,
-        Path(__file__).resolve().parent.parent / "routes" / "webhook_routes.py",
+        Path(__file__).resolve().parent.parent / "routes" / "webhook" / "webhook_routes.py",
     )
     module = importlib.util.module_from_spec(spec)
     spec.loader.exec_module(module)

tests/test_backup_import_cross_user_dedup.py

@@ -27,6 +27,9 @@ def _setup(monkeypatch, store, user="alice"):
 
     mem = MagicMock()
     mem.load_all.return_value = list(store)
+    # import_data reads through the strict loader so a store it cannot read is
+    # never overwritten (#5673); the double has to offer the same entry point.
+    mem.load_all_for_update.return_value = list(store)
     saved = {}
     mem.save.side_effect = lambda entries: saved.__setitem__("entries", entries)
 

tests/test_chat_stream_scope.py

@@ -1,3 +1,4 @@
+import re
 from pathlib import Path
 
 
@@ -13,7 +14,7 @@ def test_stream_render_helpers_are_visible_to_catch_block():
     assert "let _cancelThinkingTimer = () => {};" in outer_scope
     assert "let _removeThinkingSpinner = () => {};" in outer_scope
 
-    assert "_renderStream = () => {" in try_body
+    assert re.search(r"(?m)^\s*_renderStream\s*=", try_body)
     assert "_cancelThinkingTimer = () => {" in try_body
     assert "_removeThinkingSpinner = () => {" in try_body
     assert "function _renderStream()" not in try_body

tests/test_composer_arrow_up_recall_js.py

@@ -306,3 +306,24 @@ def test_integration_recalls_from_chat_history_dom():
     )
     assert proc.returncode == 0, proc.stderr
     assert json.loads(proc.stdout.strip()) == {"value": "stored prompt", "prevented": True}
+
+
+def test_prompt_recall_is_not_duplicated_in_app_js():
+    """Only composerArrowUpRecall.js may own ArrowUp on #message (issue #5862).
+
+    static/app.js once carried a near-verbatim copy of this recall logic, wired
+    as a second capture-phase listener on the same textarea. That copy lacked
+    the draft guard here, and because it called stopImmediatePropagation it won
+    regardless of registration order — so a typed multi-line prompt was replaced
+    by the last sent one instead of the caret moving up a line.
+    """
+    app_js = (_REPO / "static" / "app.js").read_text(encoding="utf-8")
+    for marker in (
+        "_odysseusPromptRecallCapture",
+        "_readComposerPromptHistory",
+        "odysseusRecallIndex",
+    ):
+        assert marker not in app_js, (
+            f"static/app.js reintroduces prompt recall ({marker!r}); "
+            "it belongs to static/js/composerArrowUpRecall.js alone"
+        )

tests/test_document_routes_shim.py

@@ -0,0 +1,29 @@
+"""Regression test for the document route shim (slice 2m, #4082/#4071).
+
+The backward-compat shims at ``routes/document_routes.py`` and
+``routes/document_helpers.py`` use ``sys.modules`` replacement so the legacy
+import paths and the canonical ``routes.document.*`` paths resolve to the
+*same* module objects. This is required because multiple tests do
+``import routes.document_routes as droutes`` followed by
+``droutes.SessionLocal = ...`` / ``monkeypatch.setattr(droutes, ...)`` and
+``sys.modules.pop("routes.document_helpers")`` + re-import — for those to
+take effect at runtime, the legacy and canonical module objects must be
+identical.
+"""
+
+import importlib
+
+import routes.document_routes as _shim_routes  # noqa: F401
+import routes.document_helpers as _shim_helpers  # noqa: F401
+
+
+def test_legacy_and_canonical_routes_are_same_object():
+    legacy = importlib.import_module("routes.document_routes")
+    canonical = importlib.import_module("routes.document.document_routes")
+    assert legacy is canonical
+
+
+def test_legacy_and_canonical_helpers_are_same_object():
+    legacy = importlib.import_module("routes.document_helpers")
+    canonical = importlib.import_module("routes.document.document_helpers")
+    assert legacy is canonical

tests/test_email_library_prewarm.py

@@ -0,0 +1,414 @@
+import json
+from pathlib import Path
+import shutil
+import subprocess
+
+import pytest
+
+
+_REPO = Path(__file__).resolve().parents[1]
+_EMAIL_LIBRARY = _REPO / "static" / "js" / "emailLibrary.js"
+
+
+def _source() -> str:
+    return _EMAIL_LIBRARY.read_text(encoding="utf-8")
+
+
+def _function_source(name: str) -> str:
+    """Return one top-level JS function using balanced braces."""
+    text = _source()
+    markers = (f"function {name}", f"async function {name}", f"export function {name}", f"export async function {name}")
+    starts = [text.find(marker) for marker in markers]
+    starts = [start for start in starts if start >= 0]
+    assert starts, f"missing function {name}"
+    start = min(starts)
+    paren = text.index("(", start)
+    paren_depth = 0
+    quote = None
+    escaped = False
+    for index in range(paren, len(text)):
+        char = text[index]
+        if quote:
+            if escaped:
+                escaped = False
+            elif char == "\\":
+                escaped = True
+            elif char == quote:
+                quote = None
+            continue
+        if char in ("'", '"', "`"):
+            quote = char
+        elif char == "(":
+            paren_depth += 1
+        elif char == ")":
+            paren_depth -= 1
+            if paren_depth == 0:
+                brace = text.index("{", index)
+                break
+    else:
+        raise AssertionError(f"unterminated signature {name}")
+    depth = 0
+    quote = None
+    escaped = False
+    template_depth = 0
+    for index in range(brace, len(text)):
+        char = text[index]
+        if quote:
+            if escaped:
+                escaped = False
+            elif char == "\\":
+                escaped = True
+            elif char == quote and template_depth == 0:
+                quote = None
+            elif quote == "`" and char == "$" and index + 1 < len(text) and text[index + 1] == "{":
+                template_depth += 1
+            elif quote == "`" and char == "}" and template_depth:
+                template_depth -= 1
+            continue
+        if char in ("'", '"', "`"):
+            quote = char
+        elif char == "{":
+            depth += 1
+        elif char == "}":
+            depth -= 1
+            if depth == 0:
+                return text[start:index + 1]
+    raise AssertionError(f"unterminated function {name}")
+
+
+def _run_scheduler_scenario(scenario: str):
+    node = shutil.which("node")
+    if not node:
+        pytest.skip("node not on PATH")
+    functions = "\n".join(
+        _function_source(name)
+        for name in (
+            "_isChatInteractionBusy",
+            "_canRunEmailPrewarm",
+            "_isEmailPrewarmTemporarilyBlocked",
+            "_settleEmailPrewarm",
+            "_cancelEmailPrewarm",
+            "_scheduleEmailPrewarm",
+        )
+    )
+    script = f"""
+      let now = 0;
+      Date.now = () => now;
+      const state = {{ _libOpen: false, _libLoading: false }};
+      let _libSearchInFlight = false;
+      let _libPrewarmDelayTimer = null;
+      let _libPrewarmIdleHandle = null;
+      let _libPrewarmPromise = null;
+      let _libPrewarmResolve = null;
+      let _libPrewarmAbortController = null;
+      let _libPrewarmDetachPriorityListeners = null;
+      let _libPrewarmGeneration = 0;
+      let nextHandle = 1;
+      const timers = new Map();
+      const idleCallbacks = new Map();
+      let idleRequestCount = 0;
+      function eventTarget(target) {{
+        const listeners = new Map();
+        target.addEventListener = (type, callback) => {{
+          if (!listeners.has(type)) listeners.set(type, new Set());
+          listeners.get(type).add(callback);
+        }};
+        target.removeEventListener = (type, callback) => listeners.get(type)?.delete(callback);
+        target.dispatchEvent = (event) => {{
+          for (const callback of [...(listeners.get(event.type) || [])]) callback(event);
+        }};
+        target.listenerCount = (type) => listeners.get(type)?.size || 0;
+        return target;
+      }}
+      const document = eventTarget({{ visibilityState: 'visible' }});
+      const window = {{
+        __odysseusChatBusy: false,
+        __odysseusChatBusyUntil: 0,
+        requestIdleCallback(callback) {{
+          const handle = nextHandle++;
+          idleRequestCount += 1;
+          idleCallbacks.set(handle, callback);
+          return handle;
+        }},
+        cancelIdleCallback(handle) {{ idleCallbacks.delete(handle); }},
+      }};
+      eventTarget(window);
+      function setTimeout(callback, delay) {{
+        const handle = nextHandle++;
+        timers.set(handle, {{ callback, at: now + Number(delay || 0) }});
+        return handle;
+      }}
+      function clearTimeout(handle) {{ timers.delete(handle); }}
+      async function flushMicrotasks() {{
+        for (let i = 0; i < 6; i += 1) await Promise.resolve();
+      }}
+      async function advanceTo(target) {{
+        while (true) {{
+          const pending = [...timers.entries()]
+            .filter(([, timer]) => timer.at <= target)
+            .sort((a, b) => a[1].at - b[1].at)[0];
+          if (!pending) break;
+          const [handle, timer] = pending;
+          timers.delete(handle);
+          now = timer.at;
+          timer.callback();
+          await flushMicrotasks();
+        }}
+        now = target;
+        await flushMicrotasks();
+      }}
+      async function fireNextIdle(budget = 5) {{
+        const pending = idleCallbacks.entries().next().value;
+        if (!pending) throw new Error('no idle callback pending');
+        const [handle, callback] = pending;
+        idleCallbacks.delete(handle);
+        callback({{ didTimeout: false, timeRemaining: () => budget }});
+        await flushMicrotasks();
+      }}
+      {functions}
+      {scenario}
+    """
+    proc = subprocess.run(
+        [node, "--input-type=module"],
+        input=script,
+        capture_output=True,
+        text=True,
+        cwd=str(_REPO),
+        timeout=30,
+    )
+    assert proc.returncode == 0, proc.stderr
+    return json.loads(proc.stdout.strip())
+
+
+def test_prewarm_is_genuine_idle_only_and_single_flight():
+    scheduler = _function_source("_scheduleEmailPrewarm")
+
+    assert "if (_libPrewarmPromise) return _libPrewarmPromise;" in scheduler
+    assert "typeof window.requestIdleCallback !== 'function'" in scheduler
+    assert "return Promise.resolve(false);" in scheduler
+    assert "window.requestIdleCallback((deadline)" in scheduler
+    assert "!deadline.didTimeout" in scheduler
+    assert "deadline.timeRemaining() > 0" in scheduler
+
+    idle_callback = scheduler.index("window.requestIdleCallback((deadline)")
+    assert "Promise.resolve()" in scheduler
+    task_start = scheduler.index("task({ signal: controller.signal, generation })")
+    assert idle_callback < task_start, "network work must only be reachable from the idle callback"
+
+
+def test_temporary_chat_priority_retries_one_single_flight_until_idle():
+    out = _run_scheduler_scenario("""
+      window.__odysseusChatBusyUntil = 10000;
+      let taskCalls = 0;
+      const task = async () => { taskCalls += 1; return true; };
+      const first = _scheduleEmailPrewarm(task, { delay: 1800 });
+      const joined = _scheduleEmailPrewarm(task, { delay: 0 });
+      const samePromise = first === joined;
+      await advanceTo(1800);
+      await fireNextIdle(7);
+      const callsWhileBusy = taskCalls;
+      while (now < 10300) {
+        await advanceTo(now + 500);
+        await fireNextIdle(7);
+      }
+      const result = await first;
+      console.log(JSON.stringify({
+        result, samePromise, callsWhileBusy, taskCalls, idleRequestCount,
+        timers: timers.size, idleCallbacks: idleCallbacks.size,
+      }));
+    """)
+    assert out == {
+        "result": True,
+        "samePromise": True,
+        "callsWhileBusy": 0,
+        "taskCalls": 1,
+        "idleRequestCount": 18,
+        "timers": 0,
+        "idleCallbacks": 0,
+    }
+
+
+def test_cancelled_prewarm_cannot_issue_a_delayed_duplicate():
+    out = _run_scheduler_scenario("""
+      let taskCalls = 0;
+      const pending = _scheduleEmailPrewarm(async () => { taskCalls += 1; return true; }, { delay: 1800 });
+      await advanceTo(1400);
+      _cancelEmailPrewarm();
+      await advanceTo(12000);
+      const result = await pending;
+      console.log(JSON.stringify({
+        result, taskCalls, idleRequestCount,
+        timers: timers.size, idleCallbacks: idleCallbacks.size,
+      }));
+    """)
+    assert out == {
+        "result": False,
+        "taskCalls": 0,
+        "idleRequestCount": 0,
+        "timers": 0,
+        "idleCallbacks": 0,
+    }
+
+
+@pytest.mark.parametrize("transition", ["busy", "hidden"])
+def test_active_prewarm_is_aborted_and_retried_once_after_priority_transition(transition):
+    block = (
+        "window.__odysseusChatBusy = true; "
+        "window.dispatchEvent({ type: 'odysseus:chat-busy-change' });"
+        if transition == "busy"
+        else "document.visibilityState = 'hidden'; document.dispatchEvent({ type: 'visibilitychange' });"
+    )
+    unblock = (
+        "window.__odysseusChatBusy = false; window.__odysseusChatBusyUntil = now; "
+        "window.dispatchEvent({ type: 'odysseus:chat-busy-change' });"
+        if transition == "busy"
+        else "document.visibilityState = 'visible'; document.dispatchEvent({ type: 'visibilitychange' });"
+    )
+    out = _run_scheduler_scenario(f"""
+      let taskCalls = 0;
+      let firstSignal = null;
+      let finishFirst;
+      const firstAttempt = new Promise(resolve => {{ finishFirst = resolve; }});
+      const pending = _scheduleEmailPrewarm(async ({{ signal }}) => {{
+        taskCalls += 1;
+        if (taskCalls === 1) {{ firstSignal = signal; return firstAttempt; }}
+        return true;
+      }});
+      await fireNextIdle(7);
+      {block}
+      const aborted = firstSignal.aborted;
+      {unblock}
+      const callsBeforeLateResult = taskCalls;
+      finishFirst(true);
+      await flushMicrotasks();
+      const stillPendingAfterLateResult = _libPrewarmPromise === pending;
+      await advanceTo(now + 500);
+      await fireNextIdle(7);
+      const result = await pending;
+      console.log(JSON.stringify({{
+        result, aborted, callsBeforeLateResult, taskCalls,
+        stillPendingAfterLateResult,
+        timers: timers.size, idleCallbacks: idleCallbacks.size,
+        chatListeners: window.listenerCount('odysseus:chat-busy-change'),
+        visibilityListeners: document.listenerCount('visibilitychange'),
+      }}));
+    """)
+    assert out == {
+        "result": True,
+        "aborted": True,
+        "callsBeforeLateResult": 1,
+        "taskCalls": 2,
+        "stillPendingAfterLateResult": True,
+        "timers": 0,
+        "idleCallbacks": 0,
+        "chatListeners": 0,
+        "visibilityListeners": 0,
+    }
+
+
+def test_prewarm_skips_hidden_and_foreground_work():
+    guard = _function_source("_canRunEmailPrewarm")
+
+    assert "state._libOpen" in guard
+    assert "state._libLoading" in guard
+    assert "_libSearchInFlight" in guard
+    assert "document.visibilityState !== 'visible'" in guard
+    assert "!_isChatInteractionBusy()" in guard
+
+
+def test_prewarm_selects_only_last_used_or_default_account():
+    chooser = _function_source("_chooseEmailPrewarmAccountId")
+    prewarm = _function_source("_prewarmEmailViews")
+
+    assert "_rememberedEmailAccountId()" in chooser
+    assert "a.enabled !== false" in chooser
+    assert "a.is_default" in chooser
+    assert "enabled[0]" in chooser
+
+    assert "for (" not in prewarm
+    assert "orderedAccountIds" not in prewarm
+    assert "slice(0, 4)" not in prewarm
+    assert "/api/email/folders" not in prewarm
+    assert "/api/email/unread-state" not in prewarm
+    assert prewarm.count("/api/email/list") == 1
+
+
+def test_prewarm_account_chooser_rejects_disabled_or_empty_authoritative_inventory():
+    node = shutil.which("node")
+    if not node:
+        pytest.skip("node not on PATH")
+    chooser = _function_source("_chooseEmailPrewarmAccountId")
+    script = f"""
+      const state = {{ _libAccountId: 'disabled-current' }};
+      function _rememberedEmailAccountId() {{ return 'disabled-remembered'; }}
+      {chooser}
+      const onlyDisabled = _chooseEmailPrewarmAccountId([
+        {{ id: 'disabled-remembered', enabled: false, is_default: true }},
+        {{ id: 'disabled-current', enabled: false }},
+      ]);
+      const empty = _chooseEmailPrewarmAccountId([]);
+      const mixed = _chooseEmailPrewarmAccountId([
+        {{ id: 'disabled-remembered', enabled: false, is_default: true }},
+        {{ id: 'enabled-default', enabled: true, is_default: true }},
+      ]);
+      console.log(JSON.stringify({{ onlyDisabled, empty, mixed }}));
+    """
+    proc = subprocess.run(
+        [node, "--input-type=module"],
+        input=script,
+        capture_output=True,
+        text=True,
+        cwd=str(_REPO),
+        timeout=30,
+    )
+    assert proc.returncode == 0, proc.stderr
+    assert json.loads(proc.stdout.strip()) == {
+        "onlyDisabled": "",
+        "empty": "",
+        "mixed": "enabled-default",
+    }
+
+    ensure_accounts = _function_source("_ensureEmailAccountsForPrewarm")
+    assert "if (!accountId) return null;" in ensure_accounts
+    assert ensure_accounts.index("if (!accountId) return null;") < ensure_accounts.index("_publishActiveAccount();")
+
+
+def test_prewarm_is_bounded_to_the_interactive_initial_page_size():
+    text = _source()
+    prewarm = _function_source("_prewarmEmailViews")
+
+    assert "const _LIB_INITIAL_PAGE_SIZE = 100;" in text
+    assert "limit: _LIB_INITIAL_PAGE_SIZE" in prewarm
+    assert text.count("limit=${_LIB_INITIAL_PAGE_SIZE}&offset=${offsetAtStart}") == 2
+    assert "limit: 100" not in prewarm
+
+
+def test_open_cancels_scheduled_or_inflight_prewarm_first():
+    text = _source()
+    cancel = _function_source("_cancelEmailPrewarm")
+    open_library = _function_source("openEmailLibrary")
+
+    assert "clearTimeout(_libPrewarmDelayTimer)" in cancel
+    assert "window.cancelIdleCallback(_libPrewarmIdleHandle)" in cancel
+    assert "_libPrewarmAbortController?.abort()" in cancel
+    assert "_libPrewarmGeneration += 1" in cancel
+    assert open_library.index("_cancelEmailPrewarm();") < open_library.index("state._libOpen = true;")
+    assert "_loadEmailsWhenChatIdle" not in text
+    assert text.count("_loadEmails({ useCache: true });") >= 2
+
+
+def test_close_cancels_pending_prewarm_cleanup():
+    close_library = _function_source("closeEmailLibrary")
+
+    assert close_library.index("_cancelEmailPrewarm();") < close_library.index("state._libOpen = false;")
+
+
+def test_unread_warm_joins_the_same_idle_single_flight_gate():
+    unread_entry = _function_source("prewarmUnreadEmails")
+    unread_work = _function_source("_prewarmUnreadEmailsNow")
+
+    assert "_scheduleEmailPrewarm(" in unread_entry
+    assert "fetch(" not in unread_entry
+    assert "_ensureEmailAccountsForPrewarm({ signal, generation })" in unread_work
+    assert "signal" in unread_work
+    assert "Math.min(20" in unread_work

tests/test_email_oauth_connect_smtp_security.py

@@ -0,0 +1,15 @@
+"""Regression coverage for SMTP security saved before Google OAuth."""
+
+from pathlib import Path
+
+
+_REPO = Path(__file__).resolve().parents[1]
+
+
+def test_email_tab_oauth_connect_persists_selected_smtp_security():
+    source = (_REPO / "static" / "js" / "settings.js").read_text(encoding="utf-8")
+    start = source.index("el('eaf-oauth-btn').addEventListener")
+    handler_body = source[start:source.index("if (!body.name)", start)]
+
+    assert "smtp_security: el('eaf-smtp-security').value" in handler_body
+    assert "display_name: el('eaf-display-name').value.trim()" in handler_body

tests/test_email_oauth_settings_redirect.py

@@ -0,0 +1,19 @@
+"""Regression coverage for the settings UI after Google OAuth redirects."""
+
+from pathlib import Path
+
+
+_REPO = Path(__file__).resolve().parents[1]
+
+
+def test_oauth_redirect_uses_the_module_local_settings_api():
+    source = (_REPO / "static" / "js" / "settings.js").read_text(encoding="utf-8")
+    handler = source[
+        source.index("(function _handleOauthRedirect"):
+        source.index("const settingsModule =")
+    ]
+
+    assert "open('integrations');" in handler
+    assert "window.settingsModule" not in handler
+    assert "window.__odysseusAppStarted" not in handler
+    assert "document.addEventListener('DOMContentLoaded', _showResult, { once: true })" in handler

tests/test_email_open_dedup_js.py

@@ -0,0 +1,231 @@
+"""Focused browser-side regression coverage for authoritative email opens."""
+
+import json
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+
+_REPO = Path(__file__).resolve().parent.parent
+_INBOX_JS = _REPO / "static" / "js" / "emailInbox.js"
+_LIBRARY_JS = _REPO / "static" / "js" / "emailLibrary.js"
+_HAS_NODE = shutil.which("node") is not None
+
+
+def _extract_between(source: str, signature: str, next_marker: str) -> str:
+    start = source.index(signature)
+    end = source.index(next_marker, start)
+    return source[start:end].rstrip()
+
+
+def test_library_unread_preview_has_one_authoritative_request_and_rollback():
+    source = _LIBRARY_JS.read_text(encoding="utf-8")
+    function = _extract_between(source, "async function _toggleCardPreview", "\n/**\n * Wrap a probable signature block")
+
+    assert function.count("/api/email/read/") == 1
+    assert "/api/email/mark-read/" not in function
+    assert "&mark_seen=true" in function
+    assert "_syncEmailReadState(uidAtStart, true, readContext)" in function
+    assert "_syncEmailReadState(uidAtStart, false, readContext)" in function
+    assert "openGeneration === _emailCardOpenSeq" in function
+    assert "_emailReadMutations.get(readContextKey)?.generation !== readMutation.generation" in function
+    assert "authoritativeReadSucceeded = true;" in function
+    assert "if (!authoritativeReadSucceeded) restoreUnreadState();" in function
+    assert "if (!isCurrentOpen()) return" in function
+
+
+@pytest.mark.skipif(not _HAS_NODE, reason="node binary not on PATH")
+def test_library_authoritative_success_defeats_newer_rollback_in_either_order():
+    source = _LIBRARY_JS.read_text(encoding="utf-8")
+    function = _extract_between(source, "async function _toggleCardPreview", "\n/**\n * Wrap a probable signature block")
+    settlements = _extract_between(
+        function,
+        "  const restoreUnreadState = () => {",
+        "\n\n  // Collapse any other expanded card",
+    )
+
+    harness = f"""
+const _emailReadMutations = new Map();
+const readContextKey = 'same-mailbox-message';
+const uidAtStart = '1';
+const readContext = {{ accountId: 'acct-a', folder: 'INBOX', uid: '1' }};
+const readUpdates = [];
+function _syncEmailReadState(uid, isRead, context) {{
+  readUpdates.push({{ uid, isRead, context }});
+}}
+function createSettlers(readMutation) {{
+{settlements}
+  return {{ restoreUnreadState, commitReadState }};
+}}
+function runRace(successFirst) {{
+  _emailReadMutations.clear();
+  readUpdates.length = 0;
+  const mutationA = {{ generation: 1, rollbackUnread: true }};
+  _emailReadMutations.set(readContextKey, mutationA);
+  const settlersA = createSettlers(mutationA);
+  const mutationB = {{ generation: 2, rollbackUnread: true }};
+  _emailReadMutations.set(readContextKey, mutationB);
+  const settlersB = createSettlers(mutationB);
+  if (successFirst) {{
+    settlersA.commitReadState();
+    settlersB.restoreUnreadState();
+  }} else {{
+    settlersB.restoreUnreadState();
+    settlersA.commitReadState();
+  }}
+  return {{
+    hasMutation: _emailReadMutations.has(readContextKey),
+    readUpdates: readUpdates.map(update => update.isRead),
+  }};
+}}
+console.log(JSON.stringify({{
+  successFirst: runRace(true),
+  failureFirst: runRace(false),
+}}));
+"""
+    proc = subprocess.run(
+        ["node", "--input-type=module"],
+        input=harness,
+        capture_output=True,
+        text=True,
+        cwd=str(_REPO),
+        timeout=30,
+    )
+    assert proc.returncode == 0, f"node failed: {proc.stderr}\n---\n{harness}"
+    assert json.loads(proc.stdout.strip()) == {
+        "successFirst": {"hasMutation": False, "readUpdates": [True]},
+        "failureFirst": {"hasMutation": False, "readUpdates": [False, True]},
+    }
+
+
+def test_library_reply_open_carries_immutable_mailbox_context():
+    library_source = _LIBRARY_JS.read_text(encoding="utf-8")
+    inbox_source = _INBOX_JS.read_text(encoding="utf-8")
+
+    assert "const mailboxGeneration = _emailMailboxGeneration;" in library_source
+    assert "messageFolder = String(options.email?.folder || libraryFolder)" in library_source
+    assert "return onEmailClick({ ...options, mailboxContext });" in library_source
+    assert "mailboxContext?.messageFolder || _currentFolder" in inbox_source
+    assert "mailboxContextIsCurrent()" in inbox_source
+    assert "if (!isCurrentOpen()) return;\n        let activeSid = await _createEmailChat" in inbox_source
+
+
+@pytest.mark.skipif(not _HAS_NODE, reason="node binary not on PATH")
+def test_inbox_late_read_response_cannot_apply_after_newer_open():
+    source = _INBOX_JS.read_text(encoding="utf-8")
+    function = _extract_between(source, "async function _openEmail", "\nfunction _showEmailMenu")
+    assert "let _openEmailRequestSeq = 0;" in source
+
+    harness = f"""
+const realLog = console.log;
+console.error = () => {{}};
+const API_BASE = 'https://odysseus.invalid';
+const window = {{ __odysseusActiveEmailAccount: 'acct-a' }};
+let _currentFolder = 'INBOX';
+const _acct = () => '&account_id=acct-a';
+let _openEmailRequestSeq = 0;
+let _docModule = null;
+const spinnerModule = {{ createWhirlpool() {{ throw new Error('spinner should not run'); }} }};
+const sessionModule = null;
+let firstResolve;
+const calls = [];
+async function fetch(url) {{
+  calls.push(String(url));
+  if (calls.length === 1) {{
+    return await new Promise((resolve) => {{
+      firstResolve = () => resolve({{ json: async () => ({{ uid: '1', subject: 'old' }}) }});
+    }});
+  }}
+  return {{ json: async () => ({{ error: 'newer open completed test' }}) }};
+}}
+{function}
+const oldEmail = {{ uid: '1', is_read: false }};
+const newerEmail = {{ uid: '2', is_read: false }};
+const first = _openEmail(oldEmail, null);
+await Promise.resolve();
+const second = _openEmail(newerEmail, null);
+await second;
+firstResolve();
+await first;
+realLog(JSON.stringify({{ calls, oldRead: oldEmail.is_read, newerRead: newerEmail.is_read }}));
+"""
+    proc = subprocess.run(
+        ["node", "--input-type=module"],
+        input=harness,
+        capture_output=True,
+        text=True,
+        cwd=str(_REPO),
+        timeout=30,
+    )
+    assert proc.returncode == 0, f"node failed: {proc.stderr}\n---\n{harness}"
+    result = json.loads(proc.stdout.strip())
+    assert len(result["calls"]) == 2
+    assert all("mark_seen=true" in url for url in result["calls"])
+    assert result["oldRead"] is False
+    assert result["newerRead"] is False
+
+
+@pytest.mark.skipif(not _HAS_NODE, reason="node binary not on PATH")
+@pytest.mark.parametrize("context_change", ["account", "folder", "library"])
+def test_inbox_late_read_response_cannot_apply_after_mailbox_switch(context_change):
+    source = _INBOX_JS.read_text(encoding="utf-8")
+    function = _extract_between(source, "async function _openEmail", "\nfunction _showEmailMenu")
+
+    changes = {
+        "account": "window.__odysseusActiveEmailAccount = 'acct-b';",
+        "folder": "_currentFolder = 'Archive';",
+        "library": "libraryCurrent = false;",
+    }
+    change = changes[context_change]
+    open_call = (
+        "_openEmail(email, null, null, 'reply', '', '', mailboxContext)"
+        if context_change == "library"
+        else "_openEmail(email, null)"
+    )
+    harness = f"""
+const realLog = console.log;
+console.error = () => {{}};
+const API_BASE = 'https://odysseus.invalid';
+const window = {{ __odysseusActiveEmailAccount: 'acct-a' }};
+let _currentFolder = 'INBOX';
+const _acct = () => '&account_id=acct-a';
+let _openEmailRequestSeq = 0;
+let libraryCurrent = true;
+const mailboxContext = {{
+  accountId: 'acct-a',
+  messageFolder: 'Archive',
+  isCurrent: () => libraryCurrent,
+}};
+let createCalls = 0;
+let _docModule = {{}};
+async function _createEmailChat() {{ createCalls += 1; return 'stale-session'; }}
+const spinnerModule = {{ createWhirlpool() {{ throw new Error('spinner should not run'); }} }};
+const sessionModule = null;
+let resolveRead;
+async function fetch() {{
+  return await new Promise((resolve) => {{
+    resolveRead = () => resolve({{ json: async () => ({{ uid: '1', subject: 'old' }}) }});
+  }});
+}}
+{function}
+const email = {{ uid: '1', is_read: false }};
+const pending = {open_call};
+await Promise.resolve();
+{change}
+resolveRead();
+await pending;
+realLog(JSON.stringify({{ createCalls, isRead: email.is_read }}));
+"""
+    proc = subprocess.run(
+        ["node", "--input-type=module"],
+        input=harness,
+        capture_output=True,
+        text=True,
+        cwd=str(_REPO),
+        timeout=30,
+    )
+    assert proc.returncode == 0, f"node failed: {proc.stderr}\n---\n{harness}"
+    result = json.loads(proc.stdout.strip())
+    assert result == {"createCalls": 0, "isRead": False}

tests/test_email_read_mark_seen.py

@@ -0,0 +1,278 @@
+import asyncio
+from contextlib import contextmanager
+
+import pytest
+
+
+RAW_EMAIL = (
+    b"From: Sender <sender@example.com>\r\n"
+    b"To: Alice <alice@example.com>\r\n"
+    b"Subject: Single authoritative open\r\n"
+    b"Message-ID: <single-open@example.com>\r\n"
+    b"Date: Tue, 04 Aug 2026 12:00:00 +0000\r\n"
+    b"Content-Type: text/plain; charset=utf-8\r\n"
+    b"\r\n"
+    b"Body"
+)
+
+
+def _route_endpoint(router, path: str, method: str):
+    method = method.upper()
+    for route in router.routes:
+        if route.path == path and method in getattr(route, "methods", set()):
+            return route.endpoint
+    raise AssertionError(f"route not found: {method} {path}")
+
+
+class FakeImap:
+    def __init__(self, store_status="OK", readonly_mailbox=False):
+        self.store_status = store_status
+        # Shared archives and some provider folders reject a read-write SELECT.
+        self.readonly_mailbox = readonly_mailbox
+        self.selects = []
+        self.commands = []
+
+    def select(self, mailbox, readonly=False):
+        self.selects.append((mailbox, readonly))
+        if self.readonly_mailbox and not readonly:
+            raise OSError("[READ-ONLY] Mailbox is read-only")
+        return "OK", [b"1"]
+
+    def uid(self, command, uid, *args):
+        self.commands.append((command, uid, *args))
+        if command == "FETCH":
+            header, body = RAW_EMAIL.split(b"\r\n\r\n", 1)
+            return "OK", [
+                (b"1 (UID 42 BODY[HEADER])", header + b"\r\n\r\n"),
+                (b"1 (UID 42 BODY[TEXT]<0>)", body),
+            ]
+        if command == "STORE":
+            # RFC 3501 STORE takes a parenthesized flag-list. GreenMail rejects
+            # the formerly emitted bare ``\Seen`` atom with BAD, so keep the
+            # fake strict enough to catch that provider-compatibility failure.
+            if args != ("+FLAGS", "(\\Seen)"):
+                return "BAD", [b"Expected:'(' found:'\\'"]
+            return self.store_status, []
+        raise AssertionError(f"unexpected IMAP command: {command}")
+
+
+def _install_fakes(monkeypatch, tmp_path, *, store_status="OK", readonly_mailbox=False):
+    import routes.email_helpers as email_helpers
+    import routes.email_routes as email_routes
+
+    db_path = tmp_path / "email.db"
+    monkeypatch.setattr(email_helpers, "SCHEDULED_DB", db_path)
+    monkeypatch.setattr(email_routes, "SCHEDULED_DB", db_path)
+    email_helpers._init_scheduled_db()
+
+    connections = []
+    indexed_updates = []
+
+    @contextmanager
+    def fake_imap(account_id=None, owner=""):
+        conn = FakeImap(store_status=store_status, readonly_mailbox=readonly_mailbox)
+        connections.append(conn)
+        yield conn
+
+    monkeypatch.setattr(email_routes, "_start_poller", lambda: None)
+    monkeypatch.setattr(email_routes, "_imap", fake_imap)
+    monkeypatch.setattr(email_routes, "_email_preview_cache_get", lambda *_args, **_kwargs: None)
+    monkeypatch.setattr(email_routes, "_email_preview_cache_put", lambda *_args, **_kwargs: None)
+    monkeypatch.setattr(email_routes, "_email_attachment_meta_cache_get", lambda *_args, **_kwargs: None)
+    monkeypatch.setattr(email_routes, "_email_attachment_meta_cache_put", lambda *_args, **_kwargs: None)
+    monkeypatch.setattr(
+        email_routes,
+        "_email_index_update_flags",
+        lambda *args, **_kwargs: indexed_updates.append(args),
+    )
+    return email_routes, connections, indexed_updates
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("mark_seen", [True, False])
+async def test_read_email_seen_contract_uses_one_imap_connection(monkeypatch, tmp_path, mark_seen):
+    email_routes, connections, indexed_updates = _install_fakes(monkeypatch, tmp_path)
+    router = email_routes.setup_email_routes()
+    read_email = _route_endpoint(router, "/api/email/read/{uid}", "GET")
+
+    result = await read_email(
+        "42",
+        folder="INBOX",
+        account_id="acct-a",
+        mark_seen=mark_seen,
+        full=False,
+        owner="alice",
+    )
+
+    assert result["uid"] == "42"
+    assert len(connections) == 1
+    conn = connections[0]
+    assert conn.selects == [(conn.selects[0][0], not mark_seen)]
+    assert [command[0] for command in conn.commands] == (
+        ["FETCH", "STORE"] if mark_seen else ["FETCH"]
+    )
+    assert "BODY.PEEK[HEADER]" in conn.commands[0][2]
+    if mark_seen:
+        assert conn.commands[1][2:] == ("+FLAGS", "(\\Seen)")
+        assert indexed_updates == [("alice", "acct-a", "INBOX", "42", "\\Seen", True)]
+    else:
+        assert indexed_updates == []
+
+
+@pytest.mark.asyncio
+async def test_cached_read_awaits_one_seen_store_without_refetch(monkeypatch, tmp_path):
+    email_routes, connections, indexed_updates = _install_fakes(monkeypatch, tmp_path)
+    router = email_routes.setup_email_routes()
+    read_email = _route_endpoint(router, "/api/email/read/{uid}", "GET")
+
+    first = await read_email(
+        "42", folder="INBOX", account_id="acct-a", mark_seen=False, full=False, owner="alice"
+    )
+    monkeypatch.setattr(
+        asyncio,
+        "create_task",
+        lambda *_args, **_kwargs: (_ for _ in ()).throw(
+            AssertionError("cached mark-seen must be awaited, not scheduled")
+        ),
+    )
+    second = await read_email(
+        "42", folder="INBOX", account_id="acct-a", mark_seen=True, full=False, owner="alice"
+    )
+
+    assert first["message_id"] == second["message_id"]
+    assert len(connections) == 2
+    assert [command[0] for command in connections[0].commands] == ["FETCH"]
+    assert [command[0] for command in connections[1].commands] == ["STORE"]
+    assert connections[1].commands[0][2:] == ("+FLAGS", "(\\Seen)")
+    assert connections[1].selects[0][1] is False
+    assert indexed_updates == [("alice", "acct-a", "INBOX", "42", "\\Seen", True)]
+
+
+@pytest.mark.asyncio
+async def test_seen_store_failure_returns_the_body_and_reports_the_failure(monkeypatch, tmp_path):
+    """A failed STORE must not cost the reader the message.
+
+    The body was fetched successfully before the flag update was attempted, so
+    the response stays a normal read and carries `mark_seen_failed` for the
+    client to roll its optimistic unread marker back.
+    """
+    email_routes, connections, indexed_updates = _install_fakes(
+        monkeypatch, tmp_path, store_status="NO"
+    )
+    router = email_routes.setup_email_routes()
+    read_email = _route_endpoint(router, "/api/email/read/{uid}", "GET")
+
+    result = await read_email(
+        "42", folder="INBOX", account_id="acct-a", mark_seen=True, full=False, owner="alice"
+    )
+
+    assert "error" not in result
+    assert result["uid"] == "42"
+    assert result["mark_seen_failed"] is True
+    assert len(connections) == 1
+    assert [command[0] for command in connections[0].commands] == ["FETCH", "STORE"]
+    # The local index must not claim a transition the provider rejected.
+    assert indexed_updates == []
+
+
+@pytest.mark.asyncio
+async def test_read_only_mailbox_serves_the_message_without_marking_seen(monkeypatch, tmp_path):
+    """A mailbox that refuses a read-write SELECT is still readable.
+
+    Opening the message is the user's actual goal; the \\Seen transition is a
+    side effect of it. A folder that cannot accept flag changes must therefore
+    fall back to a read-only selection rather than failing the open.
+    """
+    email_routes, connections, indexed_updates = _install_fakes(
+        monkeypatch, tmp_path, readonly_mailbox=True
+    )
+    router = email_routes.setup_email_routes()
+    read_email = _route_endpoint(router, "/api/email/read/{uid}", "GET")
+
+    result = await read_email(
+        "42", folder="Archive", account_id="acct-a", mark_seen=True, full=False, owner="alice"
+    )
+
+    assert "error" not in result
+    assert result["uid"] == "42"
+    assert result["mark_seen_failed"] is True
+    # Read-write attempt first, then the read-only retry on the same connection.
+    assert [readonly for _mailbox, readonly in connections[0].selects] == [False, True]
+    # No STORE is attempted once the mailbox is known to be read-only.
+    assert [command[0] for command in connections[0].commands] == ["FETCH"]
+    assert indexed_updates == []
+
+
+@pytest.mark.asyncio
+async def test_failed_seen_state_is_not_replayed_from_cache(monkeypatch, tmp_path):
+    """`mark_seen_failed` describes one request, not the stored message.
+
+    A second read that does not ask to mark seen must come back clean, or every
+    later reader would inherit a STORE failure it never issued.
+    """
+    email_routes, connections, _ = _install_fakes(monkeypatch, tmp_path, store_status="NO")
+    router = email_routes.setup_email_routes()
+    read_email = _route_endpoint(router, "/api/email/read/{uid}", "GET")
+
+    failed = await read_email(
+        "42", folder="INBOX", account_id="acct-a", mark_seen=True, full=False, owner="alice"
+    )
+    replayed = await read_email(
+        "42", folder="INBOX", account_id="acct-a", mark_seen=False, full=False, owner="alice"
+    )
+
+    assert failed["mark_seen_failed"] is True
+    assert replayed.get("mark_seen_failed", False) is False
+    assert replayed["uid"] == "42"
+
+
+@pytest.mark.asyncio
+async def test_unparseable_read_does_not_mark_seen(monkeypatch, tmp_path):
+    email_routes, connections, indexed_updates = _install_fakes(monkeypatch, tmp_path)
+    monkeypatch.setattr(
+        email_routes.email_mod,
+        "message_from_bytes",
+        lambda *_args, **_kwargs: (_ for _ in ()).throw(ValueError("malformed message")),
+    )
+    router = email_routes.setup_email_routes()
+    read_email = _route_endpoint(router, "/api/email/read/{uid}", "GET")
+
+    result = await read_email(
+        "42", folder="INBOX", account_id="acct-a", mark_seen=True, full=False, owner="alice"
+    )
+
+    assert result == {"error": "Mail operation failed"}
+    assert len(connections) == 1
+    assert [command[0] for command in connections[0].commands] == ["FETCH"]
+    assert indexed_updates == []
+
+
+@pytest.mark.asyncio
+async def test_cached_seen_store_failure_returns_the_cached_body(monkeypatch, tmp_path):
+    """A cache hit already holds a complete message; a failed STORE cannot take it away.
+
+    This is the path where withholding the body would be least defensible — the
+    response is served from memory and needed no network at all.
+    """
+    email_routes, connections, indexed_updates = _install_fakes(
+        monkeypatch, tmp_path, store_status="NO"
+    )
+    router = email_routes.setup_email_routes()
+    read_email = _route_endpoint(router, "/api/email/read/{uid}", "GET")
+
+    first = await read_email(
+        "42", folder="INBOX", account_id="acct-a", mark_seen=False, full=False, owner="alice"
+    )
+    second = await read_email(
+        "42", folder="INBOX", account_id="acct-a", mark_seen=True, full=False, owner="alice"
+    )
+
+    assert first["uid"] == "42"
+    assert "error" not in second
+    assert second["uid"] == "42"
+    assert second["body"] == first["body"]
+    assert second["mark_seen_failed"] is True
+    assert len(connections) == 2
+    assert [command[0] for command in connections[0].commands] == ["FETCH"]
+    assert [command[0] for command in connections[1].commands] == ["STORE"]
+    assert indexed_updates == []

tests/test_email_summary_error_ui_js.py

@@ -0,0 +1,52 @@
+import json
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+
+_REPO = Path(__file__).resolve().parent.parent
+_UTILS = (_REPO / "static" / "js" / "emailLibrary" / "utils.js").as_posix()
+_HAS_NODE = shutil.which("node") is not None
+
+pytestmark = pytest.mark.skipif(not _HAS_NODE, reason="node binary not on PATH")
+
+
+def test_email_summary_renderer_ignores_untrusted_provider_error_text():
+    secret = (
+        "endpoint=https://private.example.internal/v1 provider=ollama "
+        "model=private-model response_body=private-response "
+        "Authorization: Bearer token-secret-value"
+    )
+    script = f"""
+      import {{ _renderEmailSummaryError }} from '{_UTILS}';
+      const host = {{
+        ownerDocument: {{
+          createElement() {{ return {{ style: {{}}, textContent: '' }}; }},
+        }},
+        replaceChildren(node) {{ this.child = node; }},
+      }};
+      _renderEmailSummaryError(host, {{
+        error_code: 'email_summary_unavailable',
+        error: {json.dumps(secret)},
+      }});
+      console.log(JSON.stringify({{
+        text: host.child.textContent,
+        color: host.child.style.color,
+      }}));
+    """
+
+    proc = subprocess.run(
+        ["node", "--input-type=module"],
+        input=script,
+        capture_output=True,
+        text=True,
+        cwd=str(_REPO),
+        timeout=30,
+    )
+
+    assert proc.returncode == 0, proc.stderr
+    rendered = json.loads(proc.stdout)
+    assert rendered == {"text": "Failed to summarize", "color": "var(--red)"}
+    assert secret not in proc.stdout

tests/test_email_summary_llm.py

@@ -0,0 +1,406 @@
+import asyncio
+import json
+import logging
+import os
+import sqlite3
+import sys
+import tempfile
+from pathlib import Path
+
+import pytest
+
+
+_TMP_DATA = Path(tempfile.mkdtemp(prefix="odysseus-email-summary-"))
+os.environ.setdefault("DATA_DIR", str(_TMP_DATA))
+os.environ.setdefault("DATABASE_URL", f"sqlite:///{_TMP_DATA / 'app.db'}")
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+if str(PROJECT_ROOT) not in sys.path:
+    sys.path.insert(0, str(PROJECT_ROOT))
+
+
+def _route_endpoint(router, path: str, method: str):
+    method = method.upper()
+    for route in router.routes:
+        if route.path == path and method in getattr(route, "methods", set()):
+            return route.endpoint
+    raise AssertionError(f"route not found: {method} {path}")
+
+
+@pytest.mark.asyncio
+async def test_generate_email_summary_uses_shared_llm_adapter(monkeypatch):
+    import routes.email_helpers as email_helpers
+    import src.llm_core as llm_core
+
+    calls = {}
+
+    async def fake_llm_call_async(url, model, messages, **kwargs):
+        calls["url"] = url
+        calls["model"] = model
+        calls["messages"] = messages
+        calls["kwargs"] = kwargs
+        return "thinking before marker\n<<<SUMMARY>>>\n- Pay the invoice by Friday.\n<<<END>>>"
+
+    monkeypatch.setattr(llm_core, "llm_call_async", fake_llm_call_async)
+
+    summary = await email_helpers._generate_email_summary(
+        url="https://chatgpt.com/backend-api/codex/responses",
+        model="gpt-5.5",
+        sender="Billing <billing@example.com>",
+        subject="Invoice due",
+        body_for_llm="Please pay invoice 123 by Friday.",
+        headers={"Authorization": "Bearer test"},
+        max_tokens=1234,
+        timeout=45,
+    )
+
+    assert summary == "- Pay the invoice by Friday."
+    assert calls["url"] == "https://chatgpt.com/backend-api/codex/responses"
+    assert calls["model"] == "gpt-5.5"
+    assert calls["kwargs"]["headers"] == {"Authorization": "Bearer test"}
+    assert calls["kwargs"]["temperature"] == 0.3
+    assert calls["kwargs"]["max_tokens"] == 1234
+    assert calls["kwargs"]["timeout"] == 45
+    assert calls["kwargs"]["workload"] == "foreground"
+    assert calls["messages"][0]["role"] == "system"
+    assert calls["messages"][1]["role"] == "user"
+
+
+@pytest.mark.asyncio
+async def test_scheduled_email_summary_uses_background_fallback_chain(monkeypatch):
+    import routes.email_helpers as email_helpers
+    import src.llm_core as llm_core
+    import src.task_endpoint as task_endpoint
+
+    candidates = [
+        ("http://primary.invalid/v1", "primary-model", {"X-Candidate": "primary"}),
+        ("http://fallback.invalid/v1", "fallback-model", {"X-Candidate": "fallback"}),
+    ]
+    resolve_calls = []
+    wait_calls = []
+    llm_calls = []
+
+    def fake_resolve_task_candidates(**kwargs):
+        resolve_calls.append(kwargs)
+        return candidates
+
+    async def fake_wait_for_interactive_quiet(label):
+        wait_calls.append(label)
+        return False
+
+    async def fake_llm_call_async(url, model, messages, **kwargs):
+        llm_calls.append((url, model, messages, kwargs))
+        if model == "primary-model":
+            raise RuntimeError("primary unavailable")
+        return "<<<SUMMARY>>>\n- Used the fallback model.\n<<<END>>>"
+
+    monkeypatch.setattr(task_endpoint, "resolve_task_candidates", fake_resolve_task_candidates)
+    monkeypatch.setattr(task_endpoint, "wait_for_interactive_quiet", fake_wait_for_interactive_quiet)
+    monkeypatch.setattr(llm_core, "llm_call_async", fake_llm_call_async)
+
+    summary = await email_helpers._generate_scheduled_email_summary(
+        url="http://caller-fallback.invalid/v1",
+        model="caller-fallback-model",
+        sender="Sender <sender@example.com>",
+        subject="Scheduled subject",
+        body_for_llm="Please summarize this scheduled email.",
+        headers={"Authorization": "Bearer test"},
+        owner="alice",
+        max_tokens=321,
+        timeout=54,
+    )
+
+    assert summary == "- Used the fallback model."
+    assert resolve_calls == [{
+        "fallback_url": "http://caller-fallback.invalid/v1",
+        "fallback_model": "caller-fallback-model",
+        "fallback_headers": {"Authorization": "Bearer test"},
+        "owner": "alice",
+    }]
+    assert wait_calls == ["background task LLM"]
+    assert [call[1] for call in llm_calls] == ["primary-model", "fallback-model"]
+    assert all(call[3]["workload"] == "background" for call in llm_calls)
+    assert all(call[3]["max_tokens"] == 321 for call in llm_calls)
+    assert all(call[3]["timeout"] == 54 for call in llm_calls)
+
+
+@pytest.mark.asyncio
+async def test_scheduled_local_summary_is_preempted_by_foreground_call(monkeypatch):
+    import routes.email_helpers as email_helpers
+    import src.llm_core as llm_core
+    import src.task_endpoint as task_endpoint
+
+    local_url = "http://127.0.0.1:11434/v1/chat/completions"
+    background_started = asyncio.Event()
+    never_release = asyncio.Event()
+    observed_workloads = []
+
+    monkeypatch.setenv("ODYSSEUS_LOCAL_MODEL_GATE", "true")
+    monkeypatch.setenv("BACKGROUND_TASK_FOREGROUND_GATE", "false")
+    monkeypatch.setattr(llm_core, "_LOCAL_MODEL_LOCK", asyncio.Lock())
+    monkeypatch.setattr(llm_core, "_LOCAL_MODEL_CURRENT", {})
+    monkeypatch.setattr(llm_core, "_LOCAL_MODEL_WAITING_FOREGROUND", 0)
+    monkeypatch.setattr(
+        task_endpoint,
+        "resolve_task_candidates",
+        lambda **_kwargs: [(local_url, "scheduled-model", {})],
+    )
+
+    async def fake_wait_for_interactive_quiet(_label):
+        return False
+
+    async def gated_llm_call(url, model, messages, **kwargs):
+        assert messages
+        workload = kwargs.get("workload")
+        observed_workloads.append(workload)
+        async with llm_core._local_model_slot(url, model, workload=workload):
+            background_started.set()
+            await never_release.wait()
+        return "unreachable"
+
+    monkeypatch.setattr(task_endpoint, "wait_for_interactive_quiet", fake_wait_for_interactive_quiet)
+    monkeypatch.setattr(llm_core, "llm_call_async", gated_llm_call)
+
+    background_task = asyncio.create_task(email_helpers._generate_scheduled_email_summary(
+        url=local_url,
+        model="scheduled-model",
+        sender="Sender",
+        subject="Scheduled",
+        body_for_llm="Scheduled body",
+        owner="alice",
+    ))
+    foreground_task = None
+    try:
+        await asyncio.wait_for(background_started.wait(), timeout=1)
+
+        async def run_foreground():
+            async with llm_core._local_model_slot(
+                local_url,
+                "interactive-model",
+                workload="foreground",
+            ):
+                return True
+
+        foreground_task = asyncio.create_task(run_foreground())
+        with pytest.raises(asyncio.CancelledError):
+            await asyncio.wait_for(background_task, timeout=1)
+        assert await asyncio.wait_for(foreground_task, timeout=1) is True
+        assert observed_workloads == ["background"]
+    finally:
+        for task in (background_task, foreground_task):
+            if task is not None and not task.done():
+                task.cancel()
+
+
+@pytest.mark.asyncio
+async def test_manual_email_summary_uses_shared_helper_and_caches(tmp_path, monkeypatch):
+    import routes.email_helpers as email_helpers
+    import routes.email_routes as email_routes
+    import src.endpoint_resolver as endpoint_resolver
+
+    db_path = tmp_path / "scheduled_emails.db"
+    monkeypatch.setattr(email_helpers, "SCHEDULED_DB", db_path)
+    monkeypatch.setattr(email_routes, "SCHEDULED_DB", db_path)
+    email_helpers._init_scheduled_db()
+
+    resolve_calls = []
+
+    def fake_resolve_endpoint(kind, owner=None):
+        resolve_calls.append((kind, owner))
+        assert kind == "utility"
+        assert owner == "alice"
+        return (
+            "https://chatgpt.com/backend-api/codex/responses",
+            "gpt-5.5",
+            {"Authorization": "Bearer test"},
+        )
+
+    helper_calls = {}
+
+    async def fake_generate_email_summary(**kwargs):
+        helper_calls.update(kwargs)
+        return "- Manual summary"
+
+    monkeypatch.setattr(endpoint_resolver, "resolve_endpoint", fake_resolve_endpoint)
+    monkeypatch.setattr(email_routes, "_generate_email_summary", fake_generate_email_summary)
+
+    router = email_routes.setup_email_routes()
+    summarize = _route_endpoint(router, "/api/email/summarize", "POST")
+
+    result = await summarize(
+        {
+            "body": "This is a long enough email body for manual summary.",
+            "subject": "Manual subject",
+            "from": "Sender <sender@example.com>",
+            "message_id": "<manual@example.com>",
+            "folder": "INBOX",
+        },
+        owner="alice",
+    )
+
+    assert result == {
+        "success": True,
+        "summary": "- Manual summary",
+        "model_used": "gpt-5.5",
+    }
+    assert resolve_calls == [("utility", "alice")]
+    assert helper_calls["url"] == "https://chatgpt.com/backend-api/codex/responses"
+    assert helper_calls["model"] == "gpt-5.5"
+    assert helper_calls["headers"]["Authorization"] == "Bearer test"
+    assert helper_calls["headers"]["Content-Type"] == "application/json"
+
+    conn = sqlite3.connect(db_path)
+    try:
+        row = conn.execute(
+            "SELECT owner, summary, model_used FROM email_summaries WHERE message_id=?",
+            ("<manual@example.com>",),
+        ).fetchone()
+    finally:
+        conn.close()
+    assert row == ("alice", "- Manual summary", "gpt-5.5")
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("exception_kind", ["http", "runtime"])
+async def test_manual_email_summary_never_exposes_provider_exception(
+    monkeypatch,
+    caplog,
+    exception_kind,
+):
+    from fastapi import HTTPException
+    import routes.email_routes as email_routes
+    import src.endpoint_resolver as endpoint_resolver
+
+    secret_detail = (
+        "endpoint=https://private.example.internal/v1 provider=ollama "
+        "model=private-model response_body=private-response "
+        "Authorization: Bearer token-secret-value"
+    )
+
+    def fake_resolve_endpoint(kind, owner=None):
+        assert kind == "utility"
+        assert owner == "alice"
+        return (
+            "https://private.example.internal/v1",
+            "private-model",
+            {"Authorization": "Bearer token-secret-value"},
+        )
+
+    async def fail_summary(**_kwargs):
+        if exception_kind == "http":
+            raise HTTPException(status_code=502, detail=secret_detail)
+        raise RuntimeError(secret_detail)
+
+    monkeypatch.setattr(endpoint_resolver, "resolve_endpoint", fake_resolve_endpoint)
+    monkeypatch.setattr(email_routes, "_generate_email_summary", fail_summary)
+    caplog.set_level(logging.WARNING, logger=email_routes.__name__)
+
+    router = email_routes.setup_email_routes()
+    summarize = _route_endpoint(router, "/api/email/summarize", "POST")
+    result = await summarize(
+        {
+            "body": "This email body is long enough to summarize.",
+            "subject": "Sensitive provider failure",
+            "from": "Sender <sender@example.com>",
+        },
+        owner="alice",
+    )
+
+    assert result == {
+        "success": False,
+        "error": "Failed to summarize",
+        "error_code": "email_summary_unavailable",
+    }
+    exposed = json.dumps(result) + caplog.text
+    for marker in (
+        "private.example.internal",
+        "ollama",
+        "private-model",
+        "private-response",
+        "token-secret-value",
+    ):
+        assert marker not in exposed
+    assert f"type={'HTTPException' if exception_kind == 'http' else 'RuntimeError'}" in caplog.text
+
+
+@pytest.mark.asyncio
+async def test_scheduled_email_summary_uses_shared_helper_and_caches(tmp_path, monkeypatch):
+    import routes.email_helpers as email_helpers
+    import routes.email_pollers as email_pollers
+
+    db_path = tmp_path / "scheduled_emails.db"
+    monkeypatch.setattr(email_helpers, "SCHEDULED_DB", db_path)
+    monkeypatch.setattr(email_pollers, "SCHEDULED_DB", db_path)
+    email_helpers._init_scheduled_db()
+
+    raw_email = (
+        b"From: Sender <sender@example.com>\r\n"
+        b"To: Alice <alice@example.com>\r\n"
+        b"Subject: Scheduled subject\r\n"
+        b"Message-ID: <scheduled@example.com>\r\n"
+        b"Date: Tue, 01 Jan 2026 12:00:00 +0000\r\n"
+        b"Content-Type: text/plain; charset=utf-8\r\n"
+        b"\r\n"
+        + (b"Please review this scheduled summary email. " * 8)
+    )
+
+    class FakeImap:
+        def __init__(self):
+            self.logout_calls = 0
+
+        def select(self, _folder, readonly=True):
+            return "OK", []
+
+        def uid(self, command, *args):
+            if command == "SEARCH":
+                return "OK", [b"1"]
+            if command == "FETCH":
+                return "OK", [(b"1 (RFC822)", raw_email)]
+            raise AssertionError(f"unexpected uid command: {command!r} {args!r}")
+
+        def logout(self):
+            self.logout_calls += 1
+
+    fake_conn = FakeImap()
+
+    def fake_resolve_task_candidates(owner=None):
+        assert owner == "alice"
+        return [(
+            "https://chatgpt.com/backend-api/codex/responses",
+            "gpt-5.5",
+            {"Authorization": "Bearer test"},
+        )]
+
+    helper_calls = {}
+
+    async def fake_generate_email_summary(**kwargs):
+        helper_calls.update(kwargs)
+        return "- Scheduled summary"
+
+    monkeypatch.setattr(email_pollers, "_load_settings", lambda: {"email_auto_summarize": True})
+    monkeypatch.setattr(email_pollers, "_owner_for_email_account", lambda _account_id: "alice")
+    monkeypatch.setattr(email_pollers, "_imap_connect", lambda account_id=None, owner="": fake_conn)
+    monkeypatch.setattr(email_pollers, "_get_email_config", lambda account_id=None, owner="": {"from_address": "alice@example.com"})
+    monkeypatch.setattr(email_pollers, "resolve_task_candidates", fake_resolve_task_candidates)
+    monkeypatch.setattr(email_pollers, "_generate_scheduled_email_summary", fake_generate_email_summary)
+
+    result = await email_pollers._auto_summarize_pass_single(account_id="acct-alice")
+
+    assert "summarized 1" in result
+    assert "summary failed" not in result
+    assert helper_calls["url"] == "https://chatgpt.com/backend-api/codex/responses"
+    assert helper_calls["model"] == "gpt-5.5"
+    assert helper_calls["headers"]["Authorization"] == "Bearer test"
+    assert helper_calls["headers"]["Content-Type"] == "application/json"
+    assert helper_calls["owner"] == "alice"
+    assert fake_conn.logout_calls == 1
+
+    conn = sqlite3.connect(db_path)
+    try:
+        row = conn.execute(
+            "SELECT owner, summary, model_used FROM email_summaries WHERE message_id=?",
+            ("<scheduled@example.com>",),
+        ).fetchone()
+    finally:
+        conn.close()
+    assert row == ("alice", "- Scheduled summary", "gpt-5.5")

tests/test_fork_session_metadata.py

@@ -34,6 +34,11 @@ class _FakeSessionManager:
         self.sessions = {"src-id": source}
         self.created = None
 
+    def get_session(self, session_id):
+        # Fork looks the source up through get_session — the hydration seam —
+        # so a session only present in the DB still forks a real transcript.
+        return self.sessions[session_id]
+
     def create_session(self, session_id=None, name=None, endpoint_url=None,
                        model=None, rag=False, owner=None):
         self.created = _FakeSession(name=name, owner=owner)

tests/test_history_db_fallback_hidden.py

@@ -5,9 +5,9 @@ The in-memory branch skips messages whose metadata has ``hidden`` (e.g.
 compaction summaries that are kept for AI context but not shown to the user).
 The DB fallback (taken when the in-memory history is empty, e.g. after a
 restart) built the client response from every DB row with no such filter, so
-hidden messages leaked to the client on DB-served sessions. The rebuilt
-in-memory ``session.history`` must still keep them, though, so only the response
-is filtered.
+hidden messages leaked to the client on DB-served sessions. Hydration of
+``session.history`` belongs to ``get_session``; this fallback only shapes the
+response, so only the response is filtered.
 
 get_session_history depends on the DB, the session manager and a FastAPI
 request, so this pins the regression at the source level (as other route tests

tests/test_history_display_model_hydration.py

@@ -0,0 +1,549 @@
+"""Display pagination must stay separate from full model-context hydration."""
+
+import json
+from datetime import datetime, timedelta
+
+import pytest
+from fastapi import APIRouter, FastAPI
+from fastapi.testclient import TestClient
+from starlette.requests import Request
+from sqlalchemy import create_engine, event
+from sqlalchemy.orm import sessionmaker
+from sqlalchemy.pool import StaticPool
+
+from core.database import Base, ChatMessage as DbChatMessage, Session as DbSession
+from core.models import ChatMessage, Session
+from core.session_manager import SessionManager
+from routes import chat_routes
+from routes.history import history_routes
+from routes import session_routes
+from src.request_models import ChatRequest
+
+
+def _database():
+    engine = create_engine(
+        "sqlite://",
+        connect_args={"check_same_thread": False},
+        poolclass=StaticPool,
+    )
+    Base.metadata.create_all(
+        engine,
+        tables=[DbSession.__table__, DbChatMessage.__table__],
+    )
+    return engine, sessionmaker(bind=engine, autocommit=False, autoflush=False)
+
+
+def _seed_session(db_factory, *, session_id="session-1", message_count=6, stored_count=None):
+    """Seed `message_count` real rows; `stored_count` overrides the denormalized
+    sessions.message_count column so drift can be reproduced."""
+    db = db_factory()
+    try:
+        db.add(
+            DbSession(
+                id=session_id,
+                name="Long chat",
+                endpoint_url="http://model.test/v1",
+                model="test-model",
+                owner="alice",
+                message_count=message_count if stored_count is None else stored_count,
+            )
+        )
+        start = datetime(2026, 1, 1, 12, 0, 0)
+        for index in range(message_count):
+            db.add(
+                DbChatMessage(
+                    id=f"message-{index}",
+                    session_id=session_id,
+                    role="user" if index % 2 == 0 else "assistant",
+                    content=f"content-{index}",
+                    timestamp=start + timedelta(seconds=index),
+                )
+            )
+        db.commit()
+    finally:
+        db.close()
+
+
+def _chat_message_selects(statements):
+    return [
+        " ".join(statement.lower().split())
+        for statement in statements
+        if statement.lstrip().lower().startswith("select")
+        and "chat_messages" in statement.lower()
+    ]
+
+
+def _manager(db_factory, monkeypatch, sessions=None):
+    """A real SessionManager bound to the temp DB, with load counting."""
+    monkeypatch.setattr("core.session_manager.SessionLocal", db_factory)
+    manager = object.__new__(SessionManager)
+    manager.upload_handler = None
+    manager.sessions = sessions if sessions is not None else {}
+    manager.full_loads = 0
+
+    original_load = manager._load_session_from_db
+
+    def counting_load(session_id):
+        manager.full_loads += 1
+        return original_load(session_id)
+
+    manager._load_session_from_db = counting_load
+    return manager
+
+
+def test_paginated_history_reads_only_count_and_requested_page(monkeypatch):
+    engine, db_factory = _database()
+    _seed_session(db_factory)
+
+    class DisplayOnlyManager:
+        def get_session(self, _session_id):
+            raise AssertionError("paginated display history must not hydrate model context")
+
+    monkeypatch.setattr(history_routes, "SessionLocal", db_factory)
+    monkeypatch.setattr(history_routes, "_verify_session_owner", lambda *_args: None)
+
+    app = FastAPI()
+    app.include_router(history_routes.setup_history_routes(DisplayOnlyManager()))
+
+    statements = []
+
+    def capture_sql(_conn, _cursor, statement, _parameters, _context, _executemany):
+        statements.append(statement)
+
+    event.listen(engine, "before_cursor_execute", capture_sql)
+    try:
+        response = TestClient(app).get("/api/history/session-1?limit=2")
+    finally:
+        event.remove(engine, "before_cursor_execute", capture_sql)
+        engine.dispose()
+
+    assert response.status_code == 200
+    payload = response.json()
+    assert [message["content"] for message in payload["history"]] == [
+        "content-4",
+        "content-5",
+    ]
+    assert payload["total"] == 6
+    assert payload["offset"] == 4
+    assert payload["has_more_before"] is True
+    assert payload["has_more_after"] is False
+
+    # One COUNT for the total plus one page read — never a full-transcript
+    # select. The page bounds are asserted through the response above rather
+    # than by matching SQL text.
+    chat_selects = _chat_message_selects(statements)
+    assert len(chat_selects) == 2, chat_selects
+    assert sum("count(" in statement for statement in chat_selects) == 1
+
+
+def test_production_router_order_reaches_bounded_canonical_history(monkeypatch):
+    """The assembled app must not shadow canonical history with session routes."""
+    engine, db_factory = _database()
+    _seed_session(db_factory, message_count=1200)
+
+    class DisplayOnlyManager:
+        def get_session(self, _session_id):
+            raise AssertionError("bounded initial history must not hydrate all messages")
+
+    manager = DisplayOnlyManager()
+    monkeypatch.setattr(
+        session_routes,
+        "router",
+        APIRouter(prefix="/api", tags=["sessions"]),
+    )
+    monkeypatch.setattr(history_routes, "SessionLocal", db_factory)
+    monkeypatch.setattr(history_routes, "_verify_session_owner", lambda *_args: None)
+
+    app = FastAPI()
+    app.include_router(session_routes.setup_session_routes(manager, {}))
+    app.include_router(history_routes.setup_history_routes(manager))
+
+    try:
+        response = TestClient(app).get("/api/history/session-1?limit=24")
+    finally:
+        engine.dispose()
+
+    assert response.status_code == 200
+    assert response.request.url.params["limit"] == "24"
+    payload = response.json()
+    displayed = len(payload["history"])
+    assert 0 < displayed <= payload["limit"] <= 100
+    assert payload["total"] >= 1200
+    assert payload["has_more_before"] is True
+    assert displayed < payload["total"]
+
+
+def test_incomplete_cached_history_hydrates_once_for_model_context(monkeypatch):
+    engine, db_factory = _database()
+    raw_multimodal = json.dumps(
+        [
+            {"type": "text", "text": "look at the source image"},
+            {
+                "type": "image_url",
+                "image_url": {"url": "data:image/png;base64,AAAA"},
+            },
+        ]
+    )
+    db = db_factory()
+    try:
+        db.add(
+            DbSession(
+                id="session-1",
+                name="Long chat",
+                endpoint_url="http://model.test/v1",
+                model="test-model",
+                owner="alice",
+                message_count=3,
+            )
+        )
+        start = datetime(2026, 1, 1, 12, 0, 0)
+        db.add_all(
+            [
+                DbChatMessage(
+                    id="message-0",
+                    session_id="session-1",
+                    role="user",
+                    content=raw_multimodal,
+                    meta_data=json.dumps(
+                        {
+                            "attachments": [
+                                {
+                                    "id": "upload-1",
+                                    "filename": "source.png",
+                                    "content_type": "image/png",
+                                }
+                            ]
+                        }
+                    ),
+                    timestamp=start,
+                ),
+                DbChatMessage(
+                    id="message-1",
+                    session_id="session-1",
+                    role="assistant",
+                    content="answer",
+                    timestamp=start + timedelta(seconds=1),
+                ),
+                DbChatMessage(
+                    id="message-2",
+                    session_id="session-1",
+                    role="system",
+                    content="compaction summary",
+                    meta_data=json.dumps({"hidden": True}),
+                    timestamp=start + timedelta(seconds=2),
+                ),
+            ]
+        )
+        db.commit()
+    finally:
+        db.close()
+
+    manager = _manager(
+        db_factory,
+        monkeypatch,
+        sessions={
+            "session-1": Session(
+                id="session-1",
+                name="Long chat",
+                endpoint_url="http://model.test/v1",
+                model="test-model",
+                owner="alice",
+                history=[ChatMessage("user", "stale partial cache")],
+                # Deliberately stale too: get_session must refresh metadata before
+                # checking whether the cached transcript is complete.
+                message_count=1,
+            )
+        },
+    )
+
+    try:
+        hydrated = manager.get_session("session-1")
+        first_full_loads = manager.full_loads
+        warm = manager.get_session("session-1")
+        second_full_loads = manager.full_loads
+    finally:
+        engine.dispose()
+
+    assert hydrated is warm
+    assert len(hydrated.history) == 3
+    assert first_full_loads == 1
+    assert second_full_loads == first_full_loads
+
+    context = hydrated.get_context_messages()
+    assert len(context) == 3
+    assert context[0]["content"][1]["image_url"]["url"] == "data:image/png;base64,AAAA"
+    assert context[0]["metadata"]["attachments"] == [
+        {
+            "id": "upload-1",
+            "filename": "source.png",
+            "content_type": "image/png",
+        }
+    ]
+    hidden_summary = next(message for message in context if message["role"] == "system")
+    assert hidden_summary["content"] == "compaction summary"
+    assert hidden_summary["metadata"]["hidden"] is True
+
+
+def test_inflated_message_count_column_does_not_reload_warm_sessions(monkeypatch):
+    """A drifted-high sessions.message_count must not reload on every read.
+
+    `_persist_message` swallows a failed insert while `add_message` has already
+    appended in memory, so the next successful persist writes rows+1. Keyed on
+    that column, the hydration gate would stay true forever and re-select the
+    whole transcript on every send, edit, delete and truncate.
+    """
+    engine, db_factory = _database()
+    _seed_session(db_factory, message_count=6, stored_count=8)
+
+    manager = _manager(db_factory, monkeypatch)
+    try:
+        session = manager.get_session("session-1")
+        cold_loads = manager.full_loads
+        for _ in range(3):
+            manager.get_session("session-1")
+    finally:
+        engine.dispose()
+
+    assert len(session.history) == 6
+    assert cold_loads == 1
+    assert manager.full_loads == 1
+
+
+def test_stale_low_message_count_column_still_hydrates_for_the_model(monkeypatch):
+    """The other drift direction must not hand the model a truncated transcript.
+
+    `_persist_message` writes message_count = 0 when the session is not cached.
+    A partly-filled cache plus that stale-low column previously left the send
+    path with whatever RAM happened to hold.
+    """
+    engine, db_factory = _database()
+    _seed_session(db_factory, message_count=6, stored_count=0)
+
+    manager = _manager(
+        db_factory,
+        monkeypatch,
+        sessions={
+            "session-1": Session(
+                id="session-1",
+                name="Long chat",
+                endpoint_url="http://model.test/v1",
+                model="test-model",
+                owner="alice",
+                history=[ChatMessage("user", "content-0")],
+                message_count=0,
+            )
+        },
+    )
+    try:
+        session = manager.get_session("session-1")
+        manager.get_session("session-1")
+    finally:
+        engine.dispose()
+
+    assert [message.content for message in session.history] == [
+        f"content-{index}" for index in range(6)
+    ]
+    assert manager.full_loads == 1
+
+
+def test_fork_after_restart_copies_the_real_transcript(monkeypatch):
+    """Forking reads source.history, so it must hydrate through get_session.
+
+    Display pagination no longer fills the cache, so a fork taken after a
+    restart used to return HTTP 200 with an empty conversation.
+    """
+    engine, db_factory = _database()
+    _seed_session(db_factory, message_count=6)
+
+    # Restart state: metadata-only cache entry, exactly what load_sessions seeds.
+    manager = _manager(db_factory, monkeypatch)
+    manager.load_sessions()
+
+    monkeypatch.setattr(history_routes, "SessionLocal", db_factory)
+    monkeypatch.setattr(history_routes, "_verify_session_owner", lambda *_args: None)
+    monkeypatch.setattr("core.models._SESSION_MANAGER_INSTANCE", manager)
+
+    app = FastAPI()
+    app.include_router(history_routes.setup_history_routes(manager))
+    client = TestClient(app)
+
+    try:
+        page = client.get("/api/history/session-1?limit=2")
+        assert page.status_code == 200
+        assert len(manager.sessions["session-1"].history) == 0
+
+        response = client.post("/api/session/session-1/fork", json={"keep_count": 4})
+        assert response.status_code == 200
+        payload = response.json()
+        assert payload["kept"] == 4
+
+        forked = manager.get_session(payload["id"])
+        assert [message.content for message in forked.history] == [
+            f"content-{index}" for index in range(4)
+        ]
+    finally:
+        engine.dispose()
+
+
+class _ContextBuildReached(Exception):
+    pass
+
+
+class _ToolPolicy:
+    block_all_tool_calls = False
+
+    def blocks(self, _tool_name):
+        return False
+
+
+class _ChatHandler:
+    async def handle_memory_command(self, _session, _message):
+        return None
+
+
+def _json_request(path, payload):
+    raw = json.dumps(payload).encode()
+    sent = False
+
+    async def receive():
+        nonlocal sent
+        if sent:
+            return {"type": "http.request", "body": b"", "more_body": False}
+        sent = True
+        return {"type": "http.request", "body": raw, "more_body": False}
+
+    scope = {
+        "type": "http",
+        "asgi": {"version": "3.0"},
+        "http_version": "1.1",
+        "method": "POST",
+        "scheme": "http",
+        "path": path,
+        "raw_path": path.encode(),
+        "root_path": "",
+        "query_string": b"",
+        "headers": [(b"content-type", b"application/json")],
+        "client": ("127.0.0.1", 1234),
+        "server": ("testserver", 80),
+    }
+    return Request(scope, receive)
+
+
+def _route_endpoint(router, path):
+    return next(route.endpoint for route in router.routes if route.path == path)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("path", ["/api/chat", "/api/chat_stream"])
+async def test_model_send_routes_hydrate_before_context_build(monkeypatch, path):
+    # A real SessionManager over a real (temp) DB — a stub here would only
+    # assert that the stub hydrates, not that SessionManager does.
+    engine, db_factory = _database()
+    _seed_session(db_factory, message_count=6, stored_count=8)
+    manager = _manager(db_factory, monkeypatch)
+    manager.load_sessions()  # restart state: metadata only, no messages cached
+    contexts_built = []
+
+    async def assert_complete_context(session, *_args, **_kwargs):
+        contexts_built.append(session)
+        assert [message.content for message in session.history] == [
+            f"content-{index}" for index in range(6)
+        ]
+        raise _ContextBuildReached
+
+    monkeypatch.setattr(chat_routes, "_set_user_time_from_request", lambda *_args: None)
+    monkeypatch.setattr(chat_routes, "_verify_session_owner", lambda *_args: None)
+    monkeypatch.setattr(chat_routes, "effective_user", lambda *_args: "alice")
+    monkeypatch.setattr(
+        chat_routes,
+        "_clear_orphaned_session_endpoint",
+        lambda *_args, **_kwargs: False,
+    )
+    monkeypatch.setattr(
+        chat_routes,
+        "_recover_empty_session_model",
+        lambda *_args, **_kwargs: False,
+    )
+    monkeypatch.setattr(chat_routes, "_enforce_chat_privileges", lambda *_args: None)
+    monkeypatch.setattr(
+        chat_routes,
+        "build_effective_tool_policy",
+        lambda **_kwargs: _ToolPolicy(),
+    )
+    monkeypatch.setattr(chat_routes, "build_chat_context", assert_complete_context)
+    monkeypatch.setattr(
+        chat_routes,
+        "_resolve_request_workspace",
+        lambda *_args: (None, False),
+    )
+    monkeypatch.setattr(chat_routes, "_classify_tool_intent", lambda *_args: None)
+    monkeypatch.setattr(
+        chat_routes,
+        "_is_contextual_web_followup",
+        lambda *_args: False,
+    )
+    monkeypatch.setattr(
+        chat_routes,
+        "_is_contextual_browser_followup",
+        lambda *_args: False,
+    )
+    monkeypatch.setattr(
+        chat_routes,
+        "_resolve_workspace_from_message_path",
+        lambda *_args: (None, None),
+    )
+    monkeypatch.setattr(
+        chat_routes,
+        "_reconcile_selected_route_from_request",
+        lambda *_args, **_kwargs: False,
+    )
+    monkeypatch.setattr(chat_routes, "resolve_session_auth", lambda *_args, **_kwargs: None)
+    monkeypatch.setattr(chat_routes, "get_session_mode", lambda *_args: "chat")
+    monkeypatch.setattr(
+        chat_routes,
+        "_is_image_generation_session",
+        lambda *_args, **_kwargs: False,
+    )
+    monkeypatch.setattr(chat_routes, "web_search_enabled_for_turn", lambda *_args: False)
+
+    router = chat_routes.setup_chat_routes(
+        manager,
+        _ChatHandler(),
+        object(),
+        object(),
+        object(),
+        object(),
+    )
+    endpoint = _route_endpoint(router, path)
+
+    async def send():
+        if path == "/api/chat":
+            await endpoint(
+                _json_request(path, {}),
+                ChatRequest(message="hello", session="session-1"),
+            )
+        else:
+            await endpoint(
+                _json_request(
+                    path,
+                    {"message": "hello", "session": "session-1"},
+                )
+            )
+
+    try:
+        with pytest.raises(_ContextBuildReached):
+            await send()
+        first_loads = manager.full_loads
+
+        # Second send on the now-warm session: the transcript is complete, so
+        # it must be served from RAM even though sessions.message_count is
+        # still drifted high in the DB.
+        with pytest.raises(_ContextBuildReached):
+            await send()
+    finally:
+        engine.dispose()
+
+    assert len(contexts_built) == 2
+    assert contexts_built[0] is contexts_built[1]
+    assert first_loads == 1
+    assert manager.full_loads == 1

tests/test_imap_mailbox_quoting.py

@@ -87,7 +87,7 @@ def test_known_imap_mailbox_call_sites_are_quoted():
     assert "conn.select(sent_name" not in pollers
     assert "imap.append(sent_folder" not in pollers
 
-    document_routes = Path("routes/document_routes.py").read_text()
+    document_routes = Path("routes/document/document_routes.py").read_text()
     assert "conn.select(doc.source_email_folder" not in document_routes
 
 

tests/test_integration_api_call_ssrf.py

@@ -9,8 +9,13 @@ link-local/metadata is always rejected; RFC-1918/loopback only when
 INTEGRATION_API_BLOCK_PRIVATE_IPS=true (LAN integrations are the primary
 use case, so private stays allowed by default).
 """
+import asyncio
+import ipaddress
+import ssl
 from unittest.mock import AsyncMock, MagicMock, patch
 
+import httpcore
+import httpx
 import pytest
 
 from src import integrations
@@ -97,3 +102,238 @@ async def test_private_base_url_allowed_by_default_blocked_with_knob(monkeypatch
     assert result["exit_code"] == 1
     assert "rejected" in result["error"].lower()
     client.request.assert_not_called()
+
+
+async def _call_capturing_transport(base_url, path="/items"):
+    """Drive execute_api_call and return (result, transport) where transport is
+    the object passed to httpx.AsyncClient(transport=...)."""
+    resp = MagicMock()
+    resp.status_code = 200
+    resp.headers = {"content-type": "application/json"}
+    resp.json.return_value = {"ok": True}
+    resp.text = '{"ok": true}'
+
+    client = AsyncMock()
+    client.__aenter__ = AsyncMock(return_value=client)
+    client.__aexit__ = AsyncMock(return_value=None)
+    client.request = AsyncMock(return_value=resp)
+
+    captured = {}
+
+    def _fake_async_client(*args, **kwargs):
+        captured.update(kwargs)
+        return client
+
+    with (
+        patch.object(integrations, "_find_integration",
+                     return_value=_integration(base_url)),
+        patch("httpx.AsyncClient", side_effect=_fake_async_client),
+    ):
+        result = await integrations.execute_api_call("test_integ", "GET", path)
+    return result, captured.get("transport"), client
+
+
+@pytest.mark.asyncio
+async def test_connection_is_pinned_to_the_validated_ip(monkeypatch):
+    """DNS-rebinding defense: the guard resolves the host once to a benign
+    public IP, and the request must be pinned to *that* IP so a host that
+    rebinds to the metadata range at connect time can't be reached with the
+    integration's auth headers. Static resolution passing the guard is not
+    enough — a plain client would re-resolve at connect."""
+    monkeypatch.setattr("src.url_safety._default_resolver",
+                        lambda host: ["93.184.216.34"])
+    result, transport, client = await _call_capturing_transport(
+        "http://rebinding.attacker.example")
+
+    assert result.get("exit_code") == 0
+    client.request.assert_called_once()
+    assert isinstance(transport, integrations._PinnedAsyncTransport)
+    assert [str(ip) for ip in transport._pinned_ips] == ["93.184.216.34"]
+
+
+@pytest.mark.asyncio
+async def test_pin_carries_the_whole_validated_ip_set(monkeypatch):
+    """When a host resolves to several records the transport keeps all of them
+    (check_outbound_url validated every one), in resolver order, so it can fall
+    back past a dead first address instead of failing the whole call."""
+    monkeypatch.setattr("src.url_safety._default_resolver",
+                        lambda host: ["93.184.216.34", "198.51.100.7"])
+    result, transport, _ = await _call_capturing_transport("http://multi.example")
+
+    assert result.get("exit_code") == 0
+    assert [str(ip) for ip in transport._pinned_ips] == ["93.184.216.34", "198.51.100.7"]
+
+
+class _FakeStream:
+    """Stand-in for the connected socket the real backend returns."""
+
+
+class _RecordingBackend:
+    """Fake httpcore backend: connect_tcp fails for the addresses in `dead`
+    and succeeds for the rest, recording the order it was asked to connect."""
+
+    def __init__(self, dead):
+        self.dead = set(dead)
+        self.attempts = []
+
+    async def connect_tcp(self, host, port, timeout=None, local_address=None,
+                          socket_options=None):
+        self.attempts.append((host, timeout))
+        if host in self.dead:
+            raise httpcore.ConnectError(f"connection refused: {host}")
+        return _FakeStream()
+
+
+def _pinned_backend(ips, dead):
+    """A _PinnedAsyncBackend whose underlying connect is the recording fake."""
+    backend = integrations._PinnedAsyncBackend(ips)
+    backend._real = _RecordingBackend(dead)
+    return backend
+
+
+@pytest.mark.asyncio
+async def test_connect_falls_back_from_dead_first_to_live_second():
+    """first-dead / second-live: the pinned backend must try the next validated
+    address when the first refuses, rather than surfacing the failure. It also
+    ignores the `host` httpcore passes (the original hostname) and connects to
+    the pinned IPs, which is what keeps TLS SNI / Host on the real hostname."""
+    ips = [ipaddress.ip_address("203.0.113.10"), ipaddress.ip_address("198.51.100.7")]
+    backend = _pinned_backend(ips, dead={"203.0.113.10"})
+
+    stream = await backend.connect_tcp("original.hostname.example", 443, timeout=5.0)
+
+    assert isinstance(stream, _FakeStream)
+    # Tried the dead address first, then the live one — never the hostname.
+    assert [host for host, _ in backend._real.attempts] == ["203.0.113.10", "198.51.100.7"]
+    # Fallback shared one budget: the second attempt got the time left, not a fresh 5s.
+    assert backend._real.attempts[1][1] <= 5.0
+
+
+@pytest.mark.asyncio
+async def test_connect_raises_when_every_validated_address_is_dead():
+    ips = [ipaddress.ip_address("203.0.113.10"), ipaddress.ip_address("198.51.100.7")]
+    backend = _pinned_backend(ips, dead={"203.0.113.10", "198.51.100.7"})
+
+    with pytest.raises(httpcore.ConnectError):
+        await backend.connect_tcp("original.hostname.example", 443, timeout=5.0)
+    assert [host for host, _ in backend._real.attempts] == ["203.0.113.10", "198.51.100.7"]
+
+
+@pytest.mark.asyncio
+async def test_pinned_transport_reuses_httpx_ca_trust(monkeypatch):
+    """TLS trust must come from the same builder the default httpx client uses
+    (certifi + SSL_CERT_FILE / SSL_CERT_DIR via trust_env), not from
+    ssl.create_default_context()'s system roots — otherwise chains that verified
+    under the old default client can silently stop verifying."""
+    sentinel = ssl.create_default_context()
+    calls = []
+
+    def _fake_create(*args, **kwargs):
+        calls.append(kwargs)
+        return sentinel
+
+    monkeypatch.setattr(httpx, "create_ssl_context", _fake_create)
+    transport = integrations._PinnedAsyncTransport([ipaddress.ip_address("93.184.216.34")])
+    try:
+        assert calls, "transport did not build its context via httpx.create_ssl_context"
+        assert transport._pool._ssl_context is sentinel
+    finally:
+        await transport.aclose()
+
+
+@pytest.mark.asyncio
+async def test_real_socket_falls_back_from_dead_first_to_live_second():
+    """End-to-end over real loopback sockets: pin [127.0.0.2 (nothing
+    listening), 127.0.0.1 (live)], and the request must succeed by falling back
+    to the second address while the Host header stays the original hostname —
+    i.e. only the socket destination moved, vhost/SNI routing did not."""
+    captured = {}
+
+    async def handle(reader, writer):
+        request = await reader.read(4096)
+        for line in request.split(b"\r\n"):
+            if line.lower().startswith(b"host:"):
+                captured["host"] = line.split(b":", 1)[1].strip().decode()
+        writer.write(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nhi")
+        await writer.drain()
+        writer.close()
+
+    server = await asyncio.start_server(handle, "127.0.0.1", 0)
+    port = server.sockets[0].getsockname()[1]
+    async with server:
+        await server.start_serving()
+        transport = integrations._PinnedAsyncTransport(
+            [ipaddress.ip_address("127.0.0.2"), ipaddress.ip_address("127.0.0.1")]
+        )
+        try:
+            async with httpx.AsyncClient(transport=transport) as client:
+                resp = await client.get(f"http://pinned.example:{port}/health")
+        finally:
+            await transport.aclose()
+
+    assert resp.status_code == 200
+    assert resp.text == "hi"
+    assert captured.get("host") == f"pinned.example:{port}"
+
+
+@pytest.mark.asyncio
+async def test_ip_literal_base_url_still_pins_and_is_not_rejected():
+    """A base_url that is already an IP has nothing to rebind, but it must not
+    trip the "did not resolve" guard either.
+
+    check_outbound_url resolves even a literal (getaddrinfo returns the address
+    itself), so the captured list is populated and the pin is a no-op rather
+    than a rejection. Uses the real resolver on purpose — no monkeypatch — so
+    this would catch the fail-closed branch firing on a literal.
+    """
+    result, transport, client = await _call_capturing_transport(
+        "http://93.184.216.34")
+
+    assert result.get("exit_code") == 0
+    assert isinstance(transport, integrations._PinnedAsyncTransport)
+    assert [str(ip) for ip in transport._pinned_ips] == ["93.184.216.34"]
+
+
+@pytest.mark.asyncio
+async def test_ipv6_base_url_pins_every_validated_address(monkeypatch):
+    """IPv6 goes down the same path as v4.
+
+    Resolution is stubbed rather than using a literal so this doesn't depend on
+    the runner having IPv6 configured.
+    """
+    v6 = "2606:2800:220:1:248:1893:25c8:1946"
+    monkeypatch.setattr("src.url_safety._default_resolver", lambda host: [v6])
+    result, transport, client = await _call_capturing_transport("http://v6.example")
+
+    assert result.get("exit_code") == 0
+    assert isinstance(transport, integrations._PinnedAsyncTransport)
+    assert [str(ip) for ip in transport._pinned_ips] == [v6]
+
+
+def test_validated_ips_strips_zone_id_and_drops_junk():
+    """getaddrinfo can hand back a scoped v6 address like 'fe80::1%eth0'."""
+    got = integrations._validated_ips(
+        ["93.184.216.34", "fe80::1%eth0", "not-an-ip", None, "2001:db8::5"]
+    )
+    assert [str(ip) for ip in got] == ["93.184.216.34", "fe80::1", "2001:db8::5"]
+
+
+def test_validated_ips_deduplicates_repeated_addresses():
+    """The resolver is getaddrinfo(host, None) with no socktype filter, so glibc
+    returns one record per socktype and a single-homed host arrives three times
+    over. Duplicates must collapse (first-seen order kept) or the connect
+    fallback wastes its shared deadline retrying one dead address."""
+    got = integrations._validated_ips(
+        ["93.184.216.34", "93.184.216.34", "93.184.216.34"]
+    )
+    assert [str(ip) for ip in got] == ["93.184.216.34"]
+
+    # Order is first-seen, and distinct addresses all survive.
+    got = integrations._validated_ips(
+        ["198.51.100.7", "93.184.216.34", "198.51.100.7", "2001:db8::5"]
+    )
+    assert [str(ip) for ip in got] == ["198.51.100.7", "93.184.216.34", "2001:db8::5"]
+
+    # A zone-id variant is the same address once stripped, so it collapses too.
+    got = integrations._validated_ips(["fe80::1%eth0", "fe80::1%eth1", "fe80::1"])
+    assert [str(ip) for ip in got] == ["fe80::1"]

tests/test_integrations_api_call_truncation.py

@@ -83,9 +83,10 @@ async def _call(json_data, status=200):
     with (
         patch.object(integrations, "_find_integration", return_value=DUMMY_INTEGRATION),
         patch("httpx.AsyncClient", return_value=mock_client),
-        # api.example.com doesn't resolve; the SSRF guard would fail closed.
-        # These tests are about truncation, so stub the guard open.
-        patch("src.url_safety.check_outbound_url", return_value=(True, "ok")),
+        # api.example.com doesn't resolve. Point the resolver at a public
+        # address instead of stubbing the guard open, so the real check (and
+        # the connect-IP pinning that reads its result) still runs.
+        patch("src.url_safety._default_resolver", lambda host: ["93.184.216.34"]),
     ):
         return await integrations.execute_api_call("test_integ", "GET", "/items")
 
@@ -101,9 +102,10 @@ async def _call_with_integration(integration, path="/items"):
     with (
         patch.object(integrations, "_find_integration", return_value=integration),
         patch("httpx.AsyncClient", return_value=mock_client),
-        # api.example.com doesn't resolve; the SSRF guard would fail closed.
-        # These tests are about URL joining, so stub the guard open.
-        patch("src.url_safety.check_outbound_url", return_value=(True, "ok")),
+        # api.example.com doesn't resolve. Point the resolver at a public
+        # address instead of stubbing the guard open, so the real check (and
+        # the connect-IP pinning that reads its result) still runs.
+        patch("src.url_safety._default_resolver", lambda host: ["93.184.216.34"]),
     ):
         result = await integrations.execute_api_call("test_integ", "GET", path)
     return result, mock_client

tests/test_issue_description_check.py

@@ -0,0 +1,86 @@
+"""Regression coverage for issue-description label lifecycle events."""
+
+import json
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+
+_REPO = Path(__file__).resolve().parent.parent
+_CHECKER = _REPO / ".github" / "scripts" / "check-issue-description.js"
+_WORKFLOW = _REPO / ".github" / "workflows" / "issue-description-check.yml"
+pytestmark = pytest.mark.skipif(not shutil.which("node"), reason="node not on PATH")
+
+
+def _run_closed_issue(action):
+    harness = r"""
+const checkIssueDescription = require(process.argv[1]);
+const action = process.argv[2];
+const calls = [];
+const unexpected = (name) => async () => {
+  throw new Error(`${name} should not be called for a closed issue`);
+};
+
+const github = {
+  rest: {
+    issues: {
+      removeLabel: async (params) => calls.push({ method: 'removeLabel', params }),
+      getLabel: unexpected('getLabel'),
+      addLabels: unexpected('addLabels'),
+      listComments: unexpected('listComments'),
+      createComment: unexpected('createComment'),
+      updateComment: unexpected('updateComment'),
+      deleteComment: unexpected('deleteComment'),
+    },
+  },
+};
+const context = {
+  payload: {
+    action,
+    issue: { number: 42, state: 'closed', body: '', labels: [] },
+  },
+  repo: { owner: 'odysseus-dev', repo: 'odysseus' },
+};
+const core = {
+  warning: unexpected('core.warning'),
+  setFailed: unexpected('core.setFailed'),
+};
+
+checkIssueDescription({ github, context, core })
+  .then(() => process.stdout.write(JSON.stringify(calls)))
+  .catch((error) => {
+    console.error(error);
+    process.exitCode = 1;
+  });
+"""
+    proc = subprocess.run(
+        ["node", "-e", harness, str(_CHECKER), action],
+        capture_output=True,
+        text=True,
+        cwd=str(_REPO),
+        timeout=30,
+    )
+    assert proc.returncode == 0, proc.stderr
+    return json.loads(proc.stdout)
+
+
+def test_workflow_handles_issue_closures():
+    workflow = _WORKFLOW.read_text()
+    assert "types: [opened, edited, reopened, closed]" in workflow
+
+
+@pytest.mark.parametrize("action", ["closed", "edited"])
+def test_closed_issue_only_drops_ready_for_review(action):
+    assert _run_closed_issue(action) == [
+        {
+            "method": "removeLabel",
+            "params": {
+                "owner": "odysseus-dev",
+                "repo": "odysseus",
+                "issue_number": 42,
+                "name": "ready for review",
+            },
+        }
+    ]

tests/test_live_thinking_chat_integration.py

@@ -0,0 +1,145 @@
+"""Source-level wiring guards for live-thinking stream lifecycle.
+
+The pure scheduler suite covers timing behavior. These assertions pin the
+browser-only integration seams that are impractical to import without the full
+application DOM.
+"""
+
+from pathlib import Path
+
+
+_CHAT = (Path(__file__).resolve().parent.parent / "static" / "js" / "chat.js").read_text(
+    encoding="utf-8"
+)
+
+
+def _between(start: str, end: str) -> str:
+    return _CHAT.split(start, 1)[1].split(end, 1)[0]
+
+
+def test_in_thinking_delta_short_circuits_before_cumulative_normalization():
+    delta_handler = _between(
+        "let _delta = json.delta;",
+        "} else if (json.type === 'research_progress')",
+    )
+    delta_path = _between(
+        "// Detect thinking-in-progress:",
+        "} else if (json.type === 'research_progress')",
+    )
+    guard = "if (!_thinkingAnalysisGate.shouldAnalyze(roundText, {"
+    normalize = "markdownModule.normalizeThinkingMarkup(roundText)"
+    assert guard in delta_path
+    assert delta_path.index(guard) < delta_path.index(normalize)
+    assert "_queueLiveThinking(roundText);" in delta_path
+    assert "createThinkingAnalysisGate" in _CHAT
+    projector_append = "_roundDisplayProjector.append(_delta, roundText);"
+    assert projector_append in delta_handler
+    assert delta_handler.index(projector_append) < delta_handler.index(guard)
+    assert "_renderStream({ knownNormal: true, displayText: _roundDisplayProjector.current() });" in delta_path
+    assert "_replyDisplayProjector.append(_delta, roundReplyText)" in delta_path
+
+
+def test_short_close_grace_expires_without_another_delta():
+    assert "function _scheduleThinkingGrace()" in _CHAT
+    grace = _between(
+        "function _scheduleThinkingGrace()",
+        "function _replyAfterClosedThinking",
+    )
+    assert "setTimeout(() =>" in grace
+    assert "_finishLiveThinkingTransition();" in grace
+    cancel = _between("_cancelLiveThinkingWork = () =>", "function _finalizeLiveThinking")
+    assert "_cancelThinkingGrace();" in cancel
+    delta_path = _between(
+        "// Detect thinking-in-progress:",
+        "} else if (json.type === 'research_progress')",
+    )
+    false_close = _between(
+        "// Detect false close:",
+        "if (hasUnclosedThink && !isThinking)",
+    )
+    assert "Do NOT require a prior unclosed delta" in false_close
+    assert "_afterClose &&" in false_close
+    assert "&& isThinking" not in false_close.split("let _falseCloseDeadline", 1)[1].split("if (isThinking)", 1)[0]
+    assert "_thinkingRecheckAt = _falseCloseDeadline || 0;" in delta_path
+
+
+def test_terminal_paths_use_one_authoritative_rich_round_render():
+    tool_path = _between(
+        "} else if (json.type === 'tool_start') {",
+        "} else if (json.type === 'tool_output') {",
+    )
+    assert "_endLiveThinkingSection({ rich: false });" in tool_path
+    assert tool_path.count("_finalizeRoundRender();") == 1
+    assert "_renderStream();" not in tool_path
+
+    agent_path = _between(
+        "} else if (json.type === 'agent_step') {",
+        "} else if (json.type === 'budget_exceeded') {",
+    )
+    assert "_endLiveThinkingSection({ rich: false });" in agent_path
+    assert agent_path.count("_finalizeRoundRender();") == 1
+    assert "if (!roundFinalized)" not in agent_path
+
+    catch_path = _between(
+        "// foreground session's text.\n      const _isBgCatch",
+        "} finally {",
+    )
+    assert "if (_isBgCatch)" in catch_path
+    assert "_cancelLiveThinkingWork();" in catch_path
+    assert "_catchTerminalView = _finalizeInterruptedView();" in catch_path
+    assert "_finalizeRoundRender();" not in catch_path
+    assert "_endThinkingOnTerminalPath({ rich: false });" in catch_path
+    assert "const _catchViewHolder = _catchTerminalView?.holder || holder;" in catch_path
+
+    round_finalizer = _between(
+        "_finalizeRoundRender = () => {",
+        "_finalizeInterruptedView = () => {",
+    )
+    assert "if (roundFinalized) return roundFinalization;" in round_finalizer
+    assert round_finalizer.index("processWithThinking") < round_finalizer.rindex("roundFinalized = true;")
+    assert "lastContentRoundHolder = terminalHolder;" in round_finalizer
+
+    interrupted_finalizer = _between(
+        "_finalizeInterruptedView = () => {",
+        "function _replyAfterClosedThinking",
+    )
+    assert "finalization?.hasContent" in interrupted_finalizer
+    assert "lastContentRoundHolder || finalization?.holder" in interrupted_finalizer
+
+    stop_path = _between(
+        "// Render whatever was accumulated so far",
+        "// Reset button state",
+    )
+    assert "const _stoppedViewHolder = _terminalView?.holder || currentHolder;" in stop_path
+    assert "_stoppedViewHolder.querySelector('.body').appendChild(stoppedIndicator);" in stop_path
+
+    done_path = _between(
+        "if (data === '[DONE]') {",
+        "try {\n              const json = JSON.parse(data);",
+    )
+    assert "_finalizeLiveThinking(_closedThinkingText(roundText), false);" in done_path
+    assert "_renderStream();" not in done_path
+
+    post_loop = _between(
+        "if (!_streamSawDone) {",
+        "// --- Final render (skip if stream was ever backgrounded or currently in background) ---",
+    )
+    assert "_cancelLiveThinkingWork();" in post_loop
+    assert "_renderStream();" not in post_loop
+
+    recovery_path = _between(
+        "function _tryAutoRecover(holder, accumulated, sessionId)",
+        "function _removeStallBanner()",
+    )
+    assert "processWithThinking" not in recovery_path
+
+
+def test_detach_synchronously_cancels_delayed_view_work():
+    registration = _between("_activeStreams.set(streamSessionId", "_syncForegroundStreamGlobals();")
+    assert "cancelViewWork: () => _cancelLiveThinkingWork()" in registration
+
+    detach = _between("export function detachCurrentStream", "// _notifyStreamComplete")
+    cancel = "if (active.cancelViewWork) active.cancelViewWork();"
+    background = "_backgroundStreams.set(sessionId"
+    assert cancel in detach
+    assert detach.index(cancel) < detach.index(background)

tests/test_live_thinking_scheduler_js.py

@@ -0,0 +1,29 @@
+"""Runs the live-thinking throttle's behavioral suite under pytest.
+
+Behavior lives in tests/live_thinking_scheduler.test.mjs (node:test, no DOM).
+This wrapper only exists so the JS suite runs in the normal pytest job.
+"""
+
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+_REPO = Path(__file__).resolve().parent.parent
+_HAS_NODE = shutil.which("node") is not None
+
+
+@pytest.mark.skipif(not _HAS_NODE, reason="node binary not on PATH")
+def test_live_thinking_scheduler_behavior():
+    result = subprocess.run(
+        ["node", "--test", "tests/live_thinking_scheduler.test.mjs"],
+        cwd=_REPO,
+        capture_output=True,
+        timeout=30,
+        text=True,
+    )
+    if result.returncode != 0:
+        raise AssertionError(
+            f"node --test failed:\nSTDOUT:\n{result.stdout}\nSTDERR:\n{result.stderr}"
+        )

tests/test_llm_core_anthropic_temp_omit.py

@@ -29,6 +29,13 @@ from src.llm_core import _anthropic_rejects_temperature, _build_anthropic_payloa
         "anthropic/claude-opus-4-7",  # tolerate a provider-prefixed id
         "claude-opus-4-10",  # future minor still >= 4.7
         "claude-opus-5-0",  # future major
+        # Major-only ids: a missing minor reads as `.0`, so these are >= 4.7 too
+        # (issue #5753). Before the fix the version pattern required a minor, so
+        # these fell through to "accepts temperature" and every call 400'd.
+        "claude-opus-5",
+        "claude-opus-5-20260101",  # major-only + dated snapshot
+        "anthropic/claude-opus-5",  # major-only behind a provider prefix
+        "claude-opus-6",  # future major-only
     ],
 )
 def test_opus_47_plus_rejects_temperature(model):
@@ -48,7 +55,10 @@ def test_opus_47_plus_rejects_temperature(model):
         "claude-opus-4-6-20251201",  # dated 4.6 snapshot — older, still keeps temperature
         "claude-sonnet-4-6",
         "claude-3-5-sonnet",
-        "claude-3-opus-20240229",  # legacy Claude 3 Opus — no opus-N-M pattern, kept
+        "claude-3-opus-20240229",  # legacy Claude 3 Opus — date directly after
+        # "opus-", so the major must not swallow it as version 20240229 (that is
+        # what makes capping the major at 1-2 digits necessary once the minor
+        # became optional in #5753).
         "claude-haiku-4-5",
         "claude-x",
         "octopus-4-8",  # "opus" only as a substring of another word — must not match
@@ -87,6 +97,20 @@ def test_payload_keeps_temperature_for_older_models():
     assert _payload("claude-3-5-sonnet", 1.2)["temperature"] == 1.0
 
 
+def test_payload_omits_temperature_for_major_only_opus_5():
+    # Issue #5753: the scheduled-task path calls stream_agent_loop() without a
+    # temperature and inherits its 0.3 default, so `claude-opus-5` 400'd on every
+    # run and surfaced as "the model returned an empty response". Interactive chat
+    # leaves temperature None and never hit it.
+    assert "temperature" not in _payload("claude-opus-5", 0.3)
+
+
+def test_payload_keeps_temperature_for_legacy_claude_3_opus():
+    # Guards the major-digit cap: `opus-20240229` must not parse as version
+    # 20240229, or Claude 3 Opus would silently lose the caller's temperature.
+    assert _payload("claude-3-opus-20240229", 0.5)["temperature"] == 0.5
+
+
 def test_payload_keeps_temperature_for_dated_opus_4_0():
     # Anthropic's dated id for Opus 4.0 (claude-opus-4-20250514) is in this repo's
     # ANTHROPIC_MODELS list. The date must not be misread as a >= 4.7 minor, or the

tests/test_manage_skills_action_required.py

@@ -0,0 +1,24 @@
+import json
+
+import pytest
+
+from src.tools.system import do_manage_skills
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "payload",
+    [
+        {},
+        {"action": ""},
+        {"action": "   "},
+        {"name": "demo", "description": "x", "procedure": ["step"]},
+    ],
+)
+async def test_manage_skills_requires_action(payload):
+    result = await do_manage_skills(json.dumps(payload), owner="test")
+
+    assert result == {
+        "error": "action is required (list|view|view_ref|add|edit|patch|publish|delete|search)",
+        "exit_code": 1,
+    }

tests/test_markdown_rendering_js.py

@@ -214,6 +214,50 @@ def test_inline_code_content_is_html_escaped(node_available):
     assert "<b>" not in html
 
 
+def test_fenced_code_keeps_dollar_ampersand(node_available):
+    # Issue #5663: the block-restore pass used a string replacement, so `$&` in a
+    # restored block was read as "the matched text" and re-inserted the
+    # placeholder. `perl -pe 's/world/$& again/'` rendered as
+    # "s/world/___CODE_BLOCK_0___amp; again/" — the trailing "amp;" is the orphan
+    # left behind after `$&` consumed the `$&` of the escaped `$&amp;`.
+    html = _run_markdown_case(
+        "```sh\necho \"hello world\" | perl -pe 's/world/$& again/'\n```"
+    )
+
+    assert "___CODE_BLOCK_" not in html
+    assert "s/world/$&amp; again/" in html
+    assert "amp; again" not in html.replace("$&amp; again", "")
+
+
+def test_fenced_code_keeps_dollar_backtick_and_quote(node_available):
+    # `` $` `` and `$'` splice the text before/after the placeholder into the
+    # block. Unlike `$&` these leave no placeholder behind — the characters just
+    # vanish — so assert the content survives verbatim.
+    html = _run_markdown_case("```sh\nsed \"s/$`/x/\" && sed \"s/$'/y/\"\n```")
+
+    assert "___CODE_BLOCK_" not in html
+    assert "s/$`/x/" in html
+    assert "s/$&#39;/y/" in html
+
+
+def test_fenced_code_keeps_double_dollar(node_available):
+    # `$$` collapsed to a single `$` in the restored block.
+    html = _run_markdown_case('```sh\necho "$$USD and $$"\n```')
+
+    assert "$$USD and $$" in html
+
+
+def test_mermaid_block_keeps_dollar_ampersand(node_available):
+    # The mermaid restore site had the same hazard: a node label containing `$&`
+    # re-inserted the ___MERMAID_BLOCK_n___ placeholder into the diagram source,
+    # which then fails to parse. The math and allowed-HTML sites are fixed the
+    # same way; they need KaTeX/sanitizer conditions this harness doesn't set up.
+    html = _run_markdown_case('```mermaid\ngraph TD; A["$&"] --> B;\n```')
+
+    assert "___MERMAID_BLOCK_" not in html
+    assert "$&amp;" in html
+
+
 def test_currency_dollar_amounts_are_not_rendered_as_math(node_available):
     # "$5 to $10" used to pair the two dollar signs as inline-math delimiters
     # and render "5 to" through KaTeX. Pandoc-style rules now reject it: the

tests/test_mcp_dependency_compatibility.py

@@ -0,0 +1,15 @@
+"""Regression coverage for the built-in MCP servers' SDK compatibility line."""
+
+from pathlib import Path
+
+
+REQUIREMENTS = Path(__file__).resolve().parents[1] / "requirements.txt"
+
+
+def test_mcp_requirement_excludes_breaking_v2_sdk():
+    requirements = [
+        line.split("#", 1)[0].strip().replace(" ", "")
+        for line in REQUIREMENTS.read_text(encoding="utf-8").splitlines()
+    ]
+
+    assert "mcp<2" in requirements

tests/test_memory_add_submit_regression.py

@@ -0,0 +1,54 @@
+"""The Brain > Add Memory form must be submittable (#5828).
+
+The form previously had no submit button and relied on a deprecated
+``keypress`` listener for Enter, which is not guaranteed to fire on all
+platforms — leaving the form with no working submit path. Pins:
+
+- a visible, keyboard-accessible submit button next to the category select;
+- the button wired to ``memoryModule.addNewMemory()``;
+- Enter handled via ``keydown`` with ``preventDefault()`` (and no lingering
+  ``keypress`` handler on the input).
+"""
+from pathlib import Path
+
+APP_JS = Path("static/app.js")
+INDEX_HTML = Path("static/index.html")
+
+
+def _add_memory_row(html):
+    start = html.index('id="new-memory-input"')
+    end = html.index("</div>", html.index('id="new-memory-add-btn"', start))
+    return html[start:end]
+
+
+def test_add_memory_form_renders_a_submit_button():
+    html = INDEX_HTML.read_text()
+    row = _add_memory_row(html)
+
+    assert 'id="new-memory-category"' in row, "button must sit in the same row as the form fields"
+    btn_start = row.index('id="new-memory-add-btn"')
+    btn_tag = row[row.rindex("<button", 0, btn_start):row.index(">", btn_start)]
+    assert 'type="button"' in btn_tag, "must not rely on implicit submit semantics"
+
+
+def _new_memory_wiring_block(source):
+    start = source.index("const newMemoryInput = el('new-memory-input');")
+    end = source.index("// Voice recording", start)
+    return source[start:end]
+
+
+def test_submit_button_is_wired_to_add_new_memory():
+    block = _new_memory_wiring_block(APP_JS.read_text())
+
+    assert "el('new-memory-add-btn')" in block
+    assert "addEventListener('click', () => memoryModule.addNewMemory())" in block
+
+
+def test_enter_uses_keydown_with_prevent_default():
+    block = _new_memory_wiring_block(APP_JS.read_text())
+
+    assert "addEventListener('keydown'" in block
+    assert "addEventListener('keypress'" not in block, "keypress is deprecated and unreliable for Enter"
+    assert "e.preventDefault();" in block
+    assert "!e.isComposing" in block, "IME composition must not submit the form"
+    assert "memoryModule.addNewMemory();" in block

tests/test_memory_extractor_vector_cross_tenant.py

@@ -67,6 +67,12 @@ class FakeMemoryManager:
     def load_all(self):
         return list(self.rows)
 
+    def load_all_for_update(self):
+        # Mirrors the real MemoryManager: extraction is a read-modify-write and
+        # goes through the strict loader (#5673). A healthy store behaves the
+        # same as load_all.
+        return list(self.rows)
+
     def load(self, owner=None):
         return [r for r in self.rows if r.get("owner") == owner]
 

tests/test_memory_store_unreadable_no_wipe.py

@@ -0,0 +1,255 @@
+"""A memory store that cannot be READ must never be overwritten (issue #5673).
+
+`MemoryManager.save` is atomic, and the add/import/extract paths are all
+read-modify-write: load the whole store, append, save it back. `load_all`
+used to answer a *failed read* with `[]` — indistinguishable from "no
+memories" — so a failed read turned into
+
+    load_all() -> []  ->  [].append(new)  ->  save([new])
+
+which atomically replaced the entire store with one entry.
+
+The trigger that actually bites is a store that is **readable but not
+parseable** — a truncated file, or one holding `{}` instead of `[]`. Nothing
+obstructs the write, so the request succeeds with HTTP 200 and every existing
+memory is destroyed silently. Truncation is reachable: `core/database.py`
+rewrites memory.json during migration with a plain `open(..., "w")` +
+`json.dump`, which is not atomic.
+
+A live exclusive lock is NOT the dangerous case: it blocks the read and the
+`os.replace` alike, so the save fails too and the store survives (verified
+end-to-end — clean dev returns 500 there and loses nothing).
+
+`load_all_for_update` is the strict loader those callers now use: it raises
+`MemoryStoreUnreadable` rather than reporting an empty store.
+"""
+
+import asyncio
+import builtins
+import json
+import os
+
+import pytest
+
+from src.memory import MemoryManager, MemoryStoreUnreadable
+
+_SEED = [
+    {"id": "m1", "text": "user prefers dark mode", "owner": "alice"},
+    {"id": "m2", "text": "user lives in Berlin", "owner": "alice"},
+    {"id": "m3", "text": "bob's cat is called Mila", "owner": "bob"},
+]
+
+
+def _seeded(tmp_path):
+    m = MemoryManager(str(tmp_path))
+    m.save([dict(e) for e in _SEED])
+    return m
+
+
+def _break_reads_of(monkeypatch, target, exc):
+    """Make open() raise `exc` for `target` only, leaving every other path alone."""
+    real_open = builtins.open
+
+    def fake_open(file, mode="r", *args, **kwargs):
+        if os.path.abspath(str(file)) == os.path.abspath(target) and "r" in mode:
+            raise exc
+        return real_open(file, mode, *args, **kwargs)
+
+    monkeypatch.setattr(builtins, "open", fake_open)
+
+
+# ── the strict loader signals, rather than reporting "empty" ──────────────
+
+def test_strict_load_raises_on_permission_error(tmp_path, monkeypatch):
+    m = _seeded(tmp_path)
+    _break_reads_of(monkeypatch, m.memory_file, PermissionError(13, "locked"))
+    with pytest.raises(MemoryStoreUnreadable):
+        m.load_all_for_update()
+
+
+def test_strict_load_raises_on_corrupt_json(tmp_path):
+    m = _seeded(tmp_path)
+    with open(m.memory_file, "w", encoding="utf-8") as f:
+        f.write('[{"id": "m1", "text": "truncated mid-writ')
+    with pytest.raises(MemoryStoreUnreadable):
+        m.load_all_for_update()
+
+
+def test_strict_load_raises_when_store_is_not_a_list(tmp_path):
+    # A file holding `{}` or `null` is not an empty store, it is a broken one.
+    m = _seeded(tmp_path)
+    with open(m.memory_file, "w", encoding="utf-8") as f:
+        json.dump({}, f)
+    with pytest.raises(MemoryStoreUnreadable):
+        m.load_all_for_update()
+
+
+def test_strict_load_returns_entries_when_healthy(tmp_path):
+    m = _seeded(tmp_path)
+    assert {e["id"] for e in m.load_all_for_update()} == {"m1", "m2", "m3"}
+
+
+def test_strict_load_returns_empty_when_file_genuinely_absent(tmp_path):
+    m = _seeded(tmp_path)
+    os.remove(m.memory_file)
+    # Absent is the one case that legitimately means "no memories yet".
+    assert m.load_all_for_update() == []
+
+
+# ── read paths stay lenient, so an unreadable store can't break chat ──────
+
+def test_read_path_still_degrades_to_empty(tmp_path, monkeypatch):
+    m = _seeded(tmp_path)
+    _break_reads_of(monkeypatch, m.memory_file, PermissionError(13, "locked"))
+    # Context injection / search must not raise; they just see nothing.
+    assert m.load_all() == []
+    assert m.load(owner="alice") == []
+
+
+# ── the actual #5673 regression: the store survives ───────────────────────
+
+def test_add_cycle_under_transient_read_error_does_not_wipe(tmp_path, monkeypatch):
+    """Mirrors routes/memory/memory_routes.py api_add_memory exactly."""
+    m = _seeded(tmp_path)
+    new_entry = m.add_entry("a brand new fact", owner="alice")
+
+    with monkeypatch.context() as mp:
+        _break_reads_of(mp, m.memory_file, PermissionError(13, "locked"))
+        with pytest.raises(MemoryStoreUnreadable):
+            all_mem = m.load_all_for_update()
+            all_mem.append(new_entry)
+            m.save(all_mem)
+
+    # Reads work again; every original memory is still there and the file was
+    # never replaced by the single new entry.
+    assert {e["id"] for e in m.load_all()} == {"m1", "m2", "m3"}
+
+
+def test_audit_merge_cannot_drop_other_tenants(tmp_path, monkeypatch):
+    """The audit path rebuilds the whole file from load_all + one owner's slice.
+
+    Reading [] there would save only the audited owner's entries and destroy
+    every other tenant's memories, so it has to fail closed too.
+    """
+    m = _seeded(tmp_path)
+    alice_slice = [e for e in _SEED if e["owner"] == "alice"]
+
+    with monkeypatch.context() as mp:
+        _break_reads_of(mp, m.memory_file, PermissionError(13, "locked"))
+        with pytest.raises(MemoryStoreUnreadable):
+            all_entries = m.load_all_for_update()
+            others = [e for e in all_entries if e.get("owner") != "alice"]
+            m.save(alice_slice + others)
+
+    assert any(e["id"] == "m3" for e in m.load_all()), "bob's memory was destroyed"
+
+
+def test_uses_bump_skips_write_when_unreadable(tmp_path, monkeypatch):
+    m = _seeded(tmp_path)
+    with monkeypatch.context() as mp:
+        _break_reads_of(mp, m.memory_file, PermissionError(13, "locked"))
+        m.increment_uses(["m1"])  # must not raise, must not write
+    assert {e["id"] for e in m.load_all()} == {"m1", "m2", "m3"}
+
+
+def test_claim_ownerless_skips_write_when_unreadable(tmp_path, monkeypatch):
+    m = _seeded(tmp_path)
+    with monkeypatch.context() as mp:
+        _break_reads_of(mp, m.memory_file, PermissionError(13, "locked"))
+        m.claim_ownerless("alice")
+    assert {e["id"] for e in m.load_all()} == {"m1", "m2", "m3"}
+
+
+# ── the add sinks users actually reach ────────────────────────────────────
+#
+# The tests above replay the read-modify-write shape. These drive the real
+# entry points end to end, because those are what #5673 reports: "remember
+# that I prefer X" in ordinary chat (src/ai_interaction.py do_manage_memory,
+# routed from src/tool_execution.py) and the built-in memory MCP server
+# (mcp_servers/memory_server.py, registered in src/builtin_mcp.py).
+#
+# They use a truncated store rather than a read error on purpose: it reads
+# fine, so nothing stops the save, which is the case that silently destroyed
+# stores. The assertion is that the file is left byte-identical — still broken,
+# but still holding the user's memories, so it can be repaired by hand.
+
+
+def _truncated_store(tmp_path):
+    """Seed a store that reads back fine but no longer parses."""
+    m = _seeded(tmp_path)
+    good = json.dumps([dict(e) for e in _SEED], indent=2)
+    with open(m.memory_file, "w", encoding="utf-8") as f:
+        f.write(good[:good.rindex("]")])  # drop the closing bracket only
+    with open(m.memory_file, "rb") as f:
+        return m, f.read()
+
+
+def _on_disk(manager) -> bytes:
+    with open(manager.memory_file, "rb") as f:
+        return f.read()
+
+
+def test_agent_memory_add_does_not_overwrite_unreadable_store(tmp_path, monkeypatch):
+    """src/ai_interaction.py do_manage_memory, action "add"."""
+    from src import ai_interaction
+
+    manager, before = _truncated_store(tmp_path)
+    monkeypatch.setattr(ai_interaction, "_memory_manager", manager)
+    monkeypatch.setattr(ai_interaction, "_memory_vector", None)
+
+    result = asyncio.run(ai_interaction.do_manage_memory("add\nuser prefers tabs"))
+
+    assert _on_disk(manager) == before, "the unreadable store was overwritten"
+    assert b"m3" in _on_disk(manager)
+    assert "error" in result, "the add reported success over an unreadable store"
+
+
+def test_mcp_memory_add_does_not_overwrite_unreadable_store(tmp_path, monkeypatch):
+    """mcp_servers/memory_server.py, action "add"."""
+    import mcp_servers.memory_server as memory_server
+
+    manager, before = _truncated_store(tmp_path)
+    monkeypatch.setattr(memory_server, "_memory_manager", manager)
+    monkeypatch.setattr(memory_server, "_memory_vector", None)
+    monkeypatch.setattr(memory_server, "_initialized", True)
+    for key in memory_server._OWNER_ENV_KEYS:
+        monkeypatch.delenv(key, raising=False)
+
+    result = asyncio.run(memory_server.call_tool(
+        "manage_memory", {"action": "add", "text": "user prefers tabs"}
+    ))
+
+    assert _on_disk(manager) == before, "the unreadable store was overwritten"
+    assert b"m3" in _on_disk(manager)
+    assert result[0].text.startswith("Error:")
+
+
+def test_native_provider_remember_does_not_overwrite_unreadable_store(tmp_path):
+    """src/memory_provider.py NativeMemoryProvider.remember.
+
+    Registered into app state in src/app_initializer.py but not yet consumed
+    outside tests, so this is the pattern held in place before it goes live.
+    """
+    from src.memory_provider import NativeMemoryProvider
+
+    manager, before = _truncated_store(tmp_path)
+    provider = NativeMemoryProvider(manager)
+
+    with pytest.raises(MemoryStoreUnreadable):
+        asyncio.run(provider.remember("user prefers tabs", owner="alice"))
+
+    assert _on_disk(manager) == before
+
+
+# ── the legacy memory.txt migration is preserved ──────────────────────────
+
+def test_corrupt_store_still_migrates_from_legacy_txt(tmp_path):
+    m = _seeded(tmp_path)
+    with open(m.memory_file, "w", encoding="utf-8") as f:
+        f.write("{ not json")
+    legacy = os.path.join(str(tmp_path), "memory.txt")
+    with open(legacy, "w", encoding="utf-8") as f:
+        f.write("recovered fact one\nrecovered fact two\n")
+
+    entries = m.load_all_for_update()
+    assert [e["text"] for e in entries] == ["recovered fact one", "recovered fact two"]

tests/test_model_helper_owner_scope.py

@@ -14,7 +14,7 @@ def _function_source(path: str, name: str) -> str:
 
 
 def test_document_ai_tidy_resolves_with_owner_scope():
-    body = _function_source("routes/document_routes.py", "ai_tidy_documents")
+    body = _function_source("routes/document/document_routes.py", "ai_tidy_documents")
     assert "resolve_task_endpoint(owner=user or None)" in body
     assert 'resolve_endpoint("default", owner=user or None)' in body
 

tests/test_search_routes_shim.py

@@ -0,0 +1,11 @@
+"""Regression test for the search route shim (slice 2j, #4082/#4071)."""
+
+import importlib
+
+import routes.search_routes as _shim_search  # noqa: F401
+
+
+def test_legacy_and_canonical_search_module_are_same_object():
+    legacy = importlib.import_module("routes.search_routes")
+    canonical = importlib.import_module("routes.search.search_routes")
+    assert legacy is canonical

tests/test_skill_format_timestamp.py

@@ -0,0 +1,58 @@
+"""Regression for issue #5697 — skill timestamps must not use ``datetime.utcnow()``.
+
+``_now_iso()`` builds the ``created`` value in skill frontmatter. ``utcnow()``
+returns a *naive* datetime and has been deprecated since Python 3.12, scheduled
+for removal. The replacement must stay timezone-aware while keeping the
+serialized ``YYYY-MM-DDTHH:MM:SSZ`` shape, so skill files written by older
+versions keep parsing.
+
+The UTC check matters on its own: a bare ``datetime.now()`` also produces the
+right shape, but emits local wall time, which would silently backdate or
+postdate skills for every user outside UTC.
+"""
+
+import os
+import re
+import time
+import warnings
+from datetime import datetime, timezone
+
+import pytest
+
+from services.memory.skill_format import _now_iso
+
+_ISO_Z = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$")
+
+
+def test_now_iso_keeps_serialized_shape():
+    assert _ISO_Z.match(_now_iso())
+
+
+def test_now_iso_emits_no_deprecation_warning():
+    with warnings.catch_warnings(record=True) as caught:
+        warnings.simplefilter("always")
+        _now_iso()
+    assert not [w for w in caught if issubclass(w.category, DeprecationWarning)]
+
+
+@pytest.mark.skipif(
+    not hasattr(time, "tzset"),
+    reason="time.tzset is unavailable on this platform",
+)
+def test_now_iso_is_utc_not_local_time():
+    """Pin UTC under a non-UTC local timezone, where the two visibly diverge."""
+    original_tz = os.environ.get("TZ")
+    os.environ["TZ"] = "Asia/Amman"  # UTC+3, never UTC
+    time.tzset()
+    try:
+        emitted = datetime.strptime(_now_iso(), "%Y-%m-%dT%H:%M:%SZ").replace(
+            tzinfo=timezone.utc
+        )
+        drift = abs((emitted - datetime.now(timezone.utc)).total_seconds())
+        assert drift < 60, f"timestamp is {drift}s off UTC — local time leaked in"
+    finally:
+        if original_tz is None:
+            os.environ.pop("TZ", None)
+        else:
+            os.environ["TZ"] = original_tz
+        time.tzset()

tests/test_startup_session_bootstrap_js.py

@@ -0,0 +1,356 @@
+"""Exercise sessions.js and startupShell.js together at the bootstrap seam.
+
+The dependency-heavy session module is copied unchanged except for redirecting
+its static imports to tiny browser stubs. The real loadSessions implementation
+and the real startup-shell coordinator then run together under Node.
+"""
+
+import json
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+
+_REPO = Path(__file__).resolve().parent.parent
+_SESSIONS = _REPO / "static" / "js" / "sessions.js"
+_SHELL_URL = (_REPO / "static" / "js" / "startupShell.js").as_uri()
+_HAS_NODE = shutil.which("node") is not None
+
+_IMPORT_REWRITES = {
+    "import Storage from './storage.js';": "import Storage from './storage.mjs';",
+    "import uiModule, { autoResize, styledPrompt } from './ui.js';": (
+        "import uiModule, { autoResize, styledPrompt } from './ui.mjs';"
+    ),
+    "import chatRenderer from './chatRenderer.js?v=20260722ctxheader1';": (
+        "import chatRenderer from './chatRenderer.mjs';"
+    ),
+    "import { providerLogo } from './providers.js';": (
+        "import { providerLogo } from './providers.mjs';"
+    ),
+    "import { initModelPicker, updateModelPicker } from './modelPicker.js?v=20260722ctxheader1';": (
+        "import { initModelPicker, updateModelPicker } from './modelPicker.mjs';"
+    ),
+    "import themeModule from './theme.js';": "import themeModule from './theme.mjs';",
+    "import spinnerModule from './spinner.js';": "import spinnerModule from './spinner.mjs';",
+}
+
+_STUBS = {
+    "storage.mjs": r"""
+const Storage = {
+  get: (key, fallback = null) => localStorage.getItem(key) ?? fallback,
+  set: (key, value) => localStorage.setItem(key, value),
+  remove: (key) => localStorage.removeItem(key),
+  getJSON: (key, fallback) => {
+    try { return JSON.parse(localStorage.getItem(key) ?? JSON.stringify(fallback)); }
+    catch (_) { return fallback; }
+  },
+  setJSON: (key, value) => localStorage.setItem(key, JSON.stringify(value)),
+};
+export default Storage;
+""",
+    "ui.mjs": r"""
+export const autoResize = () => {};
+export const styledPrompt = async () => null;
+const ui = {
+  el: (id) => document.getElementById(id),
+  showError: (message) => globalThis.__sessionErrors.push(String(message)),
+  showToast: () => {},
+  styledConfirm: async () => true,
+};
+export default ui;
+""",
+    "chatRenderer.mjs": (
+        "export default { addMessage: () => null, hideWelcomeScreen: () => {} };\n"
+    ),
+    "providers.mjs": "export const providerLogo = () => '';\n",
+    "modelPicker.mjs": (
+        "export const initModelPicker = () => {};\n"
+        "export const updateModelPicker = () => {};\n"
+    ),
+    "theme.mjs": "export default {};\n",
+    "spinner.mjs": "export default {};\n",
+}
+
+_HARNESS = r"""
+const SESSIONS_URL = 'SESSIONS_PATH';
+const SHELL_URL = 'SHELL_PATH';
+
+function makeStore() {
+  const values = new Map();
+  return {
+    getItem(key) { return values.has(key) ? values.get(key) : null; },
+    setItem(key, value) { values.set(key, String(value)); },
+    removeItem(key) { values.delete(key); },
+  };
+}
+
+function makeClassList() {
+  const values = new Set();
+  return {
+    add(...names) { names.forEach(name => values.add(name)); },
+    remove(...names) { names.forEach(name => values.delete(name)); },
+    contains(name) { return values.has(name); },
+    toggle(name, force) {
+      const enabled = force === undefined ? !values.has(name) : !!force;
+      if (enabled) values.add(name); else values.delete(name);
+      return enabled;
+    },
+  };
+}
+
+function makeWorld() {
+  const byId = new Map();
+  const frames = [];
+  const cancelledFrames = new Set();
+  const timers = [];
+  let nextFrame = 1;
+  let historyWrites = 0;
+
+  function makeElement(id = '') {
+    let html = '';
+    const element = {
+      id,
+      dataset: {},
+      style: {},
+      classList: makeClassList(),
+      children: [],
+      status: null,
+      value: '',
+      disabled: false,
+      removed: false,
+      addEventListener() {},
+      removeEventListener() {},
+      setAttribute(name, value) { this[name] = value; },
+      getAttribute(name) { return this[name] ?? null; },
+      appendChild(child) { this.children.push(child); return child; },
+      insertBefore(child) { this.children.unshift(child); return child; },
+      contains() { return false; },
+      closest() { return null; },
+      querySelector(selector) {
+        if (selector === '[data-session-list-status]') return this.status;
+        return null;
+      },
+      querySelectorAll() { return []; },
+      focus() { document.activeElement = this; },
+      remove() { this.removed = true; if (this.id) byId.delete(this.id); },
+    };
+    Object.defineProperty(element, 'innerHTML', {
+      get() { return html; },
+      set(value) {
+        html = String(value);
+        if (id === 'session-list' && html === '') {
+          const row = byId.get('session-list-loading');
+          if (row) row.remove();
+        }
+      },
+    });
+    return element;
+  }
+
+  const document = {
+    activeElement: null,
+    getElementById: (id) => byId.get(id) || null,
+    querySelector: () => null,
+    querySelectorAll: () => [],
+    createElement: (tag) => makeElement(tag),
+    createDocumentFragment: () => makeElement('fragment'),
+    addEventListener() {},
+  };
+  globalThis.document = document;
+  globalThis.localStorage = makeStore();
+  globalThis.sessionStorage = makeStore();
+  Object.defineProperty(globalThis, 'navigator', {
+    value: { platform: 'Linux' },
+    configurable: true,
+  });
+  globalThis.history = { replaceState() { historyWrites += 1; } };
+  globalThis.window = {
+    document,
+    innerWidth: 1024,
+    innerHeight: 768,
+    location: { origin: 'http://odysseus.test', hash: '', pathname: '/', href: '/' },
+    addEventListener() {},
+    removeEventListener() {},
+    chatModule: {
+      detachCurrentStream() {},
+      showWelcomeScreen() {},
+    },
+    __odysseusDefaultChat: {
+      endpoint_url: 'http://model.test',
+      model: 'test/model',
+      endpoint_id: 'endpoint-1',
+    },
+  };
+  globalThis.location = window.location;
+  globalThis.requestAnimationFrame = (fn) => {
+    const id = nextFrame++;
+    frames.push({ id, fn });
+    return id;
+  };
+  globalThis.cancelAnimationFrame = (id) => cancelledFrames.add(id);
+  globalThis.setTimeout = (fn, ms) => { timers.push({ fn, ms }); return timers.length; };
+  globalThis.clearTimeout = () => {};
+  globalThis.__sessionErrors = [];
+
+  return {
+    add(id, options = {}) {
+      const element = makeElement(id);
+      if (options.statusText !== undefined) {
+        element.status = { textContent: options.statusText };
+      }
+      if (options.value !== undefined) element.value = options.value;
+      byId.set(id, element);
+      return element;
+    },
+    paint(rounds = 1) {
+      for (let i = 0; i < rounds; i += 1) {
+        const due = frames.splice(0, frames.length);
+        for (const frame of due) {
+          if (!cancelledFrames.has(frame.id)) frame.fn();
+        }
+      }
+    },
+    runTimers() {
+      const due = timers.splice(0, timers.length);
+      for (const timer of due) timer.fn();
+    },
+    byId,
+    historyWrites: () => historyWrites,
+    resetHistoryWrites: () => { historyWrites = 0; },
+  };
+}
+
+const world = makeWorld();
+world.add('session-list');
+world.add('sessions-section');
+const message = world.add('message', { value: 'draft before seed' });
+
+const responses = [
+  {
+    ok: true,
+    status: 200,
+    json: async () => [{ id: 'existing', name: 'Existing', folder: 'Assistant', archived: false }],
+  },
+  {
+    ok: false,
+    status: 503,
+    json: async () => ({ detail: 'temporarily unavailable' }),
+  },
+];
+let fetchCount = 0;
+globalThis.fetch = async () => {
+  fetchCount += 1;
+  const response = responses.shift();
+  if (!response) throw new Error('unexpected fetch');
+  return response;
+};
+
+const sessions = await import(SESSIONS_URL + '?bootstrap');
+const shell = await import(SHELL_URL + '?bootstrap');
+
+const seeded = await sessions.loadSessions();
+world.paint(1);
+localStorage.setItem('lastSessionId', 'existing');
+message.value = 'draft must survive';
+document.activeElement = null;
+world.resetHistoryWrites();
+const loader = world.add('app-loader');
+const row = world.add('session-list-loading', { statusText: 'Loading chats…' });
+let opened = 0;
+shell.deferRouteOpener('/email', () => { opened += 1; });
+
+const hydrated = await shell.settleSessionHydration(() => sessions.loadSessions());
+const beforePaint = row.status.textContent;
+world.paint(2);
+world.runTimers();
+const staleRouteRan = shell.runDeferredRouteOpener({ sessionsSettled: true });
+
+const errorsBeforeAuth = __sessionErrors.length;
+globalThis.fetch = async () => {
+  fetchCount += 1;
+  const response = { ok: false, status: 401, json: async () => ({ detail: 'expired' }) };
+  window.location.href = '/login'; // app.js global fetch-wrapper behaviour
+  return response;
+};
+const authResult = await sessions.loadSessions();
+
+console.log(JSON.stringify({
+  seeded,
+  hydrated,
+  beforePaint,
+  afterPaint: row.status.textContent,
+  rowStillPresent: world.byId.has('session-list-loading'),
+  loaderRemoved: loader.removed,
+  opened,
+  staleRouteRan,
+  fetchCount,
+  sessionIds: sessions.getSessions().map(session => session.id),
+  pendingChat: sessions.hasPendingChat(),
+  draft: message.value,
+  lastSessionId: localStorage.getItem('lastSessionId'),
+  historyWrites: world.historyWrites(),
+  errors: __sessionErrors,
+  authResult,
+  authRedirect: window.location.href,
+  authAddedError: __sessionErrors.length !== errorsBeforeAuth,
+}));
+"""
+
+
+@pytest.fixture(scope="module")
+def results(tmp_path_factory):
+    if not _HAS_NODE:
+        pytest.skip("node is not installed")
+
+    module_dir = tmp_path_factory.mktemp("session-bootstrap-js")
+    source = _SESSIONS.read_text(encoding="utf-8")
+    for original, replacement in _IMPORT_REWRITES.items():
+        assert original in source, f"sessions import changed: {original}"
+        source = source.replace(original, replacement, 1)
+    sessions_module = module_dir / "sessions.mjs"
+    sessions_module.write_text(source, encoding="utf-8")
+    for name, stub in _STUBS.items():
+        (module_dir / name).write_text(stub, encoding="utf-8")
+
+    harness = _HARNESS.replace("SESSIONS_PATH", sessions_module.as_uri()).replace(
+        "SHELL_PATH", _SHELL_URL
+    )
+    proc = subprocess.run(
+        ["node", "--input-type=module", "-e", harness],
+        capture_output=True,
+        text=True,
+        timeout=60,
+    )
+    assert proc.returncode == 0, f"node harness failed:\n{proc.stderr}"
+    return json.loads(proc.stdout.strip().splitlines()[-1])
+
+
+def test_fulfilled_503_is_not_applied_as_an_empty_session_list(results):
+    assert results["seeded"] is True
+    assert results["hydrated"] is False
+    assert results["sessionIds"] == ["existing"]
+    assert results["pendingChat"] is False, "failure created a default direct chat"
+    assert results["draft"] == "draft must survive"
+    assert results["lastSessionId"] == "existing"
+    assert results["historyWrites"] == 0
+
+
+def test_fulfilled_503_keeps_failure_state_and_route_deferred(results):
+    assert results["beforePaint"] == "Loading chats…"
+    assert results["afterPaint"] == "Chats unavailable"
+    assert results["rowStillPresent"] is True
+    assert results["loaderRemoved"] is True
+    assert results["opened"] == 0
+    assert results["staleRouteRan"] is False
+    assert results["errors"] == [
+        "Failed to load sessions: temporarily unavailable",
+    ]
+
+
+def test_401_keeps_global_auth_redirect_contract(results):
+    assert results["authResult"] is False
+    assert results["authRedirect"] == "/login"
+    assert results["authAddedError"] is False
+    assert results["sessionIds"] == ["existing"]

tests/test_startup_shell_js.py

@@ -0,0 +1,377 @@
+"""Pin the startup shell contract (static/js/startupShell.js).
+
+Driven through `node --input-type=module` against a stub DOM and a manually
+pumped frame/timer clock, so the real module runs without a browser (same
+approach as test_composer_arrow_up_recall_js.py). Skips when `node` is absent.
+
+Locks in the behaviour #5926 asks for: the shell is revealed one paint after
+wiring and does not wait on /api/sessions; the loader node survives hydration
+as a startup sentinel but is always retired once hydration settles; the sidebar
+owns its own loading/failure row and a successful zero-session render never
+shows a false failure; and a URL route opens only after the data it actually
+needs is authoritatively available.
+"""
+import json
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+_REPO = Path(__file__).resolve().parent.parent
+_MODULE = _REPO / "static" / "js" / "startupShell.js"
+_MODULE_URL = _MODULE.as_uri()
+_HAS_NODE = shutil.which("node") is not None
+
+_HARNESS = r"""
+const MODULE_URL = 'MODULE_PATH';
+
+// ── Stub DOM + a clock we pump by hand ────────────────────────────────────
+function makeWorld() {
+  const byId = new Map();
+  const frames = [];
+  const timers = [];
+  const world = {
+    byId,
+    waveStops: 0,
+    addElement(id, { statusText = null } = {}) {
+      const el = {
+        id,
+        dataset: {},
+        style: {},
+        attrs: {},
+        removed: false,
+        status: null,
+        setAttribute(k, v) { this.attrs[k] = v; },
+        getAttribute(k) { return this.attrs[k]; },
+        remove() { this.removed = true; byId.delete(this.id); },
+        querySelector(sel) {
+          return sel === '[data-session-list-status]' ? this.status : null;
+        },
+      };
+      if (statusText !== null) el.status = { textContent: statusText };
+      byId.set(id, el);
+      return el;
+    },
+    // One "paint" = one round of already-queued rAF callbacks. afterNextPaint
+    // chains two, so a committed paint takes two rounds.
+    paint(rounds = 1) {
+      for (let i = 0; i < rounds; i++) {
+        const due = frames.splice(0, frames.length);
+        for (const fn of due) fn();
+      }
+    },
+    runTimers() {
+      const due = timers.splice(0, timers.length);
+      for (const t of due) t.fn();
+    },
+    pendingTimers() { return timers.length; },
+  };
+  globalThis.document = { getElementById: (id) => byId.get(id) || null };
+  globalThis.window = { __odysseusLoaderWaveStop: () => { world.waveStops += 1; } };
+  globalThis.requestAnimationFrame = (fn) => { frames.push(fn); return frames.length; };
+  globalThis.setTimeout = (fn, ms) => { timers.push({ fn, ms }); return timers.length; };
+  return world;
+}
+
+// Fresh module instance per case so deferred-route state cannot leak.
+let _instance = 0;
+async function loadModule() {
+  _instance += 1;
+  return import(MODULE_URL + '?case=' + _instance);
+}
+
+function loaderSnapshot(loader) {
+  return {
+    revealed: loader.dataset.shellRevealed === 'true',
+    opacity: loader.style.opacity ?? null,
+    pointerEvents: loader.style.pointerEvents ?? null,
+    ariaHidden: loader.getAttribute('aria-hidden') ?? null,
+    removed: loader.removed,
+  };
+}
+
+const cases = {};
+
+cases.reveal_waits_one_paint_then_keeps_node = async () => {
+  const w = makeWorld();
+  const loader = w.addElement('app-loader');
+  const shell = await loadModule();
+  shell.revealApplicationShellAfterPaint();
+  const beforePaint = loaderSnapshot(loader);
+  w.paint(1);
+  const afterOneFrame = loaderSnapshot(loader);
+  w.paint(1);
+  return {
+    beforePaint,
+    afterOneFrame,
+    afterPaint: loaderSnapshot(loader),
+    waveStops: w.waveStops,
+    stillInDocument: w.byId.has('app-loader'),
+  };
+};
+
+cases.reveal_is_idempotent = async () => {
+  const w = makeWorld();
+  const loader = w.addElement('app-loader');
+  const shell = await loadModule();
+  shell.revealApplicationShellAfterPaint();
+  shell.revealApplicationShellAfterPaint();
+  w.paint(2);
+  shell.revealApplicationShellAfterPaint();
+  w.paint(2);
+  return { waveStops: w.waveStops, snapshot: loaderSnapshot(loader) };
+};
+
+cases.remove_retires_the_loader_node = async () => {
+  const w = makeWorld();
+  const loader = w.addElement('app-loader');
+  const shell = await loadModule();
+  shell.removeApplicationLoader();
+  const beforeTimers = loaderSnapshot(loader);
+  w.runTimers();
+  return { beforeTimers, afterTimers: loaderSnapshot(loader) };
+};
+
+cases.failed_hydration_marks_sidebar_row = async () => {
+  const w = makeWorld();
+  w.addElement('app-loader');
+  const row = w.addElement('session-list-loading', { statusText: 'Loading chats…' });
+  const shell = await loadModule();
+  await shell.settleSessionHydration(() => Promise.reject(new Error('boom')));
+  const beforePaint = row.status.textContent;
+  w.paint(2);
+  w.runTimers();
+  return {
+    beforePaint,
+    afterPaint: row.status.textContent,
+    loaderRemoved: !w.byId.has('app-loader'),
+  };
+};
+
+// A successful load with zero sessions must not schedule a failure write.
+cases.zero_session_success_shows_no_failure = async () => {
+  const w = makeWorld();
+  w.addElement('app-loader');
+  const row = w.addElement('session-list-loading', { statusText: 'Loading chats…' });
+  const shell = await loadModule();
+  await shell.settleSessionHydration(() => Promise.resolve(true));
+  w.paint(1);
+  row.remove();            // renderSessionList() clearing #session-list
+  w.paint(1);
+  return { statusText: row.status.textContent, rowRemoved: row.removed };
+};
+
+// The whole point is getting /api/sessions off the critical path, not later.
+cases.hydration_starts_synchronously = async () => {
+  const w = makeWorld();
+  w.addElement('app-loader');
+  const shell = await loadModule();
+  let started = false;
+  const done = shell.settleSessionHydration(() => { started = true; return Promise.resolve(true); });
+  const startedBeforeAwait = started;
+  await done;
+  return { startedBeforeAwait };
+};
+
+cases.synchronous_load_failure_still_settles = async () => {
+  const w = makeWorld();
+  w.addElement('app-loader');
+  const row = w.addElement('session-list-loading', { statusText: 'Loading chats…' });
+  const shell = await loadModule();
+  let opened = 0;
+  shell.deferRouteOpener('/email', () => { opened += 1; });
+  let threw = false;
+  let succeeded = true;
+  try {
+    succeeded = await shell.settleSessionHydration(() => { throw new Error('module blew up'); });
+  } catch (_) { threw = true; }
+  w.paint(2);
+  w.runTimers();
+  return {
+    threw,
+    succeeded,
+    opened,
+    statusText: row.status.textContent,
+    loaderRemoved: !w.byId.has('app-loader'),
+    ranAfterFailure: shell.runDeferredRouteOpener({ sessionsSettled: true }),
+  };
+};
+
+cases.route_without_session_data_opens_before_hydration = async () => {
+  const w = makeWorld();
+  w.addElement('app-loader');
+  const shell = await loadModule();
+  let opened = 0;
+  shell.deferRouteOpener('/notes', () => { opened += 1; });
+  const ranEarly = shell.runDeferredRouteOpener();
+  const openedAfterEarly = opened;
+  const ranAgain = shell.runDeferredRouteOpener({ sessionsSettled: true });
+  return { ranEarly, openedAfterEarly, ranAgain, opened };
+};
+
+cases.route_with_session_data_waits_for_hydration = async () => {
+  const w = makeWorld();
+  w.addElement('app-loader');
+  const shell = await loadModule();
+  let opened = 0;
+  shell.deferRouteOpener('/email', () => { opened += 1; });
+  const ranEarly = shell.runDeferredRouteOpener();
+  const openedAfterEarly = opened;
+  const succeeded = await shell.settleSessionHydration(() => Promise.resolve(true));
+  return {
+    ranEarly,
+    openedAfterEarly,
+    openedAfterHydration: opened,
+    succeeded,
+    needsSessions: [shell.routeNeedsSessionData('/email'), shell.routeNeedsSessionData('/notes')],
+  };
+};
+
+cases.missing_session_module_keeps_route_deferred = async () => {
+  const w = makeWorld();
+  w.addElement('app-loader');
+  const row = w.addElement('session-list-loading', { statusText: 'Loading chats…' });
+  const shell = await loadModule();
+  let opened = 0;
+  shell.deferRouteOpener('/email', () => { opened += 1; });
+  const succeeded = await shell.settleSessionHydration(null);
+  w.paint(2);
+  w.runTimers();
+  return {
+    opened,
+    succeeded,
+    statusText: row.status.textContent,
+    loaderRemoved: !w.byId.has('app-loader'),
+    ranAfterFailure: shell.runDeferredRouteOpener({ sessionsSettled: true }),
+  };
+};
+
+cases.throwing_route_opener_is_contained = async () => {
+  const w = makeWorld();
+  w.addElement('app-loader');
+  const shell = await loadModule();
+  shell.deferRouteOpener('/notes', () => { throw new Error('opener blew up'); });
+  let threw = false;
+  let ran = false;
+  try { ran = shell.runDeferredRouteOpener(); } catch (_) { threw = true; }
+  return { threw, ran, ranAgain: shell.runDeferredRouteOpener({ sessionsSettled: true }) };
+};
+
+const out = {};
+for (const [name, fn] of Object.entries(cases)) out[name] = await fn();
+console.log(JSON.stringify(out));
+""".replace("MODULE_PATH", _MODULE_URL)
+
+
+@pytest.fixture(scope="module")
+def results():
+    if not _HAS_NODE:
+        pytest.skip("node is not installed")
+    proc = subprocess.run(
+        ["node", "--input-type=module", "-e", _HARNESS],
+        capture_output=True,
+        text=True,
+        timeout=60,
+    )
+    assert proc.returncode == 0, f"node harness failed:\n{proc.stderr}"
+    return json.loads(proc.stdout.strip().splitlines()[-1])
+
+
+def test_module_exists():
+    assert _MODULE.is_file(), f"missing {_MODULE}"
+
+
+def test_shell_is_revealed_one_paint_after_wiring(results):
+    r = results["reveal_waits_one_paint_then_keeps_node"]
+    assert r["beforePaint"]["revealed"] is False, "revealed before any frame ran"
+    assert r["afterOneFrame"]["revealed"] is False, "revealed before the paint committed"
+    assert r["afterPaint"] == {
+        "revealed": True,
+        "opacity": "0",
+        "pointerEvents": "none",
+        "ariaHidden": "true",
+        "removed": False,
+    }
+    assert r["waveStops"] == 1, "loader wave interval kept running after reveal"
+
+
+def test_revealed_loader_stays_as_startup_sentinel(results):
+    # sessions.js / sidebar-layout.js read #app-loader as "startup in progress".
+    r = results["reveal_waits_one_paint_then_keeps_node"]
+    assert r["stillInDocument"] is True
+    assert r["afterPaint"]["removed"] is False
+
+
+def test_reveal_is_idempotent(results):
+    r = results["reveal_is_idempotent"]
+    assert r["waveStops"] == 1, "reveal ran its side effects more than once"
+    assert r["snapshot"]["revealed"] is True
+
+
+def test_loader_node_is_retired_after_the_fade(results):
+    r = results["remove_retires_the_loader_node"]
+    assert r["beforeTimers"]["revealed"] is True, "removal should hide immediately"
+    assert r["beforeTimers"]["removed"] is False, "removal should wait for the fade"
+    assert r["afterTimers"]["removed"] is True, "loader node outlived hydration"
+
+
+def test_failed_session_load_marks_the_sidebar_row(results):
+    r = results["failed_hydration_marks_sidebar_row"]
+    assert r["beforePaint"] == "Loading chats…", "failure written before the render frame"
+    assert r["afterPaint"] == "Chats unavailable"
+    assert r["loaderRemoved"] is True, "a failed load must still free the shell"
+
+
+def test_zero_session_success_never_shows_a_failure(results):
+    r = results["zero_session_success_shows_no_failure"]
+    assert r["rowRemoved"] is True
+    assert r["statusText"] == "Loading chats…", "false 'Chats unavailable' on empty success"
+
+
+def test_hydration_request_starts_synchronously(results):
+    r = results["hydration_starts_synchronously"]
+    assert r["startedBeforeAwait"] is True, "/api/sessions start was deferred a microtask"
+
+
+def test_synchronous_load_failure_still_settles(results):
+    r = results["synchronous_load_failure_still_settles"]
+    assert r["threw"] is False, "a throwing loadSessions must not escape"
+    assert r["succeeded"] is False
+    assert r["opened"] == 0, "session-dependent route opened without session data"
+    assert r["ranAfterFailure"] is False, "failed startup left a stale route opener"
+    assert r["statusText"] == "Chats unavailable"
+    assert r["loaderRemoved"] is True
+
+
+def test_route_needing_no_session_data_opens_before_hydration(results):
+    r = results["route_without_session_data_opens_before_hydration"]
+    assert r["ranEarly"] is True, "/notes waited on /api/sessions it does not read"
+    assert r["openedAfterEarly"] == 1
+    assert r["ranAgain"] is False, "route opener fired twice"
+    assert r["opened"] == 1
+
+
+def test_route_needing_session_data_waits_for_hydration(results):
+    r = results["route_with_session_data_waits_for_hydration"]
+    assert r["ranEarly"] is False, "/email opened before the session list was there"
+    assert r["openedAfterEarly"] == 0
+    assert r["openedAfterHydration"] == 1
+    assert r["succeeded"] is True
+    assert r["needsSessions"] == [True, False]
+
+
+def test_missing_session_module_still_settles_without_opening_data_route(results):
+    r = results["missing_session_module_keeps_route_deferred"]
+    assert r["succeeded"] is False
+    assert r["opened"] == 0, "route opened without the session module it depends on"
+    assert r["ranAfterFailure"] is False, "missing module left a stale route opener"
+    assert r["statusText"] == "Chats unavailable"
+    assert r["loaderRemoved"] is True
+
+
+def test_throwing_route_opener_is_contained(results):
+    r = results["throwing_route_opener_is_contained"]
+    assert r["threw"] is False
+    assert r["ran"] is True
+    assert r["ranAgain"] is False, "a failed opener must not be retried"

tests/test_tool_parsing_bare_end_marker.py

@@ -0,0 +1,96 @@
+"""Regression: the Qwen bare-marker scrub must not eat a lone `end` (#5547).
+
+`_QWEN_BARE_MARKER_RE` cleans Qwen turn markers that leak into content. Its
+`end` branch was `\\|?end\\|?` — both pipes optional — so it also matched a bare
+`end` surrounded by whitespace and replaced it with a space. Any message
+containing Ruby, Lua or shell code that closes a block with a lone `end` had
+those lines silently deleted, in the stored text and in the rendered message.
+
+Requiring at least one pipe keeps every real marker (`|end`, `end|`, `|end|`,
+`/|end|`) stripping as before. The same pattern is duplicated in
+static/js/chatRenderer.js, so the JS copy is checked here too — the two must
+not drift.
+"""
+import json
+import re
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+import src.agent_tools  # noqa: F401  (break agent_tools<->tool_parsing import cycle)
+from src.tool_parsing import strip_tool_blocks
+
+_REPO = Path(__file__).resolve().parent.parent
+_CHAT_RENDERER = _REPO / "static" / "js" / "chatRenderer.js"
+
+# Inputs that must survive untouched, and the substring that proves they did.
+KEPT = [
+    ("loop do\n    puts \"yo\"\nend\n", "\nend"),          # the reported Ruby case
+    ("if x then\nend", "\nend"),
+    ("function f()\nend\n", "\nend"),
+    ("a end b", "a end b"),
+    ("append end", "append end"),
+    ("END", "END"),
+    ("\nEnd\n", "End"),
+]
+
+# Real markers — at least one pipe, plus the role word — with the exact output
+# they must still produce. Asserted as equality rather than "marker not in out"
+# so narrowing the pattern can't pass by deleting more than it should.
+STRIPPED = [
+    ("a |end| b", "a  b"),
+    ("a /|end| b", "a  b"),
+    ("a |end b", "a  b"),
+    ("a end| b", "a  b"),
+    ("x assistant y", "x  y"),
+]
+
+
+@pytest.mark.parametrize("text,kept", KEPT)
+def test_bare_end_survives_stripping(text, kept):
+    assert kept in strip_tool_blocks(text)
+
+
+@pytest.mark.parametrize("text,expected", STRIPPED)
+def test_piped_end_markers_are_still_stripped(text, expected):
+    assert strip_tool_blocks(text) == expected
+
+
+def test_bare_end_inside_a_fenced_block_survives():
+    """The scrub runs over the whole message, fenced regions included."""
+    out = strip_tool_blocks("Here:\n```ruby\nloop do\n  puts 1\nend\n```\nDone.")
+    assert "\nend\n" in out
+
+
+def _js_bare_marker_regex_source():
+    src = _CHAT_RENDERER.read_text(encoding="utf-8")
+    m = re.search(r"^const QWEN_BARE_MARKER_RE = (/.*/[gimsuy]*);$", src, re.MULTILINE)
+    assert m, "QWEN_BARE_MARKER_RE literal not found in chatRenderer.js"
+    return m.group(1)
+
+
+def test_js_copy_of_the_pattern_matches_the_python_one():
+    """Guard the duplication: the JS branch must require a pipe too."""
+    if shutil.which("node") is None:
+        pytest.skip("node binary not on PATH")
+
+    cases = [text for text, _ in KEPT] + [text for text, _ in STRIPPED]
+    script = (
+        "const RE = %s;\n"
+        "const cases = JSON.parse(process.argv[1]);\n"
+        "console.log(JSON.stringify(cases.map(c => c.replace(RE, ' '))));"
+        % _js_bare_marker_regex_source()
+    )
+    result = subprocess.run(
+        ["node", "--input-type=module", "-e", script, json.dumps(cases)],
+        cwd=_REPO, capture_output=True, timeout=15, text=True,
+    )
+    assert result.returncode == 0, f"node failed:\n{result.stderr}"
+    got = json.loads(result.stdout.splitlines()[-1])
+
+    for (text, kept), out in zip(KEPT, got):
+        assert kept in out, f"JS regex dropped {kept!r} from {text!r}"
+    for (text, expected), out in zip(STRIPPED, got[len(KEPT):]):
+        assert out == expected, f"JS regex: {text!r} -> {out!r}, expected {expected!r}"

tests/test_tts_service_enforce_cache_limit.py

@@ -0,0 +1,97 @@
+import os
+import time
+from pathlib import Path
+import pytest
+
+# Adjust the import path if your file is directly in ./services instead of ./services/tts
+from services.tts.tts_service import TTSService
+
+def test_cache_under_limit(tmp_path, monkeypatch):
+    """Test that writing a file under the size limit does not trigger eviction."""
+    # Set a tiny limit: 100 bytes
+    monkeypatch.setenv("ODYSSEUS_TTS_CACHE_MAX_BYTES", "100")
+    
+    # Initialize service with pytest's temporary directory
+    service = TTSService(cache_dir=str(tmp_path))
+    
+    # Write a 40-byte file (under the 100-byte limit)
+    service._put_cache("test_key", b"x" * 40)
+    
+    # Verify the file was written and nothing was deleted
+    files = list(tmp_path.glob("*.*"))
+    assert len(files) == 1
+    assert sum(f.stat().st_size for f in files) == 40
+
+def test_cache_exceeds_limit_triggers_eviction(tmp_path, monkeypatch):
+    """Test that exceeding the limit evicts the oldest files down to 80% capacity."""
+    # Set limit to 100 bytes. 80% target capacity will be 80 bytes.
+    monkeypatch.setenv("ODYSSEUS_TTS_CACHE_MAX_BYTES", "100")
+    service = TTSService(cache_dir=str(tmp_path))
+    
+    # 1. Setup: Manually create two older files (40 bytes each)
+    file1 = tmp_path / "oldest.wav"
+    file2 = tmp_path / "middle.wav"
+    
+    file1.write_bytes(b"a" * 40)
+    file2.write_bytes(b"b" * 40)
+    
+    # Spoof timestamps so file1 is explicitly older than file2
+    now = time.time()
+    os.utime(file1, (now - 100, now - 100)) # 100 seconds ago
+    os.utime(file2, (now - 50, now - 50))   # 50 seconds ago
+    
+    # 2. Action: Write a 3rd file using the service method (40 bytes)
+    # Total cache is now 120 bytes, which exceeds 100.
+    # It should delete oldest (file1) to drop to 80 bytes (which matches the 80% target).
+    service._put_cache("newest", b"c" * 40)
+    
+    # 3. Assertions
+    # The newest file should exist (saved as .wav because it lacks MP3 magic bytes)
+    newest_file = tmp_path / "newest.wav"
+    
+    assert not file1.exists(), "The oldest file should have been evicted."
+    assert file2.exists(), "The middle file should still exist."
+    assert newest_file.exists(), "The newest file should have been saved."
+    
+    # Verify the final directory size is <= 80 bytes
+    total_size = sum(f.stat().st_size for f in tmp_path.glob("*.*"))
+    assert total_size <= 80
+
+def test_cache_limit_disabled(tmp_path, monkeypatch):
+    """Test that setting max bytes to 0 disables eviction."""
+    monkeypatch.setenv("ODYSSEUS_TTS_CACHE_MAX_BYTES", "0")
+    service = TTSService(cache_dir=str(tmp_path))
+    
+    # Write 3 large files that would normally trigger eviction
+    service._put_cache("file1", b"x" * 1000)
+    service._put_cache("file2", b"x" * 1000)
+    service._put_cache("file3", b"x" * 1000)
+    
+    # Ensure nothing was deleted
+    files = list(tmp_path.glob("*.*"))
+    assert len(files) == 3
+    assert sum(f.stat().st_size for f in files) == 3000
+
+def test_cache_eviction_handles_unlink_error_gracefully(tmp_path, monkeypatch):
+    """Test that if unlinking a file fails, _put_cache still succeeds without raising."""
+    service = TTSService(cache_dir=str(tmp_path))
+    service.max_cache_bytes = 50
+
+    # Create a file to evict
+    old_file = tmp_path / "old.wav"
+    old_file.write_bytes(b"x" * 40)
+
+    # Monkeypatch unlink on Path objects to simulate a PermissionError / file-lock failure
+    def mock_unlink(self_path):
+        raise OSError("Permission denied / file locked")
+
+    monkeypatch.setattr(Path, "unlink", mock_unlink)
+
+    # Writing a new file triggers eviction which encounters the mocked unlink error
+    try:
+        service._put_cache("new_key", b"y" * 40)
+    except Exception as e:
+        pytest.fail(f"_put_cache raised an exception during failed eviction: {e}")
+
+    # The new file should still be written successfully
+    assert (tmp_path / "new_key.wav").exists()

tests/test_vault_routes_shim.py

@@ -0,0 +1,11 @@
+"""Regression test for the vault route shim (slice 2k, #4082/#4071)."""
+
+import importlib
+
+import routes.vault_routes as _shim_vault  # noqa: F401
+
+
+def test_legacy_and_canonical_vault_module_are_same_object():
+    legacy = importlib.import_module("routes.vault_routes")
+    canonical = importlib.import_module("routes.vault.vault_routes")
+    assert legacy is canonical

tests/test_vision_owner_scope.py

@@ -88,7 +88,7 @@ def test_request_vision_call_sites_pass_owner():
     chat_source = (ROOT / "src" / "chat_handler.py").read_text()
     processor_source = (ROOT / "src" / "document_processor.py").read_text()
     upload_source = (ROOT / "routes" / "upload_routes.py").read_text()
-    document_source = (ROOT / "routes" / "document_routes.py").read_text()
+    document_source = (ROOT / "routes" / "document" / "document_routes.py").read_text()
     gallery_source = (ROOT / "routes" / "gallery" / "gallery_routes.py").read_text()
     memory_source = (ROOT / "routes" / "memory" / "memory_routes.py").read_text()
 

tests/test_webhook_routes_shim.py

@@ -0,0 +1,11 @@
+"""Regression test for the webhook route shim (slice 2l, #4082/#4071)."""
+
+import importlib
+
+import routes.webhook_routes as _shim_webhook  # noqa: F401
+
+
+def test_legacy_and_canonical_webhook_module_are_same_object():
+    legacy = importlib.import_module("routes.webhook_routes")
+    canonical = importlib.import_module("routes.webhook.webhook_routes")
+    assert legacy is canonical