docs(discovery): publish provisional feature baseline

* fix(rag): skip hidden and junk directories when indexing (#5559)

index_personal_documents walked the whole tree with no pruning, so
pointing RAG at a real-world folder silently swept in .obsidian/ plugin
JS, .git/ internals, node_modules/, and __pycache__/ — multiplying
indexing time and polluting retrieval with junk chunks.

Prune hidden directories and well-known junk directories from the walk,
and skip hidden files. The explicitly passed root is exempt, so a user
who deliberately indexes a hidden directory still gets its contents.

* fix(rag): prune hidden/junk dirs in the keyword index too, via a shared helper

The #5559 fix pruned only VectorRAG.index_personal_documents (the vector index).
The parallel keyword index built by PersonalDocsManager.refresh_index ->
load_personal_index walked the same tree unpruned, so .obsidian/, .git/,
node_modules/ etc. still swept into keyword retrieval and the file listing —
the 'end-to-end' guarantee was only half true.

Single-source the pruning policy in src/index_walk (prune_index_dirs +
is_indexable_file) and use it from both walkers so they cannot drift again.
The junk-dir match is now case-insensitive, so a Node_Modules on a
case-insensitive filesystem is pruned too.

Tests: keyword-path regressions covering hidden/junk dirs, hidden files, junk
at depth (not just top level), case-insensitive junk, and the explicit-hidden-
root exemption. The existing vector tests still pass against the shared helper.

.env.example

@@ -130,6 +130,27 @@ SEARXNG_INSTANCE=http://localhost:8080
 # FASTEMBED_MODEL=sentence-transformers/all-MiniLM-L6-v2
 # FASTEMBED_CACHE_PATH=  # defaults to ~/.cache/fastembed
 
+# ============================================================
+# Google OAuth2 (Google Workspace / .edu email accounts)
+# ============================================================
+# Required to use the "Connect with Google" OAuth flow in email account setup.
+# Create credentials at: console.cloud.google.com → APIs & Services → Credentials
+#   1. Enable the Gmail API for your project.
+#   2. Configure the OAuth consent screen (User Type: Internal for Workspace orgs).
+#      Add scopes: https://mail.google.com/ and email.
+#   3. Create an OAuth 2.0 Client ID (type: Web application).
+#      Add your redirect URI: http://localhost:7000/api/email/oauth/google/callback
+#      (replace host/port for hosted installs).
+#   4. Copy the Client ID and Client Secret below.
+#
+# GOOGLE_OAUTH_CLIENT_ID=your-client-id.apps.googleusercontent.com
+# GOOGLE_OAUTH_CLIENT_SECRET=replace-with-client-secret
+#
+# Set this explicitly for HTTPS, reverse-proxy, or hosted deployments. The
+# value must exactly match an authorized redirect URI in the Google client.
+# Local HTTP setups may use the callback URL inferred by the application.
+# GOOGLE_OAUTH_REDIRECT_URI=https://your-domain.com/api/email/oauth/google/callback
+
 # ============================================================
 # Misc
 # ============================================================

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -6,22 +6,22 @@ body:
   - type: markdown
     attributes:
       value: |
-        **Before submitting:** search [open issues](https://github.com/pewdiepie-archdaemon/odysseus/issues)
-        and [discussions](https://github.com/pewdiepie-archdaemon/odysseus/discussions) first.
+        **Before submitting:** search [open issues](https://github.com/odysseus-dev/odysseus/issues)
+        and [discussions](https://github.com/odysseus-dev/odysseus/discussions) first.
         Duplicate reports slow things down.
 
         For security vulnerabilities, **do not open a public issue** —
-        use [GitHub Security Advisories](https://github.com/pewdiepie-archdaemon/odysseus/security/advisories/new)
-        and read [SECURITY.md](https://github.com/pewdiepie-archdaemon/odysseus/blob/main/SECURITY.md) first.
+        use [GitHub Security Advisories](https://github.com/odysseus-dev/odysseus/security/advisories/new)
+        and read [SECURITY.md](https://github.com/odysseus-dev/odysseus/blob/main/SECURITY.md) first.
 
   - type: checkboxes
     id: prerequisites
     attributes:
       label: Prerequisites
       options:
-        - label: I searched [open issues](https://github.com/pewdiepie-archdaemon/odysseus/issues?q=is%3Aissue+is%3Aopen) and [discussions](https://github.com/pewdiepie-archdaemon/odysseus/discussions) and did not find an existing report of this bug.
+        - label: I searched [open issues](https://github.com/odysseus-dev/odysseus/issues?q=is%3Aissue+is%3Aopen) and [discussions](https://github.com/odysseus-dev/odysseus/discussions) and did not find an existing report of this bug.
           required: true
-        - label: This is **not** a security vulnerability. (Vulnerabilities go to [GitHub Security Advisories](https://github.com/pewdiepie-archdaemon/odysseus/security/advisories/new) — see [SECURITY.md](https://github.com/pewdiepie-archdaemon/odysseus/blob/main/SECURITY.md).)
+        - label: This is **not** a security vulnerability. (Vulnerabilities go to [GitHub Security Advisories](https://github.com/odysseus-dev/odysseus/security/advisories/new) — see [SECURITY.md](https://github.com/odysseus-dev/odysseus/blob/main/SECURITY.md).)
           required: true
         - label: I am running the latest code from the `dev` branch (the default branch you get on clone, where fixes land first) and the bug still reproduces there. Please `git pull` the latest `dev` before filing.
           required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -1,13 +1,13 @@
 blank_issues_enabled: false
 contact_links:
   - name: Question / Need Help
-    url: https://github.com/pewdiepie-archdaemon/odysseus/discussions/categories/q-a
+    url: https://github.com/odysseus-dev/odysseus/discussions/categories/q-a
     about: Ask how-to questions, setup help, and model configuration questions here. Issues are for confirmed bugs and concrete proposals only.
 
   - name: Idea or Suggestion
-    url: https://github.com/pewdiepie-archdaemon/odysseus/discussions/categories/ideas
+    url: https://github.com/odysseus-dev/odysseus/discussions/categories/ideas
     about: Discuss ideas and gauge interest before opening a formal feature request. If there is already a discussion, link it in your feature request.
 
   - name: Security Vulnerability
-    url: https://github.com/pewdiepie-archdaemon/odysseus/security/advisories/new
+    url: https://github.com/odysseus-dev/odysseus/security/advisories/new
     about: Report vulnerabilities privately via GitHub Security Advisories — never as a public issue. Read SECURITY.md before reporting.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -6,22 +6,22 @@ body:
   - type: markdown
     attributes:
       value: |
-        **Before submitting:** search [open issues](https://github.com/pewdiepie-archdaemon/odysseus/issues)
-        and [discussions](https://github.com/pewdiepie-archdaemon/odysseus/discussions) first.
-        Feature requests that duplicate [ROADMAP.md](https://github.com/pewdiepie-archdaemon/odysseus/blob/main/ROADMAP.md)
+        **Before submitting:** search [open issues](https://github.com/odysseus-dev/odysseus/issues)
+        and [discussions](https://github.com/odysseus-dev/odysseus/discussions) first.
+        Feature requests that duplicate [ROADMAP.md](https://github.com/odysseus-dev/odysseus/blob/main/ROADMAP.md)
         or an existing open issue will be closed as duplicates.
 
         If your idea needs community input before it becomes a concrete proposal,
-        start a [discussion](https://github.com/pewdiepie-archdaemon/odysseus/discussions/categories/ideas) instead.
+        start a [discussion](https://github.com/odysseus-dev/odysseus/discussions/categories/ideas) instead.
 
   - type: checkboxes
     id: prerequisites
     attributes:
       label: Prerequisites
       options:
-        - label: I searched [open issues](https://github.com/pewdiepie-archdaemon/odysseus/issues?q=is%3Aissue+is%3Aopen) and this has not already been proposed.
+        - label: I searched [open issues](https://github.com/odysseus-dev/odysseus/issues?q=is%3Aissue+is%3Aopen) and this has not already been proposed.
           required: true
-        - label: I searched [discussions](https://github.com/pewdiepie-archdaemon/odysseus/discussions) and this is not already being debated there.
+        - label: I searched [discussions](https://github.com/odysseus-dev/odysseus/discussions) and this is not already being debated there.
           required: true
         - label: This is a concrete, actionable proposal — not a vague "it would be nice if..." request.
           required: true

.github/pull_request_template.md

@@ -24,7 +24,7 @@ Fixes #
 
 ## Checklist
 
-- [ ] I searched [open issues](https://github.com/pewdiepie-archdaemon/odysseus/issues) and [open PRs](https://github.com/pewdiepie-archdaemon/odysseus/pulls) — this is not a duplicate.
+- [ ] I searched [open issues](https://github.com/odysseus-dev/odysseus/issues) and [open PRs](https://github.com/odysseus-dev/odysseus/pulls) — this is not a duplicate.
 - [ ] This PR targets `dev`
 - [ ] My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
 - [ ] I actually ran the app (`docker compose up` or `uvicorn app:app`) and verified the change works end-to-end. Type-checks and unit tests are not enough.

.github/workflows/codeql.yml

@@ -0,0 +1,41 @@
+name: CodeQL
+
+# Advanced setup so CodeQL also runs on pull requests (including from forks),
+# surfacing findings before merge instead of only after a change lands on dev.
+on:
+  push:
+    branches: [dev, main]
+  pull_request:
+    branches: [dev]
+  schedule:
+    - cron: "17 3 * * 1"
+
+permissions:
+  contents: read
+
+jobs:
+  analyze:
+    name: Analyze (${{ matrix.language }})
+    runs-on: ubuntu-latest
+    permissions:
+      security-events: write
+      actions: read
+      contents: read
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [actions, javascript-typescript, python]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        with:
+          persist-credentials: false
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e  # v4.36.2
+        with:
+          languages: ${{ matrix.language }}
+          build-mode: none
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e  # v4.36.2
+        with:
+          category: "/language:${{ matrix.language }}"

CONTRIBUTING.md

@@ -25,7 +25,7 @@ End-users cloning the repo will land on `dev` by default. To run the curated/sta
 Docker is the recommended path for normal testing:
 
 ```bash
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 cp .env.example .env
 docker compose up -d --build

Dockerfile

@@ -16,7 +16,8 @@ FROM python:3.14-slim
 # downloads, and serves from Docker installs.
 # git/cmake are required when Cookbook builds llama.cpp on first llama.cpp
 # launch inside Docker.
-# nodejs/npm provide npx for the optional built-in Browser MCP server.
+# nodejs/npm provide npx for the built-in Browser MCP server.
+# chromium provides the actual browser binary used by that MCP server.
 # gosu lets the entrypoint drop privileges cleanly so signals still reach
 # uvicorn directly (no extra shell layer like `su`/`sudo` would add).
 RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -26,6 +27,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     git \
     nodejs \
     npm \
+    chromium \
     tmux \
     openssh-client \
     gosu \
@@ -54,7 +56,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 # /var/run/docker.sock mount). The Debian `docker.io` package ships
 # dockerd but not the client binary on slim, so grab the static client
 # tarball from download.docker.com instead.
-ARG DOCKER_CLI_VERSION=27.5.1
+ARG DOCKER_CLI_VERSION=29.6.2
 RUN ARCH="$(dpkg --print-architecture)" \
     && case "$ARCH" in \
          amd64) DARCH=x86_64 ;; \

README.md

@@ -25,10 +25,10 @@
 
 ## Quick Start
 
-> `dev` is the default branch and gets the newest changes first. Use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main) if you want the more curated branch.
+> `dev` is the default branch and gets the newest changes first. Use [`main`](https://github.com/odysseus-dev/odysseus/tree/main) if you want the more curated branch.
 
 ```bash
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 cp .env.example .env
 docker compose up -d --build
@@ -63,11 +63,11 @@ Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled
 
 ## Star History
 
-<a href="https://www.star-history.com/?repos=pewdiepie-archdaemon%2Fodysseus&type=date&legend=top-left">
+<a href="https://www.star-history.com/?repos=odysseus-dev%2Fodysseus&type=date&legend=top-left">
  <picture>
-   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=pewdiepie-archdaemon/odysseus&type=date&theme=dark&legend=top-left" />
-   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=pewdiepie-archdaemon/odysseus&type=date&legend=top-left" />
-   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=pewdiepie-archdaemon/odysseus&type=date&legend=top-left" />
+   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&theme=dark&legend=top-left" />
+   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
+   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
  </picture>
 </a>
 

ROADMAP.md

@@ -12,7 +12,6 @@ the codebase, you are probably right to stay away.
   and WSL all need coverage.
 
 - Integration audit: do integrations even work? Confirm what works, what needs setup docs, and what should be removed or hidden. 
-- Self-host troubleshooting cookbook. Document the weird 30-second fixes that otherwise become 30-minute searches: Dovecot cleartext auth for local stacks, ntfy Android Instant Delivery for non-ntfy.sh servers, clipboard limits on plain-HTTP Tailscale URLs, Radicale collection URLs, and similar traps.
 - Cookbook reliability on other computers. This is probably the area most likely to need work across different machines, GPUs, drivers, shells, and Python environments.
 - Cookbook SGLang support across platforms. Make sure SGLang setup/serve works
   predictably on Linux, Windows/WSL, macOS where possible, Docker, and common
@@ -33,6 +32,14 @@ the codebase, you are probably right to stay away.
   before the user request really starts. We need slimmer prompts, better tool
   selection, smaller default tool sets, and clearer guidance for models with
   4k/8k/16k context windows.
+- Local model speculative decoding support. For Odysseus-tuned local models,
+  plan to ship or recommend a small same-tokenizer draft model when the serving
+  backend supports it. Early vLLM testing showed a generic `Qwen3-0.6B` draft
+  beside `Qwen3-8B` can materially reduce wall time, while an unsupported
+  DSpark conversion performed poorly. Treat this as a supported draft-model lane
+  first; keep MTP-specific packaging as future work only when the architecture
+  and runtime support are real. Judge this by time-to-success, tool correctness,
+  grammar, and unchanged target output, not tokens/sec alone.
 - Skill/tool prompt-injection audit. User-editable skills, notes, documents,
   fetched pages, and memories should be treated as untrusted data. Keep testing
   whether models follow malicious instructions from those surfaces.

app.py

@@ -655,10 +655,15 @@ app.include_router(setup_emoji_routes())
 # Sessions
 from routes.session_routes import setup_session_routes
 session_config = {"REQUEST_TIMEOUT": REQUEST_TIMEOUT, "OPENAI_API_KEY": OPENAI_API_KEY, "SESSIONS_FILE": SESSIONS_FILE}
-app.include_router(setup_session_routes(session_manager, session_config, webhook_manager=webhook_manager))
+app.include_router(setup_session_routes(
+    session_manager,
+    session_config,
+    webhook_manager=webhook_manager,
+    upload_handler=upload_handler,
+))
 
 # Admin Danger Zone wipes (Settings → System → Danger Zone)
-from routes.admin_wipe_routes import setup_admin_wipe_routes
+from routes.admin_wipe.admin_wipe_routes import setup_admin_wipe_routes
 app.include_router(setup_admin_wipe_routes(session_manager))
 
 # Memory
@@ -684,7 +689,7 @@ app.include_router(setup_research_routes(research_handler, session_manager=sessi
 
 # History
 from routes.history.history_routes import setup_history_routes
-app.include_router(setup_history_routes(session_manager))
+app.include_router(setup_history_routes(session_manager, upload_handler=upload_handler))
 
 # Search
 from routes.search_routes import setup_search_routes
@@ -699,7 +704,7 @@ from routes.diagnostics_routes import setup_diagnostics_routes
 app.include_router(setup_diagnostics_routes(rag_manager, rag_available, research_handler, memory_vector))
 
 # Cleanup
-from routes.cleanup_routes import setup_cleanup_routes
+from routes.cleanup.cleanup_routes import setup_cleanup_routes
 app.include_router(setup_cleanup_routes(session_manager))
 
 # Personal docs
@@ -763,7 +768,7 @@ app.include_router(setup_assistant_routes(task_scheduler))
 
 # Calendar (CalDAV)
 from routes.calendar_routes import setup_calendar_routes
-calendar_router = setup_calendar_routes()
+calendar_router = setup_calendar_routes(upload_handler=upload_handler)
 app.include_router(calendar_router)
 
 # Shell (user-facing command execution)
@@ -782,7 +787,7 @@ from routes.hwfit_routes import setup_hwfit_routes
 app.include_router(setup_hwfit_routes())
 
 # Model A/B Comparison
-from routes.compare_routes import setup_compare_routes
+from routes.compare.compare_routes import setup_compare_routes
 app.include_router(setup_compare_routes(session_manager))
 
 # User Preferences
@@ -825,8 +830,8 @@ app.include_router(setup_api_token_routes())
 logger.info("Webhook & API token routes initialized")
 
 # Notes (Google Keep-style notes/todos)
-from routes.note_routes import setup_note_routes
-app.include_router(setup_note_routes(task_scheduler))
+from routes.note.note_routes import setup_note_routes
+app.include_router(setup_note_routes(task_scheduler, upload_handler=upload_handler))
 
 # Email
 from routes.email_routes import setup_email_routes
@@ -1043,7 +1048,7 @@ async def _startup_event():
         except BaseException as e:
             logger.warning(f"Built-in MCP registration failed (non-critical): {type(e).__name__}: {e}")
         try:
-            await asyncio.wait_for(mcp_manager.connect_all_enabled(), timeout=20)
+            await mcp_manager.connect_all_enabled()
         except asyncio.TimeoutError:
             logger.warning("User MCP startup timed out (non-critical)")
         except BaseException as e:

core/database.py

@@ -3,13 +3,16 @@ import logging
 import sqlite3
 from datetime import datetime, timezone
 from pathlib import Path
+from typing import Optional
+from urllib.parse import unquote, urlparse
 from sqlalchemy import event, create_engine, Column, String, Text, Boolean, DateTime, Integer, ForeignKey, JSON, Index, func, text
-from sqlalchemy.engine import Engine
+from sqlalchemy.engine import Engine, make_url
 from sqlalchemy.types import TypeDecorator
 from sqlalchemy.ext.declarative import declarative_base, declared_attr
 from sqlalchemy.orm import relationship, sessionmaker, backref
 
 from src.runtime_paths import get_app_root
+from core.platform_compat import safe_chmod, IS_WINDOWS
 
 logger = logging.getLogger(__name__)
 
@@ -42,12 +45,28 @@ def _default_database_url() -> str:
 
 
 def _normalize_sqlite_url(url: str) -> str:
-    if not url.startswith("sqlite:///"):
+    """Resolve relative ordinary SQLite paths without rewriting URI filenames."""
+    try:
+        parsed = make_url(url)
+    except Exception:
         return url
-    db_path = url.replace("sqlite:///", "", 1)
-    if db_path == ":memory:" or os.path.isabs(db_path):
+
+    if parsed.get_backend_name() != "sqlite":
         return url
-    return f"sqlite:///{(Path(get_app_root()) / db_path).resolve().as_posix()}"
+
+    db_path = parsed.database
+    if (
+        not db_path
+        or db_path == ":memory:"
+        or str(db_path).lower().startswith("file:")
+        or os.path.isabs(str(db_path))
+    ):
+        return url
+
+    absolute_path = (Path(get_app_root()) / str(db_path)).resolve().as_posix()
+    return parsed.set(database=absolute_path).render_as_string(
+        hide_password=False
+    )
 
 
 # Get database URL from environment, default to SQLite in DATA_DIR
@@ -59,6 +78,59 @@ engine = create_engine(
     connect_args={"check_same_thread": False} if "sqlite" in DATABASE_URL else {}
 )
 
+
+# Sidecar files SQLite can create next to the main DB. -journal is the default
+# rollback journal; -wal/-shm appear once WAL is enabled. Each can hold copies of
+# secret-bearing pages, so they get the same 0o600 lockdown as the DB itself.
+_SQLITE_SIDECARS = ("-journal", "-wal", "-shm")
+
+
+def _sqlite_db_path(url) -> Optional[str]:
+    """Return the filesystem path for a file-backed SQLite URL.
+
+    SQLite query parameters such as ``mode=memory`` only affect filename
+    semantics when SQLAlchemy enables URI handling with ``uri=true``. Ordinary
+    file URLs must therefore remain file-backed even when they contain a query
+    parameter named ``mode``.
+
+    For SQLite ``file:`` URIs, an empty authority or ``localhost`` identifies a
+    local path. Other authorities are retained as UNC-style paths.
+    """
+    if url.get_backend_name() != "sqlite":
+        return None
+
+    db_path = url.database
+    if not db_path or db_path == ":memory:":
+        return None
+
+    db_path = str(db_path)
+    query = {
+        str(key).lower(): str(value).strip().lower()
+        for key, value in dict(getattr(url, "query", {}) or {}).items()
+    }
+    uri_enabled = query.get("uri") in {"1", "true", "yes", "on"}
+    is_file_uri = db_path.lower().startswith("file:")
+
+    if not uri_enabled or not is_file_uri:
+        return db_path
+
+    if (
+        db_path.lower().startswith("file::memory:")
+        or query.get("mode") == "memory"
+    ):
+        return None
+
+    parsed = urlparse(db_path)
+    fs_path = parsed.path or ""
+    if not fs_path or fs_path == ":memory:":
+        return None
+
+    authority = parsed.netloc
+    if authority and authority.lower() != "localhost":
+        fs_path = f"//{authority}{fs_path}"
+
+    return unquote(fs_path)
+
 # Create session factory
 SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
 
@@ -1819,6 +1891,41 @@ def init_db():
     """
     _migrate_model_endpoints()
     Base.metadata.create_all(bind=engine)
+    # Lock the DB file (and any SQLite sidecars) to 0o600 — it holds bearer-token
+    # + bcrypt hashes and encrypted provider keys. POSIX only; safe_chmod no-ops
+    # on Windows (ACL-restricted profile dir) and the path helper returns None for
+    # Postgres / in-memory. Must stay AFTER create_all: the file is born here at
+    # the umask default, and nothing below resets the mode. The path comes from
+    # engine.url (SQLAlchemy's parsed URL), so a driver-qualified or query-tagged
+    # DATABASE_URL still resolves to the real file instead of slipping through.
+    db_path = _sqlite_db_path(engine.url)
+    if db_path is not None:
+        # Fail closed-loud on the main file: this is the only access control on
+        # it, so if the chmod genuinely fails (read-only FS, foreign owner) an
+        # operator should hear about it. safe_chmod also returns False as a
+        # Windows no-op, so guard on IS_WINDOWS to avoid a spurious warning there.
+        if not safe_chmod(db_path, 0o600) and not IS_WINDOWS:
+            logger.warning(
+                "Could not restrict %s to 0o600; it holds secrets and may be "
+                "world-readable. Check filesystem permissions and ownership.",
+                db_path,
+            )
+        # Re-lock any sidecars present at startup. New ones inherit the main
+        # file's mode (now 0o600, since we set it first), and they're usually
+        # absent here, but a stale -wal/-shm/-journal left by an older 0o644
+        # install could still expose secret pages. Absent sidecars are the
+        # normal case, not an error — only a failed chmod warrants a warning.
+        for suffix in _SQLITE_SIDECARS:
+            sidecar = db_path + suffix
+            if (
+                os.path.exists(sidecar)
+                and not safe_chmod(sidecar, 0o600)
+                and not IS_WINDOWS
+            ):
+                logger.warning(
+                    "Could not restrict %s to 0o600; it may expose DB pages.",
+                    sidecar,
+                )
     _migrate_add_hidden_models_column()
     _migrate_add_cached_models_column()
     _migrate_add_pinned_models_column()
@@ -1904,6 +2011,20 @@ def _migrate_chat_messages_fts():
     conn = None
     try:
         conn = sqlite3.connect(db_path)
+        fts_content_expr_new = (
+            "CASE WHEN instr(COALESCE(new.content, ''), ';base64,') > 0 "
+            "OR instr(COALESCE(new.content, ''), 'data:image/') > 0 "
+            "OR instr(COALESCE(new.content, ''), 'data:audio/') > 0 "
+            "THEN '[inline media omitted from search index]' "
+            "ELSE COALESCE(new.content, '') END"
+        )
+        fts_content_expr_cm = (
+            "CASE WHEN instr(COALESCE(cm.content, ''), ';base64,') > 0 "
+            "OR instr(COALESCE(cm.content, ''), 'data:image/') > 0 "
+            "OR instr(COALESCE(cm.content, ''), 'data:audio/') > 0 "
+            "THEN '[inline media omitted from search index]' "
+            "ELSE COALESCE(cm.content, '') END"
+        )
         try:
             conn.execute("CREATE VIRTUAL TABLE IF NOT EXISTS temp._odysseus_fts5_probe USING fts5(content)")
             conn.execute("DROP TABLE IF EXISTS temp._odysseus_fts5_probe")
@@ -1912,7 +2033,7 @@ def _migrate_chat_messages_fts():
             return
 
         conn.executescript(
-            """
+            f"""
             CREATE VIRTUAL TABLE IF NOT EXISTS chat_messages_fts USING fts5(
                 content,
                 message_id UNINDEXED,
@@ -1920,10 +2041,14 @@ def _migrate_chat_messages_fts():
                 role UNINDEXED
             );
 
+            DROP TRIGGER IF EXISTS chat_messages_fts_ai;
+            DROP TRIGGER IF EXISTS chat_messages_fts_ad;
+            DROP TRIGGER IF EXISTS chat_messages_fts_au;
+
             CREATE TRIGGER IF NOT EXISTS chat_messages_fts_ai
             AFTER INSERT ON chat_messages BEGIN
                 INSERT INTO chat_messages_fts(content, message_id, session_id, role)
-                VALUES (COALESCE(new.content, ''), new.id, new.session_id, new.role);
+                VALUES ({fts_content_expr_new}, new.id, new.session_id, new.role);
             END;
 
             CREATE TRIGGER IF NOT EXISTS chat_messages_fts_ad
@@ -1935,14 +2060,14 @@ def _migrate_chat_messages_fts():
             AFTER UPDATE ON chat_messages BEGIN
                 DELETE FROM chat_messages_fts WHERE message_id = old.id;
                 INSERT INTO chat_messages_fts(content, message_id, session_id, role)
-                VALUES (COALESCE(new.content, ''), new.id, new.session_id, new.role);
+                VALUES ({fts_content_expr_new}, new.id, new.session_id, new.role);
             END;
             """
         )
         conn.execute(
-            """
+            f"""
             INSERT INTO chat_messages_fts(content, message_id, session_id, role)
-            SELECT COALESCE(cm.content, ''), cm.id, cm.session_id, cm.role
+            SELECT {fts_content_expr_cm}, cm.id, cm.session_id, cm.role
             FROM chat_messages cm
             WHERE NOT EXISTS (
                 SELECT 1 FROM chat_messages_fts fts
@@ -1950,6 +2075,7 @@ def _migrate_chat_messages_fts():
             )
             """
         )
+        _scrub_legacy_chat_message_fts_media(conn)
         conn.commit()
     except Exception as e:
         logging.getLogger(__name__).warning(f"chat_messages FTS migration failed: {e}")
@@ -1960,6 +2086,37 @@ def _migrate_chat_messages_fts():
             pass
 
 
+def _scrub_legacy_chat_message_fts_media(conn) -> None:
+    """Replace already-indexed inline media rows with searchable text only."""
+    try:
+        from src.attachment_refs import search_index_text
+    except Exception as e:
+        logging.getLogger(__name__).warning(f"chat_messages FTS media scrub skipped: {e}")
+        return
+
+    try:
+        rows = conn.execute(
+            """
+            SELECT id, session_id, role, content
+            FROM chat_messages
+            WHERE instr(COALESCE(content, ''), ';base64,') > 0
+               OR instr(COALESCE(content, ''), 'data:image/') > 0
+               OR instr(COALESCE(content, ''), 'data:audio/') > 0
+            """
+        ).fetchall()
+        for message_id, session_id, role, content in rows:
+            conn.execute("DELETE FROM chat_messages_fts WHERE message_id = ?", (message_id,))
+            conn.execute(
+                """
+                INSERT INTO chat_messages_fts(content, message_id, session_id, role)
+                VALUES (?, ?, ?, ?)
+                """,
+                (search_index_text(content), message_id, session_id, role),
+            )
+    except Exception as e:
+        logging.getLogger(__name__).warning(f"chat_messages FTS media scrub failed: {e}")
+
+
 def _migrate_add_email_smtp_security():
     """Add explicit SMTP security mode for Proton Bridge/custom local SMTP."""
     import sqlite3

core/session_manager.py

@@ -16,6 +16,8 @@ from typing import Dict, Optional
 
 from .database import Session as DbSession, ChatMessage as DbChatMessage, Document as DbDocument, SessionLocal, utcnow_naive
 from .models import Session, ChatMessage
+from src.attachment_refs import persistable_message_content
+from src.upload_handler import reserve_message_upload_references
 
 # Re-export singleton accessors from models for convenience
 from .models import set_session_manager_instance, get_session_manager_instance
@@ -72,6 +74,7 @@ class SessionManager:
     def __init__(self, sessions_file: str = None):
         # sessions_file kept for backward compat, not used
         self.sessions: Dict[str, Session] = {}
+        self.upload_handler = None
         self.load_sessions()
 
     # ------------------------------------------------------------------
@@ -230,17 +233,26 @@ class SessionManager:
                 logger.warning("Dropping message for deleted session %s", session_id)
                 return
 
+            missing_upload_id = reserve_message_upload_references(
+                getattr(self, "upload_handler", None),
+                getattr(db_session, "owner", None),
+                message.content,
+                message.metadata,
+            )
+            if missing_upload_id:
+                raise ValueError(
+                    f"Referenced upload is no longer available: {missing_upload_id}"
+                )
+
             msg_id = str(uuid.uuid4())
             msg_time = datetime.utcnow()
             if message.metadata is None:
                 message.metadata = {}
             message.metadata.setdefault('timestamp', _message_timestamp_iso(msg_time))
-            # Multimodal content (image/audio attachments) is a list — serialize
-            # to JSON so the Text column can store it.  On reload, _db_to_session
-            # detects the JSON-array prefix and parses it back.
-            _content = message.content
-            if isinstance(_content, list):
-                _content = json.dumps(_content)
+            # Multimodal content may contain provider data URLs for the live
+            # model call. Persist only readable text plus attachment references
+            # so chat_messages/FTS do not duplicate upload bytes.
+            _content = persistable_message_content(message.content, message.metadata)
             db_message = DbChatMessage(
                 id=msg_id,
                 session_id=session_id,
@@ -322,6 +334,28 @@ class SessionManager:
         session = self.get_session(session_id)
         db = SessionLocal()
         try:
+            db_session = db.query(DbSession).filter(DbSession.id == session_id).first()
+            if db_session is None:
+                logger.warning("Cannot replace history for missing session %s", session_id)
+                return False
+
+            # Reserve every incoming attachment before removing any durable
+            # message row. reserve_upload() shares the upload lifecycle lock
+            # with cleanup, so an upload cannot be deleted between this
+            # ownership check/access touch and the replacement transaction.
+            # A failed reservation must leave the existing transcript intact.
+            for message in messages:
+                missing_upload_id = reserve_message_upload_references(
+                    getattr(self, "upload_handler", None),
+                    getattr(db_session, "owner", None),
+                    message.content,
+                    message.metadata,
+                )
+                if missing_upload_id:
+                    raise ValueError(
+                        f"Referenced upload is no longer available: {missing_upload_id}"
+                    )
+
             db.query(DbChatMessage).filter(DbChatMessage.session_id == session_id).delete()
             now = datetime.now(timezone.utc)
             for i, message in enumerate(messages):
@@ -330,15 +364,9 @@ class SessionManager:
                     id=msg_id,
                     session_id=session_id,
                     role=message.role,
-                    # Multimodal content (image/audio attachments) is a list;
-                    # serialize to JSON so the Text column round-trips via
-                    # _parse_msg_content. Storing the raw list let SQLAlchemy
-                    # bind its single-quoted repr, which _parse_msg_content
-                    # cannot parse (it looks for double-quoted "type"), so the
-                    # attachment was destroyed on reload. Mirrors _persist_message.
-                    content=(json.dumps(message.content)
-                             if isinstance(message.content, list)
-                             else message.content),
+                    # Mirrors _persist_message: keep raw media bytes out of the
+                    # persisted transcript and search index.
+                    content=persistable_message_content(message.content, message.metadata),
                     meta_data=json.dumps(message.metadata) if message.metadata else None,
                     timestamp=now + timedelta(microseconds=i),
                 )
@@ -347,12 +375,10 @@ class SessionManager:
                     message.metadata = {}
                 message.metadata["_db_id"] = msg_id
 
-            db_session = db.query(DbSession).filter(DbSession.id == session_id).first()
-            if db_session:
-                db_session.message_count = len(messages)
-                db_session.updated_at = now
-                db_session.last_accessed = now
-                db_session.last_message_at = now
+            db_session.message_count = len(messages)
+            db_session.updated_at = now
+            db_session.last_accessed = now
+            db_session.last_message_at = now
 
             db.commit()
             session.history = list(messages)
@@ -517,6 +543,12 @@ class SessionManager:
         """Permanently delete a session and all its messages."""
         db = SessionLocal()
         try:
+            try:
+                from src.session_image_cleanup import cleanup_session_images
+                cleanup_session_images(session_id, db=db)
+            except Exception as e:
+                logger.warning(f"Image cleanup failed while deleting session {session_id}: {e}")
+
             # Detach documents so they survive as orphans in the library
             db.query(DbDocument).filter(DbDocument.session_id == session_id).update(
                 {DbDocument.session_id: None}, synchronize_session=False

docker-compose.gpu-amd.yml

@@ -70,6 +70,9 @@ services:
       - DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
       - GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
       - GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}
+      - GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
+      - GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
+      - GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
       - TAVILY_API_KEY=${TAVILY_API_KEY:-}
       - SERPER_API_KEY=${SERPER_API_KEY:-}
       # PUID / PGID — the user/group the container drops to before

docker-compose.gpu-nvidia.yml

@@ -69,6 +69,9 @@ services:
       - DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
       - GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
       - GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}
+      - GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
+      - GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
+      - GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
       - TAVILY_API_KEY=${TAVILY_API_KEY:-}
       - SERPER_API_KEY=${SERPER_API_KEY:-}
       # PUID / PGID — the user/group the container drops to before

docker-compose.yml

@@ -58,6 +58,9 @@ services:
       - DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
       - GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
       - GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}
+      - GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
+      - GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
+      - GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
       - TAVILY_API_KEY=${TAVILY_API_KEY:-}
       - SERPER_API_KEY=${SERPER_API_KEY:-}
       # PUID / PGID — the user/group the container drops to before

docs/attachments.md

@@ -0,0 +1,85 @@
+# Attachment References and Upload Storage
+
+Odysseus stores uploaded bytes once under the configured upload directory and
+passes stable references through chat history, tools, and future artifact work.
+The goal is to avoid duplicating large inline media payloads in
+`chat_messages.content` or the SQLite FTS index.
+
+## Reference Shape
+
+Attachment references use this minimum shape:
+
+```json
+{
+  "type": "attachment_ref",
+  "attachment_id": "32hex-or-32hex.ext",
+  "name": "original-filename.png",
+  "mime": "image/png",
+  "size": 12345,
+  "checksum_sha256": "hex-digest",
+  "created_at": "2026-07-09T12:00:00"
+}
+```
+
+Optional fields such as `width`, `height`, `vision`, `vision_model`, and
+`gallery_id` may be present when the uploader or preprocessing path knows them.
+
+## Persistence
+
+The live model call may still receive provider-specific multimodal blocks for
+the current turn. Persistence is different:
+
+- `chat_messages.content` stores readable text plus compact attachment reference
+  lines, never raw `data:*;base64,...` upload bytes.
+- `chat_messages.metadata.attachments` stores structured attachment reference
+  metadata for UI reloads and future processing.
+- The SQLite FTS migration recreates chat-message FTS triggers so new rows do
+  not index inline media payloads, and it scrubs legacy rows that were already
+  indexed with data URLs.
+
+## Tool Access
+
+Agent/tool context receives upload entries as `attachment_ref` manifests with an
+`odysseus://attachment/<id>` URI and `read_policy: "owner_checked_upload"`.
+
+For compatibility with existing built-in tools, a local `path` may be included
+only after all of these checks pass:
+
+- the upload ID resolves through `UploadHandler.resolve_upload`;
+- the requested owner is allowed to read the upload;
+- the file remains inside the configured upload directory;
+- the file path is inside the tool-readable roots.
+
+External MCP/custom tools should treat the URI and attachment ID as the stable
+contract and request bytes through an owner-checked server path, not by assuming
+host filesystem layout.
+
+## Retention and Deletion
+
+Current retention behavior is conservative:
+
+- uploads are indexed in `uploads.json` with owner, checksum, MIME type, size,
+  and creation time;
+- admin cleanup first scans persisted chat metadata/content, document versions,
+  PDF source markers, gallery hashes, notes, and calendar records for live
+  references;
+- cleanup fails closed if that reference scan cannot complete, and the lower-level
+  cleanup API removes nothing unless it receives a complete reference snapshot;
+- expired, unreferenced uploads are removed during the completed scan, while
+  attachment-bearing writers must first take an owner-checked reservation that
+  serializes with deletion and refreshes the upload's access timestamp;
+- deliberate removal atomically drops matching `uploads.json` rows before deleting
+  the bytes and restores those rows if filesystem removal fails;
+- deleting a chat removes the chat rows but does not immediately delete shared
+  upload bytes, because the same upload may also be referenced by gallery items,
+  documents, duplicate-upload rows, or future artifact records.
+
+There is no distinct artifact table in the current schema. Artifact-like upload
+references persisted in chat or document text are covered by the canonical
+attachment-ID scan; any future artifact store must be added to reference discovery
+before cleanup is allowed to consider its uploads unreferenced.
+
+Cleanup and write reservations share the upload-index lock. This closes the
+scan/write/delete race in the documented single-worker deployment; a future
+multi-process deployment must add an inter-process lock or move lifecycle state
+into the database before enabling destructive cleanup in more than one worker.

docs/discovery/BASELINE-STATUS.md

@@ -0,0 +1,62 @@
+# Discovery Baseline Status
+
+## Purpose and scope
+
+This package is a commit-pinned discovery baseline and feature index for the `discovery` branch at frozen commit `d8a2059df8e53bc7275c45339849d14c8651e73c`. It inventories **79 feature records** across **16 domains** to help maintainers locate likely implementation areas and identify validation gaps. It is not an authoritative architecture reference or a runtime-certification record.
+
+The canonical inventory is [`feature-catalog.json`](feature-catalog.json); [`feature-catalog.md`](feature-catalog.md) and the files in [`domains/`](domains/) are derived reading views. See [`audit-method.md`](audit-method.md) for the status and maturity definitions.
+
+## What maintainers may use now
+
+- Use the catalog and domain views as a frozen discovery index, including their feature IDs, stated scope, likely source locations, and declared runtime prerequisites.
+- Treat a catalog status such as `verified` as meaning implementation was identified during discovery. It does **not** mean every evidence locator, line range, test claim, or runtime behaviour has passed semantic validation.
+- Use the structural checks to confirm package shape and cross-view consistency; use the evidence validator to assess whether individual evidence assertions are semantically supported.
+
+## Validation snapshot
+
+Structural validation passes: the catalog has 79 records, the 16 domain views match it, and the package structural validators pass. All 11 focused evidence-validator tests pass.
+
+The evidence validator found **170 evidence entries**: **82 valid**, **66 invalid**, **6 ambiguous**, and **16 unsupported**. Its non-zero exit is expected while those semantic evidence defects remain.
+
+Structural validity checks the documentation schema, record counts, derived-view consistency, file existence, line-range bounds, links, and public-safety rules. Semantic evidence validity additionally checks whether the cited locator exists, falls within its cited range, uses a supported parser, and actually supports the feature claim. Passing the former does not establish the latter.
+
+## E2 review decisions
+
+E2 means directly relevant automated test evidence supports the feature claim; a test file’s existence alone is insufficient. The generated evidence report was used to reassess all ten E2 records.
+
+| Feature | Decision | Reason |
+|---|---|---|
+| `CHAT-001` | Retain E2 | Two cited tests directly exercise documented streaming-related behaviour. |
+| `MODEL-006` | Retain E2 | The cited device-flow test exercises the Copilot start/poll contract and verification URI. |
+| `MODEL-007` | Retain E2 | The cited device-flow test exercises the ChatGPT subscription contract and verification URI. |
+| `RESEARCH-003` | Demote to E1 | The cited test covers ranking, not provider connectivity or dispatch; the route-to-provider implementation path was identified. |
+| `DOCUMENT-002` | Demote to E1 | The cited marker test does not support form processing or rendering; the document route does call the PDF form handlers. |
+| `EMAIL-001` | Demote to E1 | The cited health test is narrower than setup, SMTP, and polling; application setup invokes the email router and its poller. |
+| `SECURITY-002` | Demote to E1 | Relevant vault-password tests exist, but the cited test locator is fabricated; the application mounts the vault route implementation. |
+| `SECURITY-004` | Demote to E1 | Relevant injection tests exist, but the cited locator names are fabricated; callers use the documented context guard. |
+| `SECURITY-005` | Demote to E1 | Relevant guard tests exist, but the cited locator names are fabricated; route code calls the documented URL guard. |
+| `PLATFORM-009` | Demote to E0 | The manifests and diagnostic script establish discovered operational artifacts, not a traced application path or directly relevant automated test. |
+
+The current maturity distribution is **E0: 68**, **E1: 8**, **E2: 3**, **E3: 0**, **E4: 0**. Runtime validation is still pending where the catalog says it requires external services, interactive authentication, specialised hardware, or host Docker GPU support.
+
+## Known discovery caveats
+
+- `AGENT-004` includes a legacy no-op activity-log shim rather than active assistant-log behaviour.
+- `FRONTEND-007` points to a missing `static/backgrounds.html` target; its existing variant pages do not make that route functional.
+- `RESEARCH-003` retains a compatibility module that aliases the canonical search implementation, and `DOCUMENT-002` separates optional PDF viewing from form handling.
+- Secret-storage and vault-command handling are distinct implementation areas; this index does not make an end-to-end security guarantee.
+
+## Recommended next documentation work
+
+Repair semantic evidence selectively while architecture and operations documentation is written, beginning with the seven E2 demotions and maintainer-owned feature descriptions. Do not wait for the complete evidence queue before documenting the system. Record controlled runtime observations when external services, credentials, hardware, or Docker GPU access are available, and label unsupported claims explicitly.
+
+## Validation commands
+
+```bash
+PYTHONDONTWRITEBYTECODE=1 python3 docs/discovery/tools/validate_discovery_docs.py --repo-root .
+PYTHONDONTWRITEBYTECODE=1 python3 docs/discovery/tools/validate_discovery_consistency.py
+PYTHONDONTWRITEBYTECODE=1 python3 -m unittest docs/discovery/tools/test_validate_discovery_evidence.py -v
+PYTHONDONTWRITEBYTECODE=1 python3 docs/discovery/tools/validate_discovery_evidence.py --repo-root . --catalog docs/discovery/feature-catalog.json --output-dir <local-report-directory>
+```
+
+Supply a local report directory outside `docs/discovery/` for the final command so generated reports are not added to the package.

docs/discovery/README.md

@@ -0,0 +1,21 @@
+# Odysseus Discovery Package
+
+## Provisional discovery baseline
+
+This is a commit-pinned discovery baseline and feature index for the `discovery` branch at `d8a2059df8e53bc7275c45339849d14c8651e73c`. It contains 79 feature records across 16 domains. It is **not** an authoritative architecture reference, a runtime certification, or a claim that every evidence citation is semantically valid.
+
+Read [`BASELINE-STATUS.md`](BASELINE-STATUS.md) first for the publication status, evidence-validation totals, E2 decisions, known caveats, and the recommended next documentation work.
+
+## Package contents
+
+- [`feature-catalog.json`](feature-catalog.json) is the canonical machine-readable catalog.
+- [`feature-catalog.md`](feature-catalog.md) and [`domains/`](domains/) are derived reading views.
+- [`audit-method.md`](audit-method.md) defines feature status and evidence maturity.
+- [`references/source-provenance.md`](references/source-provenance.md) records the frozen snapshot.
+- [`tools/`](tools/) contains the structural, consistency, and evidence validators.
+
+A feature status such as `verified` means implementation was identified during discovery. It does not mean every evidence locator, line range, test claim, or runtime behaviour has passed semantic validation.
+
+## Validation
+
+Run the commands in [`BASELINE-STATUS.md`](BASELINE-STATUS.md#validation-commands). Structural checks and semantic evidence checks have different purposes; see that status document for the current results and interpretation.

docs/discovery/audit-method.md

@@ -0,0 +1,80 @@
+# Odysseus Functional Audit Methodology
+
+## Purpose
+
+This document specifies the methodology and evidence standards for the read-only discovery audit of **Odysseus**.
+
+## Snapshot Baseline
+
+- **Repository**: `odysseus-dev/odysseus`
+- **Audit Target Branch**: `discovery`
+- **Frozen Commit SHA**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Snapshot Date**: `2026-07-23T14:49:02Z`
+- **Audit Mode**: Read-Only inventory & documentation review
+
+## Rules of Engagement
+
+1. **No Code Mutations**: Application code and tests outside `docs/discovery/` remain untouched.
+2. **No External Operations**: No GitHub issues, PRs, comments, labels, or branch mutations.
+3. **Zero Inferred Success**: Documentation claims require empirical evidence of implementation and reachability. Filenames, README descriptions, and docstrings alone do not constitute proof.
+4. **Strict Status Categorization**: All capabilities are assigned exactly one authorized status:
+   - `verified`: Implemented, reachable, and supported by code evidence.
+   - `partial`: Partially implemented or missing full frontend/backend connection.
+   - `disabled`: Gated off by default feature flags or configuration.
+   - `experimental`: Active but requiring non-standard hardware or runtimes.
+   - `legacy`: Obsolete feature retained for backwards compatibility.
+   - `dead-code-candidate`: Code exists but is unreachable from UI or API routes.
+   - `unverified`: Implementation present but untestable without external secrets or hardware.
+
+## Evidence Maturity Scale
+
+Evidence maturity is evaluated independently from catalog feature status. A feature status such as `verified` records that implementation was identified during discovery; it is not a statement that every evidence locator, test claim, line range, or runtime behaviour has passed semantic validation.
+
+- **E0 - Discovered**: Candidate identified in documentation, route declaration, or source file.
+- **E1 - Code-path traced**: Frontend/API entry point connected through services and data handlers.
+- **E2 - Test-backed**: At least one directly relevant automated test assertion supports the feature claim. A test file's existence, an unrelated assertion, or an invalid test locator does not establish E2.
+- **E3 - Runtime-validated**: Maintainer reproduced behavior in a recorded local environment.
+- **E4 - Maintainer-accepted**: Maintainers accepted the feature description and support status.
+
+## Audit Workflow
+
+```mermaid
+flowchart TD
+    P0["Phase 0: Snapshot Isolation<br/>(Commit d8a2059)"] --> P1["Phase 1: Codebase Discovery<br/>(Routes, Services, Static JS, Specs)"]
+    P1 --> P2["Phase 2: Feature Reachability & Verification<br/>(Route matching, FE entrypoints, tests)"]
+    P2 --> P3["Phase 3: Catalog & Evidence Compilation<br/>(File paths, symbols, exact line ranges)"]
+    P3 --> P4["Phase 4: Quality & Integrity Audit<br/>(100% path existence check, schema validation)"]
+```
+
+### Phase 0: Snapshot Isolation
+The audit is pinned to git commit `d8a2059df8e53bc7275c45339849d14c8651e73c`. All file paths, symbol declarations, and line ranges map strictly to this commit.
+
+### Phase 1: Codebase Discovery
+All top-level and nested directories were traversed, including:
+- Backend Entry Points (`app.py`, `routes/`, `routes/*/*.py`, `companion/`)
+- Core Framework (`core/database.py`, `core/session_manager.py`, `core/auth.py`)
+- Business Logic Services (`src/`, `services/`, `mcp_servers/`)
+- Frontend Assets (`static/app.js`, `static/js/`, `static/index.html`)
+- Test Suites (`tests/`, `tests/cli/`, `tests/streaming/`)
+- Operations & Docker (`Dockerfile`, `docker-compose*.yml`, `scripts/`)
+
+### Phase 2: Verification Protocol
+For each feature candidate, the following table was evaluated:
+- **User Reachability**: Frontend UI element, modal, route, or CLI script.
+- **API Entrypoint**: FastAPI `@router` declaration or WebSocket/SSE handler.
+- **Backend Execution**: Concrete Python module method, service, or tool call.
+- **Data Persistence**: Disk file, SQLite table, or vector collection.
+- **Test Coverage**: Automated test file executing assertions against the component.
+
+### Phase 3: Evidence Linking Standard
+Every feature entry in `feature-catalog.json` contains a structured `evidence` list with:
+- `path`: Relative path from repository root.
+- `symbol`: Route, class, function, or element symbol name.
+- `line_range`: Inclusive line range (e.g. `L120-L250`).
+- `explanation`: Short factual statement proving reachability or implementation.
+
+### Phase 4: Quality Check & Schema Constraints
+Before finalization:
+1. Every evidence file path is validated against the checkout.
+2. Every Markdown entry matches `feature-catalog.json`.
+3. Recommendation language is separated from empirical factual observations.

docs/discovery/domains/agent.md

@@ -0,0 +1,153 @@
+# Agent
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## AGENT-001 — Autonomous Agent Loop & Tool Execution Engine
+
+- **Domain**: `agent`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Executes multi-step agent reasoning loops, tool invocation parsing, and automated response generation.
+
+### Evidence summary
+
+- `src/agent_loop.py` — `run_agent_loop` — Core loop evaluating model tool requests and executing handlers.
+- `src/tool_execution.py` — `execute_tool_call` — Dispatches tool invocation requests to underlying tool handlers.
+
+### Unknowns
+
+- Infinite tool loop if termination condition fails.
+
+## AGENT-002 — Scheduled Tasks & Event Bus Dispatcher
+
+- **Domain**: `agent`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Schedules background recurring or delayed tasks, emits event bus triggers, and executes automated flows.
+
+### Evidence summary
+
+- `routes/task_routes.py` — `@router.get('')` — Fetches active scheduled tasks.
+- `src/task_scheduler.py` — `TaskScheduler` — Async task scheduler dispatching cron and delay triggers.
+
+### Unknowns
+
+- Task execution failure handling on system restart.
+
+## AGENT-003 — Webhook Event Subscriptions & Trigger Processing
+
+- **Domain**: `agent`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Manages incoming/outgoing webhook subscriptions, endpoint authentication tokens, and event triggers.
+
+### Evidence summary
+
+- `routes/webhook_routes.py` — `@router.get('/webhooks')` — Returns list of registered webhooks.
+- `src/webhook_manager.py` — `WebhookManager` — Handles payload delivery and signature verification.
+
+### Unknowns
+
+- SSRF risks when contacting external webhook URLs if unvalidated.
+
+## AGENT-004 — Assistant Settings, Task Check-Ins & Background Job Monitor
+
+- **Domain**: `agent`
+- **Status**: `partial`
+- **Evidence Maturity**: `E1`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Manages per-user assistant sessions and scheduled check-in settings, drains background job completions, and retains a legacy no-op activity logging shim.
+
+### Evidence summary
+
+- `routes/assistant_routes.py` — `setup_assistant_routes` — Active assistant session, settings, manual check-in, run-status and timezone-list endpoints, including the owner-scoping guards.
+- `src/bg_monitor.py` — `_drain_agent` — Runs the agent loop headless against a session to produce the background-job follow-up turn.
+- `src/bg_monitor.py` — `_run_followup` — Drains completed background jobs and auto-continues the owning session, deferring while a live turn is in progress.
+- `src/assistant_log.py` — `log_to_assistant` — Legacy no-op activity logging shim retained for existing callers; documented as inactive rather than as current behaviour.
+
+### Unknowns
+
+- Route `/api/assistant/logs` cited in legacy docs is absent from assistant router.
+- Existing unit test `tests/cli/test_logs_cli_resolve_nonstring.py` tests CLI target-name resolution logic, not active assistant routes or bg_monitor execution loop.
+
+## AGENT-005 — Model Context Protocol (MCP) Server Integration
+
+- **Domain**: `agent`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Integrates external MCP servers over stdio/SSE to expand agent capabilities dynamically.
+
+### Evidence summary
+
+- `routes/mcp_routes.py` — `setup_mcp_routes` — Exposes management endpoints for external MCP servers.
+- `src/mcp_manager.py` — `McpManager` — Manages MCP server subprocess lifecycles.
+
+### Unknowns
+
+- Subprocess leaks if external MCP server process fails to terminate clean.
+
+## AGENT-006 — AI Interaction Tools & Pipeline Orchestration
+
+- **Domain**: `agent`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides specialized AI interaction tools for agent self-debugging, debate, and multi-model collaboration.
+
+### Evidence summary
+
+- `src/ai_interaction.py` — `init_ai_interaction_tools` — Registers specialized multi-agent interaction primitives.
+- `src/builtin_actions.py` — `execute_builtin_action` — Executes pre-built action intent sequences.
+
+### Unknowns
+
+- High API token consumption during extended agent debates.
+
+## AGENT-007 — Subprocess & Background Job Execution Tools
+
+- **Domain**: `agent`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides sandboxed bash/shell tool execution capabilities with output streaming and background tracking.
+
+### Evidence summary
+
+- `src/agent_tools/subprocess_tools.py` — `run_command` — Executes shell commands in background/foreground.
+- `src/bg_jobs.py` — `JobManager` — Tracks async background subprocess tasks.
+
+### Unknowns
+
+- Arbitrary shell command execution permissions if sandbox confinement fails.

docs/discovery/domains/calendar.md

@@ -0,0 +1,45 @@
+# Calendar
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## CALENDAR-001 — CalDAV Calendar Synchronization & Account Setup
+
+- **Domain**: `calendar`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires a controlled external CalDAV server.
+
+### Purpose
+
+Connects to remote CalDAV servers (Apple iCloud, Nextcloud, Google) to sync calendar event feeds.
+
+### Evidence summary
+
+- `routes/calendar_routes.py` — `setup_calendar_routes` — Exposes CalDAV setup and manual sync trigger routes.
+- `src/caldav_sync.py` — `CalDavSync` — Fetches and parses remote iCalendar VEVENT objects.
+
+### Unknowns
+
+- Invalid SSL certificates on self-hosted CalDAV servers.
+
+## CALENDAR-002 — Calendar Event Operations & iCalendar Parsing
+
+- **Domain**: `calendar`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Creates, updates, deletes, and displays calendar events with timezone conversion and reminder notifications.
+
+### Evidence summary
+
+- `routes/calendar_routes.py` — `@router.get('/events')` — Fetches calendar events for requested date window.
+- `src/tools/calendar.py` — `CalendarTool` — Agent tool for creating and modifying calendar entries.
+
+### Unknowns
+
+- Recurring RRULE event expansion calculation bugs across leap years.

docs/discovery/domains/chat.md

@@ -0,0 +1,197 @@
+# Chat
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## CHAT-001 — Core Chat Streaming & SSE Message Generation
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E2`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires access to a live LLM provider endpoint (OpenAI API key or local Ollama server).
+
+### Purpose
+
+Handles real-time Server-Sent Events (SSE) chat streaming, token rendering, and model response generation.
+
+### Evidence summary
+
+- `routes/chat_routes.py` — `chat_stream` — POST /api/chat_stream SSE endpoint; builds the shared chat context, then dispatches to the chat-mode or agent-mode streaming path.
+- `routes/chat_helpers.py` — `build_chat_context` — Shared context builder invoked by chat_stream; runs message preprocessing and assembles the memory/RAG/web context preface.
+- `src/chat_handler.py` — `ChatHandler.preprocess_message` — Message preprocessing (attachments, URLs, tool preprocessing) reached from build_chat_context via routes/chat_helpers.py:preprocess.
+- `src/chat_processor.py` — `ChatProcessor.build_context_preface` — Builds the retrieval and web-source context preface injected into the streamed request.
+- `src/llm_core.py` — `stream_llm_with_fallback` — Chat-mode streaming dispatcher called from chat_stream; wraps stream_llm with an ordered provider fallback chain.
+- `src/llm_core.py` — `stream_llm` — Per-request streaming entry wrapped by stream_llm_with_fallback; acquires the local model slot and delegates to _stream_llm_inner.
+- `src/agent_loop.py` — `stream_agent_loop` — Agent-mode streaming path called from chat_stream when the request selects agent mode.
+- `tests/test_chat_metrics.py` — `test_stream_llm_passes_through_llamacpp_timings` — Inspected unit test asserting stream_llm forwards backend generation timings into the emitted metrics chunk.
+- `tests/test_resend_message_nondestructive.py` — `test_resend_message_does_not_truncate_by_default` — Inspected unit test asserting the frontend resend path does not truncate prior conversation turns.
+
+### Unknowns
+
+- Stream interruption on connection drops requires retry logic.
+
+## CHAT-002 — Session Management & Conversation State
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Manages session creation, listing, switching, renaming, and persistence of conversation metadata.
+
+### Evidence summary
+
+- `routes/session_routes.py` — `@router.get('/api/sessions')` — Lists active sessions filtered by user owner scope.
+- `core/session_manager.py` — `SessionManager` — Provides thread-safe session storage operations.
+
+### Unknowns
+
+- Concurrent file writes to sessions.json under high load.
+
+## CHAT-003 — Chat History & Message Editing/Truncation
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides history retrieval, message content updating, message deletion, and history branch truncation.
+
+### Evidence summary
+
+- `routes/history/history_routes.py` — `@router.get('/api/history/{session_id}')` — Fetches message history timeline for a session.
+- `routes/history_routes.py` — `_sys.modules[__name__] = _canonical` — Backward-compatibility shim module.
+
+### Unknowns
+
+- Truncating messages re-indexes context window and clears cached tool calls.
+
+## CHAT-004 — File & Multimodal Attachment Handling
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Handles uploading, mime validation, image preview, vision encoding, and file attachments in chat messages.
+
+### Evidence summary
+
+- `routes/upload_routes.py` — `@router.post('')` — Accepts multi-part file uploads and generates vision metadata.
+- `src/upload_handler.py` — `UploadHandler.save_file` — Validates upload size and atomicity on disk.
+
+### Unknowns
+
+- Large file uploads may consume server disk space if cleanup task fails.
+
+## CHAT-005 — Chat Message Search
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Enables full-text keyword search across stored chat messages and sessions.
+
+### Evidence summary
+
+- `routes/search_routes.py` — `setup_search_routes` — Registers chat message search endpoint.
+- `src/session_search.py` — `search_sessions` — Executes query matching against session transcripts.
+
+### Unknowns
+
+- Full table scans on un-indexed text columns for very large databases.
+
+## CHAT-006 — System Prompts & Preset Management
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides creation, selection, and customization of system prompt presets for chat sessions.
+
+### Evidence summary
+
+- `routes/preset_routes.py` — `setup_preset_routes` — API routes for listing and modifying system prompt presets.
+- `src/preset_manager.py` — `PresetManager` — Disk-backed manager for prompt presets.
+
+### Unknowns
+
+- Invalid JSON syntax in user presets file can corrupt preset loading.
+
+## CHAT-007 — Emoji Rendering & Twemoji SVG Proxy
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Proxies Twemoji SVG icons locally to render flat SVG emojis in message text without external CDN dependencies.
+
+### Evidence summary
+
+- `routes/emoji_routes.py` — `setup_emoji_routes` — Serves locally cached Twemoji SVGs.
+
+### Unknowns
+
+- First request fetches SVG from remote CDN before caching locally.
+
+## CHAT-008 — Input History Recall (Arrow Up)
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Allows users to cycle through previously sent prompt messages in the chat composer input using Arrow-Up/Down keys.
+
+### Evidence summary
+
+- `static/js/composerArrowUpRecall.js` — `initComposerRecall` — Listens for ArrowUp keypress on composer textarea.
+
+### Unknowns
+
+- Client-side browser storage limits.
+
+## CHAT-009 — Context Window Compaction & Truncation
+
+- **Domain**: `chat`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Compacts session transcript history when prompt size exceeds context limits using summarization.
+
+### Evidence summary
+
+- `routes/history/history_routes.py` — `@router.post('/api/session/{session_id}/compact')` — Triggers context summarization and compaction.
+- `src/context_compactor.py` — `compact_context` — Executes context token pruning and summary generation.
+
+### Unknowns
+
+- Aggressive compaction may discard subtle user instructions.

docs/discovery/domains/contact.md

@@ -0,0 +1,24 @@
+# Contact
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## CONTACT-001 — CardDAV Contact Management & Address Book Integration
+
+- **Domain**: `contact`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Connects to CardDAV servers, imports VCard contacts, and provides contact lookup for email/calendar autocomplete.
+
+### Evidence summary
+
+- `routes/contacts/contacts_routes.py` — `@router.get('/list')` — Returns contact list filtered by search query.
+- `src/tools/contacts.py` — `ContactsTool` — Agent tool for querying user address book contacts.
+
+### Unknowns
+
+- VCard 3.0 vs 4.0 property parsing mismatches.

docs/discovery/domains/cookbook.md

@@ -0,0 +1,85 @@
+# Cookbook
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## COOKBOOK-001 — Local Model Download & Recipe Lifecycle Management
+
+- **Domain**: `cookbook`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Downloads HuggingFace models, configures execution parameters, and manages local GGUF/MLX model servers.
+
+### Evidence summary
+
+- `routes/cookbook_routes.py` — `setup_cookbook_routes` — Exposes model downloading and process serving endpoints.
+- `static/js/cookbook.js` — `initCookbook` — UI manager for local model library.
+
+### Unknowns
+
+- Disk space exhaustion during multi-gigabyte GGUF weights downloads.
+
+## COOKBOOK-002 — Hardware Model Fitting ('What Fits?') Analysis Engine
+
+- **Domain**: `cookbook`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Calculates RAM/VRAM requirements, quantized size, and context overhead to determine model compatibility.
+
+### Evidence summary
+
+- `routes/hwfit_routes.py` — `setup_hwfit_routes` — Calculates hardware model compatibility.
+- `services/hwfit/fit.py` — `calculate_fit` — Performs parameter and memory fit calculations.
+
+### Unknowns
+
+- Inaccurate VRAM estimation for non-standard KV-cache quantization.
+
+## COOKBOOK-003 — HuggingFace & MLX Model Discovery Services
+
+- **Domain**: `cookbook`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Searches HuggingFace Hub and local MLX model repositories for compatible GGUF and MLX weights.
+
+### Evidence summary
+
+- `services/hwfit/hf_discovery.py` — `search_hf_models` — Queries HuggingFace API for model tags and files.
+
+### Unknowns
+
+- HuggingFace API rate limits when searching without an API token.
+
+## COOKBOOK-004 — Host Docker Access for Inference Container Runtimes
+
+- **Domain**: `cookbook`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires Docker access and supported physical GPU hardware.
+
+### Purpose
+
+Detects and connects to host Docker engine to launch containerized Ollama, vLLM, or SGLang runtimes.
+
+### Evidence summary
+
+- `src/host_docker_access.py` — `HostDockerAccess` — Interacts with host docker daemon.
+
+### Unknowns
+
+- Permission denied accessing docker socket on non-root setups.

docs/discovery/domains/document.md

@@ -0,0 +1,108 @@
+# Document
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## DOCUMENT-001 — Document & Canvas Artifact Management
+
+- **Domain**: `document`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Renders dynamic canvas documents, handles live editing, markdown preview, and side-by-side artifact display.
+
+### Evidence summary
+
+- `routes/document_routes.py` — `setup_document_routes` — Registers document artifact CRUD routes.
+- `static/js/document.js` — `initDocumentView` — Renders interactive canvas document panel.
+
+### Unknowns
+
+- Concurrent edits on the same document artifact.
+
+## DOCUMENT-002 — PDF Form Processing & High-Fidelity Rendering
+
+- **Domain**: `document`
+- **Status**: `verified`
+- **Evidence Maturity**: `E1`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires optional PyMuPDF (`fitz`) or pypdf runtime dependency.
+
+### Purpose
+
+Extracts form fields from PDF files, fills dynamic values, and generates PDF previews.
+
+### Evidence summary
+
+- `src/pdf_runtime.py` — `load_pymupdf_for_pdf_viewer` — Loads optional PyMuPDF runtime for PDF viewing.
+- `src/pdf_forms.py` — `extract_form_fields` — Handles PDF form field extraction and filling.
+- `tests/test_document_pdf_marker.py` — `test_marker_removed_without_eating_following_text` — Tests PDF text extraction wrapper stripping without content corruption.
+
+### Unknowns
+
+- Complex XFA PDF forms may not extract cleanly with standard pdf parsers.
+
+## DOCUMENT-003 — Personal Document Indexing & RAG Retrieval
+
+- **Domain**: `document`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Indexes local user documents (PDF, DOCX, TXT) into ChromaDB for semantic vector retrieval.
+
+### Evidence summary
+
+- `routes/personal_routes.py` — `setup_personal_routes` — Personal document RAG indexing and search API endpoints.
+- `src/personal_docs.py` — `PersonalDocsManager` — Handles file text chunking and vector storage.
+
+### Unknowns
+
+- Slow vector embedding indexing step for massive multi-thousand page documents.
+
+## DOCUMENT-004 — Document Conversion & Text Extraction Engine
+
+- **Domain**: `document`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Converts office formats (.docx, .xlsx, .pptx) and HTML into clean Markdown text representations.
+
+### Evidence summary
+
+- `src/markitdown_runtime.py` — `convert_to_markdown` — Converts binary office documents into structured Markdown text.
+
+### Unknowns
+
+- Formatting loss when parsing legacy binary doc/xls files.
+
+## DOCUMENT-005 — Document Library UI Navigation
+
+- **Domain**: `document`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides dedicated UI view for browsing, filtering, and organizing saved user documents.
+
+### Evidence summary
+
+- `static/js/documentLibrary.js` — `initDocumentLibrary` — Renders document library navigation grid.
+- `app.py` — `serve_library` — Serves SPA shell for /library route.
+
+### Unknowns
+
+- Large folder trees may cause initial DOM render slowdown.

docs/discovery/domains/email.md

@@ -0,0 +1,87 @@
+# Email
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## EMAIL-001 — Email Account Setup, IMAP/SMTP Connection & Polling
+
+- **Domain**: `email`
+- **Status**: `verified`
+- **Evidence Maturity**: `E1`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires a controlled live IMAP account and network access.
+
+### Purpose
+
+Configures IMAP/SMTP email accounts, validates TLS certificates, and polls background inbox updates.
+
+### Evidence summary
+
+- `routes/email_routes.py` — `setup_email_routes` — Sets up email account management and synchronization routes.
+- `routes/email_pollers.py` — `_start_poller` — Background poller for email inbox synchronization.
+- `tests/test_service_health_email.py` — `test_email_ok_all_connect` — Tests IMAP connection health probing and status reporting.
+
+### Unknowns
+
+- Account lockouts if bad credentials are repeatedly polled.
+
+## EMAIL-002 — Email Searching, Threading & Message Operations
+
+- **Domain**: `email`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Parses email headers, folds signatures, groups messages into threads, and executes full-text email search.
+
+### Evidence summary
+
+- `routes/email_routes.py` — `@router.get('/search')` — Executes search across cached email headers and text.
+- `src/email_thread_parser.py` — `parse_email_thread` — Builds conversation tree from Message-ID and In-Reply-To headers.
+
+### Unknowns
+
+- Malformed MIME email structures failing HTML sanitization.
+
+## EMAIL-003 — Email Composition, Draft Management & Sending
+
+- **Domain**: `email`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires a controlled live SMTP account and network access.
+
+### Purpose
+
+Creates, saves, and dispatches HTML/plaintext email messages via SMTP.
+
+### Evidence summary
+
+- `routes/email_routes.py` — `@router.post('/send')` — Sends email message via user SMTP credentials.
+
+### Unknowns
+
+- SMTP connection drop mid-send causing unsent mail state.
+
+## EMAIL-004 — Email MCP Server & Codex Integration Bridge
+
+- **Domain**: `email`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Exposes constrained email reading and draft capabilities to external Codex / MCP agents with scope checks.
+
+### Evidence summary
+
+- `mcp_servers/email_server.py` — `EmailMcpServer` — MCP server exposing email tools over stdio/SSE.
+- `routes/codex_routes.py` — `setup_codex_routes` — Bridge endpoints for external Codex plugin integration.
+
+### Unknowns
+
+- Unauthorized mail sending if token scopes are improperly scoped.

docs/discovery/domains/frontend.md

@@ -0,0 +1,149 @@
+# Frontend
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## FRONTEND-001 — Single Page Application Shell & Client Router
+
+- **Domain**: `frontend`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Main HTML5 SPA shell, DOM lifecycle initializers, tab navigation, and deep-link route handlers.
+
+### Evidence summary
+
+- `static/index.html` — `index.html` — Main SPA entry point containing modal roots and CSS bundles.
+- `app.py` — `serve_index` — Serves index.html with dynamically generated CSP nonces.
+
+### Unknowns
+
+- Stale browser static cache if asset hashing is omitted during deployment.
+
+## FRONTEND-002 — Dynamic Theme, Color System & Custom Fonts
+
+- **Domain**: `frontend`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Supports dark/light themes, custom CSS variables, color picker controls, and user font uploads.
+
+### Evidence summary
+
+- `static/js/theme.js` — `applyTheme` — Applies custom HSL theme variables to DOM document root.
+- `routes/font_routes.py` — `setup_font_routes` — Allows uploading and serving custom WOFF2 font files.
+
+### Unknowns
+
+- Flash of unstyled content (FOUC) on slow connections.
+
+## FRONTEND-003 — Window Manager, Tile Layout & Modal Control System
+
+- **Domain**: `frontend`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Manages draggable tool windows, snapped multi-tile viewports, modal dialog Z-ordering, and ESC key stacks.
+
+### Evidence summary
+
+- `static/js/modalManager.js` — `ModalManager` — Controls modal open/close transitions and focus trapping.
+- `static/js/tileManager.js` — `TileManager` — Handles viewport split-pane grid arrangements.
+
+### Unknowns
+
+- Overlap artifacts when opening many simultaneous tool floating windows.
+
+## FRONTEND-004 — Global Keyboard Shortcuts & Accessibility Controls
+
+- **Domain**: `frontend`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides configurable hotkeys (Ctrl+K search, Esc close, Alt+1-9 tabs) and high-contrast accessibility options.
+
+### Evidence summary
+
+- `static/js/keyboard-shortcuts.js` — `initShortcuts` — Binds global keydown handlers for system shortcuts.
+- `static/js/a11y.js` — `initA11y` — Applies ARIA roles and dyslexic font toggles.
+
+### Unknowns
+
+- Browser keybinding collisions with browser default hotkeys.
+
+## FRONTEND-005 — Markdown, LaTeX & Code Block Streaming Renderer
+
+- **Domain**: `frontend`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Parses incoming SSE markdown streams, renders KaTeX math formulas, syntax-highlighted code, and interactive runners.
+
+### Evidence summary
+
+- `static/js/markdown.js` — `renderMarkdown` — Converts markdown prose to HTML nodes with syntax highlighting.
+- `static/js/streamingSegmenter.js` — `Segmenter` — Parses un-closed markdown fences during live stream.
+
+### Unknowns
+
+- DOM thrashing if streaming segmenter updates UI too frequently.
+
+## FRONTEND-006 — Interactive Tour & Guided Onboarding System
+
+- **Domain**: `frontend`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Presents interactive step-by-step feature tours and UI tooltip hints for new users.
+
+### Evidence summary
+
+- `static/js/tourHints.js` — `startTour` — Renders guided feature tour overlays over target UI elements.
+
+### Unknowns
+
+- Tour step misalignment if window is resized mid-tour.
+
+## FRONTEND-007 — Background Effects Prototyping Sandbox
+
+- **Domain**: `frontend`
+- **Status**: `dead-code-candidate`
+- **Evidence Maturity**: `E1`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Standalone sandbox page for prototyping visual background animations, waves, and whirlpool effects.
+
+### Evidence summary
+
+- `app.py` — `serve_backgrounds` — Serves visual background sandbox HTML page route.
+- `static/wave-variants.html` — `wave-variants.html` — Interactive background effect prototyping sandbox variant.
+
+### Unknowns
+
+- Route `/backgrounds` in app.py L918 attempts to serve `static/backgrounds.html` which is missing from disk; variant templates `wave-variants.html` and `whirlpool-variants.html` exist.

docs/discovery/domains/media.md

@@ -0,0 +1,170 @@
+# Media
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## MEDIA-001 — Gallery Image Library & Album Operations
+
+- **Domain**: `media`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Organizes images into custom albums, provides grid browsing, tagging, and album metadata management.
+
+### Evidence summary
+
+- `routes/gallery/gallery_routes.py` — `@router.get('/api/gallery/library')` — Fetches image library list with tag filters.
+- `static/js/gallery.js` — `initGallery` — Main gallery grid renderer and uploader.
+
+### Unknowns
+
+- Thumbnail generation overhead for high-resolution RAW camera images.
+
+## MEDIA-002 — Image Processing, AI Upscaling & Style Transfer
+
+- **Domain**: `media`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Executes local image enhancement, background removal, face sharpening, and AI upscaling operations.
+
+### Evidence summary
+
+- `routes/gallery/gallery_routes.py` — `@router.post('/api/gallery/ai-upscale')` — Runs RealESRGAN image upscaling.
+- `routes/gallery/gallery_routes.py` — `@router.post('/api/image/remove-bg')` — Executes background removal pass.
+
+### Unknowns
+
+- High GPU memory allocation when upscaling 4K images.
+
+## MEDIA-003 — Interactive Image Canvas Editor & Persisted Drafts
+
+- **Domain**: `media`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides full multi-layer raster canvas editor, brush tools, transforms, masks, and draft project persistence.
+
+### Evidence summary
+
+- `routes/editor_draft_routes.py` — `setup_editor_draft_routes` — API routes for saving and loading canvas project drafts.
+- `static/js/editor/history-panel.js` — `HistoryManager` — Canvas undo/redo stack manager.
+
+### Unknowns
+
+- Browser memory leak if multi-gigabyte layer undo buffers are kept indefinitely.
+
+## MEDIA-004 — Text-to-Speech (TTS) Synthesis Service
+
+- **Domain**: `media`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Synthesizes spoken audio from text using local Kokoro, EdgeTTS, or OpenAI TTS engines.
+
+### Evidence summary
+
+- `routes/tts_routes.py` — `@router.post('/synthesize')` — Synthesizes TTS audio clip.
+- `services/tts/tts_service.py` — `TTSService` — Provider abstraction layer for audio speech generation.
+
+### Unknowns
+
+- Audio synthesis latency on CPU-only hardware setups.
+
+## MEDIA-005 — Speech-to-Text (STT) Audio Transcription Service
+
+- **Domain**: `media`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Transcribes user audio recordings into text using faster-whisper or local speech models.
+
+### Evidence summary
+
+- `routes/stt_routes.py` — `@router.post('/transcribe')` — Accepts multipart audio file and returns transcription text.
+- `services/stt/stt_service.py` — `STTService` — Whisper audio transcription engine wrapper.
+
+### Unknowns
+
+- Missing ffmpeg system dependency prevents audio format decoding.
+
+## MEDIA-006 — Digital Signature Stamp Storage & Placement
+
+- **Domain**: `media`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Stores transparent PNG user signatures and stamps for placement onto PDF forms and documents.
+
+### Evidence summary
+
+- `routes/signature_routes.py` — `setup_signature_routes` — CRUD endpoints for managing user signature PNG stamps.
+
+### Unknowns
+
+- Cross-site scripting if signature image titles contain unescaped user input.
+
+## MEDIA-007 — Generated Image Artifact Route & MCP Integration
+
+- **Domain**: `media`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Serves generated AI artwork artifacts and integrates with image generation MCP server.
+
+### Evidence summary
+
+- `app.py` — `serve_generated_image` — Serves generated image artifacts with cache headers.
+- `src/generated_images.py` — `resolve_generated_image_path` — Confines requested image path within artifacts directory.
+
+### Unknowns
+
+- Path traversal vulnerability if filename parameter is un-sanitized.
+
+## MEDIA-008 — Native MLX Image Bridge (macOS Apple Silicon)
+
+- **Domain**: `media`
+- **Status**: `experimental`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires Apple Silicon, macOS tooling, and the compiled MLX bridge.
+
+### Purpose
+
+Native Apple Swift bridge for hardware-accelerated diffusion and MLX image colorization on macOS.
+
+### Evidence summary
+
+- `swift/odysseus-mlx-image-bridge/Package.swift` — `Package` — Swift package manifest for native MLX image bridge.
+- `scripts/mlx_image_server.py` — `main` — Python daemon wrapping native Swift MLX binary.
+
+### Unknowns
+
+- Binary build requires Xcode command line tools build step (`build-macos-app.sh`).

docs/discovery/domains/memory.md

@@ -0,0 +1,24 @@
+# Memory
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## MEMORY-001 — Persistent Long-Term Memory & Vector Indexing
+
+- **Domain**: `memory`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Extracts facts, user preferences, and temporal memories from chat sessions into vector/relational storage.
+
+### Evidence summary
+
+- `routes/memory/memory_routes.py` — `@router.get('')` — Fetches long-term user memory timeline.
+- `services/memory/memory_extractor.py` — `MemoryExtractor` — LLM-driven fact extraction from conversation transcripts.
+
+### Unknowns
+
+- Conflicting memory facts extracted from contradictory user prompts.

docs/discovery/domains/model.md

@@ -0,0 +1,177 @@
+# Model
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## MODEL-001 — Multi-Provider LLM Model Discovery & Metadata Management
+
+- **Domain**: `model`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Discovers models from OpenAI, Anthropic, Ollama, vLLM, LMStudio, OpenRouter, and Google AI Studio endpoints.
+
+### Evidence summary
+
+- `routes/model_routes.py` — `@router.get('/api/models')` — Returns unified list of available models across providers.
+- `src/model_discovery.py` — `ModelDiscovery.discover_all` — Queries connected provider endpoints for available model IDs.
+
+### Unknowns
+
+- Remote endpoint timeouts may slow down full discovery refresh.
+
+## MODEL-002 — Model Capability & Context Limits Detection
+
+- **Domain**: `model`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Detects vision, tool calling, reasoning, and context window limits for connected model endpoints.
+
+### Evidence summary
+
+- `src/model_capabilities.py` — `get_model_capabilities` — Maps model names to vision and tool support flags.
+- `src/endpoint_resolver.py` — `resolve_endpoint_headers` — Resolves auth headers and target URLs for model endpoints.
+
+### Unknowns
+
+- Incorrect context limit metadata for unlisted custom fine-tunes.
+
+## MODEL-003 — LLM Core Provider Communication & Fallback Routing
+
+- **Domain**: `model`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Provider dispatch, header injection and fallback advancement are only observable against a reachable LLM provider endpoint; not exercised in this documentation pass.
+
+### Purpose
+
+Manages HTTP request dispatching, authorization header injection, and fallback provider routing for LLM calls.
+
+### Evidence summary
+
+- `src/llm_core.py` — `llm_call_async` — Non-streaming provider request dispatcher: resolves the endpoint, injects authorization headers and executes the HTTP call.
+- `src/llm_core.py` — `llm_call_async_with_fallback` — Ordered fallback wrapper that retries llm_call_async across the configured candidate endpoints.
+- `src/llm_core.py` — `stream_llm_with_fallback` — Ordered fallback wrapper for the streaming path; advances to the next candidate when a provider yields an empty completion.
+
+### Unknowns
+
+- Unexpected API changes in upstream third-party model providers.
+
+## MODEL-004 — Model Selection & Display Ordering Preferences
+
+- **Domain**: `model`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Allows pinning, sorting, and hiding specific models in the UI selection dropdown.
+
+### Evidence summary
+
+- `routes/model_routes.py` — `@router.post('/order')` — Saves custom model display order preference.
+
+### Unknowns
+
+- Stale model IDs in custom order lists after model endpoints are removed.
+
+## MODEL-005 — Side-by-Side Model Comparison (A/B Testing)
+
+- **Domain**: `model`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Enables dual-model side-by-side response evaluation, arena scoring, and latency comparison.
+
+### Evidence summary
+
+- `routes/compare/compare_routes.py` — `@router.post('/start')` — Starts a parallel dual-model comparison stream.
+- `static/js/compare/index.js` — `initCompareView` — Renders side-by-side model chat panes.
+
+### Unknowns
+
+- High memory and network usage when streaming two model responses simultaneously.
+
+## MODEL-006 — GitHub Copilot Device Flow Authentication
+
+- **Domain**: `model`
+- **Status**: `verified`
+- **Evidence Maturity**: `E2`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires an interactive GitHub Copilot OAuth device-flow account.
+
+### Purpose
+
+Authenticates with GitHub Copilot via OAuth device flow to use Copilot models directly.
+
+### Evidence summary
+
+- `routes/copilot_routes.py` — `setup_copilot_routes` — Builds the Copilot device-flow router at prefix /api/copilot, wiring _start_device_flow and _poll_device_flow.
+- `routes/device_flow.py` — `create_device_flow_router` — Shared factory registering POST /device/start and POST /device/poll under the caller-supplied prefix.
+- `src/copilot.py` — `request_device_code` — Issues the GitHub device-code request that begins the Copilot OAuth device flow.
+- `src/copilot.py` — `poll_access_token` — Polls GitHub for the access token once the user has authorized the device code.
+- `tests/test_provider_device_flow_js.py` — `test_copilot_success_uses_complete_verification_uri` — Inspected unit test asserting the Copilot device-flow runner surfaces the complete verification URI returned by the backend.
+
+### Unknowns
+
+- Token expiration requires manual device re-authentication.
+
+## MODEL-007 — ChatGPT Subscription Device Flow Authentication
+
+- **Domain**: `model`
+- **Status**: `verified`
+- **Evidence Maturity**: `E2`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires an interactive ChatGPT subscription OAuth flow.
+
+### Purpose
+
+Authenticates with ChatGPT Pro/Plus subscription tokens via device login flow.
+
+### Evidence summary
+
+- `routes/chatgpt_subscription_routes.py` — `setup_chatgpt_subscription_routes` — Builds the ChatGPT subscription device-flow router at prefix /api/chatgpt-subscription.
+- `routes/device_flow.py` — `create_device_flow_router` — Shared factory registering POST /device/start and POST /device/poll under the caller-supplied prefix.
+- `src/chatgpt_subscription.py` — `request_device_code` — Issues the ChatGPT device-authorization request that begins the subscription OAuth device flow.
+- `src/chatgpt_subscription.py` — `poll_device_auth` — Polls the ChatGPT device-authorization endpoint for completion using the stored device_auth_id and user_code.
+- `tests/test_provider_device_flow_js.py` — `test_chatgpt_success_uses_plain_verification_uri` — Inspected unit test asserting the ChatGPT device-flow runner uses the plain verification URI rather than the Copilot complete-URI form.
+
+### Unknowns
+
+- Changes in OpenAI auth endpoint security challenges.
+
+## MODEL-008 — Embedding Model Lane & Vector Provider Setup
+
+- **Domain**: `model`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Configures local sentence-transformers, FastEmbed, or remote OpenAI embedding model lanes.
+
+### Evidence summary
+
+- `routes/embedding_routes.py` — `setup_embedding_routes` — Provides embedding provider configuration endpoints.
+- `src/embeddings.py` — `EmbeddingManager` — Generates dense vector embeddings for RAG and memory.
+
+### Unknowns
+
+- First-time download of heavy PyTorch model weights on CPU-only machines.

docs/discovery/domains/note.md

@@ -0,0 +1,24 @@
+# Note
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## NOTE-001 — Interactive Notes & Checklist Management
+
+- **Domain**: `note`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides Google Keep-style notes, rich markdown text, checklist items, pinning, color tags, and reminders.
+
+### Evidence summary
+
+- `routes/note/note_routes.py` — `@router.get('')` — Lists all user notes with pin and archive states.
+- `static/js/notes.js` — `initNotesView` — Main interactive notes grid and modal manager.
+
+### Unknowns
+
+- Concurrent edits on note item checkboxes.

docs/discovery/domains/platform.md

@@ -0,0 +1,213 @@
+# Platform
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## PLATFORM-001 — Application Initialization & Lifespan Management
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Orchestrates server startup, database table migration, background daemon initialization, and clean shutdown.
+
+### Evidence summary
+
+- `app.py` — `_lifespan` — FastAPI lifespan context manager executing startup tasks.
+- `src/app_initializer.py` — `initialize_app` — Initializes app directories, DB schemas, and logging.
+
+### Unknowns
+
+- Un-handled exceptions during startup halt application launch.
+
+## PLATFORM-002 — System Health, Readiness & Version Monitoring APIs
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Exposes Liveness (/api/health), Readiness (/api/ready), App Version (/api/version), and Client Perf APIs.
+
+### Evidence summary
+
+- `app.py` — `readiness_check` — Performs system component integrity check.
+- `src/readiness.py` — `check_readiness` — Checks database, storage, and key paths for read/write access.
+
+### Unknowns
+
+- Readiness check delays if verifying connectivity to offline remote endpoints.
+
+## PLATFORM-003 — Database Schema, Migrations & SQLite Persistence
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Defines core relational tables (users, tokens, tasks, sessions) and executes automated SQLite schema upgrades.
+
+### Evidence summary
+
+- `core/database.py` — `init_db` — Creates ORM tables and establishes connection pool.
+- `scripts/update_database.py` — `run_migrations` — Applies missing schema columns and indices.
+
+### Unknowns
+
+- SQLite file lock contention under high concurrent write loads.
+
+## PLATFORM-004 — User Data Export & Import Backup Infrastructure
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Exports complete user workspace state (sessions, memory, skills, notes, presets) into a zip archive.
+
+### Evidence summary
+
+- `routes/backup_routes.py` — `setup_backup_routes` — Handles workspace data export and import upload unpack.
+- `docs/backup-restore.md` — `Documentation` — Backup and restore operational documentation.
+
+### Unknowns
+
+- Corrupt archive files causing partial data restore.
+
+## PLATFORM-005 — File Cleanup & Storage Maintenance Engine
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Scans data directories for orphaned files, old uploads, temporary vision images, and frees disk space.
+
+### Evidence summary
+
+- `routes/cleanup/cleanup_routes.py` — `@router.get('/preview')` — Previews reclaimable disk space across storage directories.
+- `src/cleanup_service.py` — `CleanupService` — Executes filesystem purge of orphaned asset files.
+
+### Unknowns
+
+- Deletes files uploaded in active sessions if retention window is set too short.
+
+## PLATFORM-006 — System Health & RAG Diagnostic Suite
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Executes real-time integrity diagnostics across ChromaDB, SearXNG, local models, and network interfaces.
+
+### Evidence summary
+
+- `routes/diagnostics_routes.py` — `setup_diagnostics_routes` — Runs subsystem health check suite.
+- `src/service_health.py` — `collect_health_status` — Inspects vector database, email, search, and local provider status.
+
+### Unknowns
+
+- Diagnostic timeout if external search provider is unreachable.
+
+## PLATFORM-007 — Desktop CLI Utilities & Shell Integration Tools
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides command-line interface tools (`odysseus`, `odysseus-mcp`, `odysseus-mail`) for terminal usage.
+
+### Evidence summary
+
+- `scripts/_lib/cli.py` — `main` — Shared CLI framework for terminal helper commands.
+- `scripts/odysseus` — `odysseus` — Main terminal launcher script.
+
+### Unknowns
+
+- Outdated CLI scripts if backend API schemas change.
+
+## PLATFORM-008 — Desktop Companion App Integration
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides API routes and pairing mechanisms for the native macOS/desktop menu bar companion app.
+
+### Evidence summary
+
+- `companion/routes.py` — `setup_companion_routes` — Endpoints for pairing and status sync with desktop companion.
+- `companion/pairing.py` — `PairingManager` — Generates and validates companion pairing codes.
+
+### Unknowns
+
+- Pairing code expiration timing window.
+
+## PLATFORM-009 — Docker Containerization & GPU Hardware Manifests
+
+- **Domain**: `platform`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires Docker GPU pass-through and compatible host drivers.
+
+### Purpose
+
+Provides multi-stage Dockerfile and Docker Compose manifests for CPU, NVIDIA CUDA, and AMD ROCm GPUs.
+
+### Evidence summary
+
+- `Dockerfile` — `multi-stage-build` — Multi-stage container build environment.
+- `docker-compose.gpu-nvidia.yml` — `nvidia-gpu-manifest` — NVIDIA GPU pass-through container specification.
+- `scripts/check-docker-gpu.sh` — `check-docker-gpu` — Automated diagnostic test script for host NVIDIA GPU passthrough.
+
+### Unknowns
+
+- Driver version incompatibility with host NVIDIA/AMD kernel drivers.
+
+## PLATFORM-010 — Legacy FAISS Vector Index Migration Script
+
+- **Domain**: `platform`
+- **Status**: `legacy`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Legacy utility script to migrate older FAISS vector indices into ChromaDB.
+
+### Evidence summary
+
+- `scripts/migrate_faiss_to_chroma.py` — `migrate_faiss` — Reads FAISS vector index files and writes to ChromaDB collection.
+
+### Unknowns
+
+- Superseded by native ChromaDB vector index pipeline.

docs/discovery/domains/research.md

@@ -0,0 +1,86 @@
+# Research
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## RESEARCH-001 — Deep Research Execution Engine & SSE Progress Streaming
+
+- **Domain**: `research`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Executes multi-step recursive deep research tasks, web page scraping, synthesis, and streams live progress.
+
+### Evidence summary
+
+- `routes/research/research_routes.py` — `@router.post('/api/research/start')` — Initiates deep research job.
+- `src/deep_research.py` — `DeepResearchEngine` — Recursive search and summary crawler.
+
+### Unknowns
+
+- High memory consumption when parsing multi-megabyte HTML target pages.
+
+## RESEARCH-002 — Research Library, Detail View & Image Controls
+
+- **Domain**: `research`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Stores completed research reports, generated diagrams, reference links, and manages image visibility.
+
+### Evidence summary
+
+- `routes/research/research_routes.py` — `@router.get('/api/research/library')` — Returns all saved research reports.
+
+### Unknowns
+
+- Orphaned report files if storage directory is modified out-of-band.
+
+## RESEARCH-003 — Web Search Engine Integration (SearXNG & Multi-Provider)
+
+- **Domain**: `research`
+- **Status**: `verified`
+- **Evidence Maturity**: `E1`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires an active SearXNG instance or external search API provider.
+
+### Purpose
+
+Queries SearXNG, DuckDuckGo, or Google Search instances to retrieve web search snippets.
+
+### Evidence summary
+
+- `routes/search_routes.py` — `setup_search_routes` — Defines /api/search, /api/search/config, and /api/search/query endpoints.
+- `src/search/core.py` — `SearchEngine` — Compatibility module aliasing services.search.core.
+- `tests/test_search_ranking.py` — `test_news_queries_prefer_news_sources_over_sports_and_social_results` — Tests search result domain ranking and scoring.
+
+### Unknowns
+
+- Search provider IP throttling or rate-limiting.
+
+## RESEARCH-004 — Research Result Peeking & Topic Spinoff Generation
+
+- **Domain**: `research`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Extracts preliminary research snippets and spawns child research sessions focused on specific sub-topics.
+
+### Evidence summary
+
+- `routes/research/research_routes.py` — `@router.post('/api/research/spinoff/{session_id}')` — Spawns child research session for specific query.
+
+### Unknowns
+
+- Deep recursion tree depth when spawning multiple nested spinoffs.

docs/discovery/domains/security.md

@@ -0,0 +1,154 @@
+# Security
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## SECURITY-001 — Authentication, Session Cookies & User Management
+
+- **Domain**: `security`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Handles bcrypt password hashing, session cookie issuance, authentication enforcement, and user administration.
+
+### Evidence summary
+
+- `routes/auth_routes.py` — `@router.post('/login')` — Authenticates credentials and sets session cookie.
+- `core/auth.py` — `AuthManager` — Handles user creation, password verification, and session tokens.
+
+### Unknowns
+
+- Cookie session hijack if deployed over unencrypted HTTP without HTTPS cookie flags.
+
+## SECURITY-002 — System Vault Encrypted Secret Storage
+
+- **Domain**: `security`
+- **Status**: `verified`
+- **Evidence Maturity**: `E1`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: pending — Requires installed Bitwarden CLI (`bw`) executable.
+
+### Purpose
+
+Encrypts API keys, passphrases, and third-party secrets on disk using AES-GCM / PBKDF2 key derivation.
+
+### Evidence summary
+
+- `routes/vault_routes.py` — `setup_vault_routes` — Admin routes for vault configuration, login, unlock, lock, and logout.
+- `src/secret_storage.py` — `SecretStorage` — Fernet symmetric key DB secret encryption.
+- `tests/test_vault_password_not_in_argv.py` — `test_bw_password_not_in_argv` — Verifies master password is fed via stdin and never appears in process argv.
+
+### Unknowns
+
+- Loss of vault master passphrase renders all encrypted secrets permanently unrecoverable.
+
+## SECURITY-003 — API Token Management & Scope Access Control
+
+- **Domain**: `security`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Generates scoped API bearer tokens (read/write/admin) for external tool and script authentication.
+
+### Evidence summary
+
+- `routes/api_token_routes.py` — `setup_api_token_routes` — Exposes API token creation, scope assignment, and revocation.
+- `core/database.py` — `ApiToken` — SQLAlchemy ORM schema for API tokens and permissions.
+
+### Unknowns
+
+- Leaked API bearer tokens with excessive permission scopes.
+
+## SECURITY-004 — Prompt Security & Injection Defense Engine
+
+- **Domain**: `security`
+- **Status**: `verified`
+- **Evidence Maturity**: `E1`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Scans system prompts and external inputs for prompt injection attempts, jailbreaks, and sensitive data leaks.
+
+### Evidence summary
+
+- `src/prompt_security.py` — `untrusted_context_message` — Wraps untrusted context with guard delimiters and sets metadata.trusted = False.
+- `src/tool_security.py` — `NON_ADMIN_BLOCKED_TOOLS` — Enforces tool execution safety for non-admin user roles.
+- `tests/test_skill_index_prompt_injection.py` — `test_skill_index` — Verifies skill index descriptions cannot leak into trusted system prompts.
+- `tests/test_tool_output_prompt_injection.py` — `test_tool_output` — Tool output injection guards.
+
+### Unknowns
+
+- False positives blocking legitimate complex coding or security prompts.
+
+## SECURITY-005 — URL & Path Confinement Security Guards
+
+- **Domain**: `security`
+- **Status**: `verified`
+- **Evidence Maturity**: `E1`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Prevents SSRF attacks and path traversal by validating target IP addresses and resolving symlinks.
+
+### Evidence summary
+
+- `src/url_safety.py` — `check_outbound_url` — Rejects non-HTTP(S) schemes, link-local, cloud metadata SSRF addresses.
+- `src/url_security.py` — `validate_public_http_url` — Validates public-facing endpoints.
+- `tests/test_url_safety.py` — `test_url_safety` — Scheme validation, cloud metadata SSRF rejection, IP classification.
+- `tests/test_tool_path_confinement.py` — `test_path_confinement` — Path traversal checks.
+- `tests/test_workspace_confine.py` — `test_workspace_confine` — Workspace confinement checks.
+
+### Unknowns
+
+- DNS rebinding attacks if IP address is re-resolved post-validation.
+
+## SECURITY-006 — HTTP Security Headers Middleware
+
+- **Domain**: `security`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Injects standard OWASP HTTP security headers (CSP, HSTS, X-Content-Type-Options, X-Frame-Options).
+
+### Evidence summary
+
+- `core/middleware.py` — `SecurityHeadersMiddleware` — Sets strict security headers and CSP nonces on HTTP responses.
+
+### Unknowns
+
+- Strict Content Security Policy (CSP) blocking third-party embedded web resources.
+
+## SECURITY-007 — Admin System Data Wipe ('Danger Zone')
+
+- **Domain**: `security`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Provides administrative reset operations to wipe sessions, cache, uploaded files, or factory reset state.
+
+### Evidence summary
+
+- `routes/admin_wipe/admin_wipe_routes.py` — `@router.delete('/wipe/{kind}')` — Executes systemic data wipe based on requested scope.
+
+### Unknowns
+
+- Accidental catastrophic data loss if triggered without user confirmation.

docs/discovery/domains/skill.md

@@ -0,0 +1,24 @@
+# Skill
+
+Features in this document are generated from [`../feature-catalog.json`](../feature-catalog.json), the canonical inventory.
+
+## SKILL-001 — Dynamic Skill Management & Code Execution Engine
+
+- **Domain**: `skill`
+- **Status**: `verified`
+- **Evidence Maturity**: `E0`
+- **Commit Verified**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Runtime Validation**: not-required — No separate environment-dependent runtime validation was identified during this documentation pass.
+
+### Purpose
+
+Allows users to create, import, edit, test, and execute custom Python/Markdown skills dynamically.
+
+### Evidence summary
+
+- `routes/skills_routes.py` — `setup_skills_routes` — Exposes CRUD and remote import routes for user skills.
+- `services/memory/skills.py` — `SkillsManager` — Handles skill storage, parsing, and execution.
+
+### Unknowns
+
+- Arbitrary code execution risks if skill import URL is untrusted.

docs/discovery/feature-catalog.md

@@ -0,0 +1,85 @@
+# Feature Catalog
+
+This is a human-readable index derived from [`feature-catalog.json`](feature-catalog.json). The JSON file is canonical.
+
+| ID | Feature | Domain | Status | Evidence | Runtime |
+|---|---|---|---|---|---|
+| `AGENT-001` | Autonomous Agent Loop & Tool Execution Engine | `agent` | `verified` | `E0` | not required |
+| `AGENT-002` | Scheduled Tasks & Event Bus Dispatcher | `agent` | `verified` | `E0` | not required |
+| `AGENT-003` | Webhook Event Subscriptions & Trigger Processing | `agent` | `verified` | `E0` | not required |
+| `AGENT-004` | Assistant Settings, Task Check-Ins & Background Job Monitor | `agent` | `partial` | `E1` | not required |
+| `AGENT-005` | Model Context Protocol (MCP) Server Integration | `agent` | `verified` | `E0` | not required |
+| `AGENT-006` | AI Interaction Tools & Pipeline Orchestration | `agent` | `verified` | `E0` | not required |
+| `AGENT-007` | Subprocess & Background Job Execution Tools | `agent` | `verified` | `E0` | not required |
+| `CALENDAR-001` | CalDAV Calendar Synchronization & Account Setup | `calendar` | `verified` | `E0` | pending |
+| `CALENDAR-002` | Calendar Event Operations & iCalendar Parsing | `calendar` | `verified` | `E0` | not required |
+| `CHAT-001` | Core Chat Streaming & SSE Message Generation | `chat` | `verified` | `E2` | pending |
+| `CHAT-002` | Session Management & Conversation State | `chat` | `verified` | `E0` | not required |
+| `CHAT-003` | Chat History & Message Editing/Truncation | `chat` | `verified` | `E0` | not required |
+| `CHAT-004` | File & Multimodal Attachment Handling | `chat` | `verified` | `E0` | not required |
+| `CHAT-005` | Chat Message Search | `chat` | `verified` | `E0` | not required |
+| `CHAT-006` | System Prompts & Preset Management | `chat` | `verified` | `E0` | not required |
+| `CHAT-007` | Emoji Rendering & Twemoji SVG Proxy | `chat` | `verified` | `E0` | not required |
+| `CHAT-008` | Input History Recall (Arrow Up) | `chat` | `verified` | `E0` | not required |
+| `CHAT-009` | Context Window Compaction & Truncation | `chat` | `verified` | `E0` | not required |
+| `CONTACT-001` | CardDAV Contact Management & Address Book Integration | `contact` | `verified` | `E0` | not required |
+| `COOKBOOK-001` | Local Model Download & Recipe Lifecycle Management | `cookbook` | `verified` | `E0` | not required |
+| `COOKBOOK-002` | Hardware Model Fitting ('What Fits?') Analysis Engine | `cookbook` | `verified` | `E0` | not required |
+| `COOKBOOK-003` | HuggingFace & MLX Model Discovery Services | `cookbook` | `verified` | `E0` | not required |
+| `COOKBOOK-004` | Host Docker Access for Inference Container Runtimes | `cookbook` | `verified` | `E0` | pending |
+| `DOCUMENT-001` | Document & Canvas Artifact Management | `document` | `verified` | `E0` | not required |
+| `DOCUMENT-002` | PDF Form Processing & High-Fidelity Rendering | `document` | `verified` | `E1` | pending |
+| `DOCUMENT-003` | Personal Document Indexing & RAG Retrieval | `document` | `verified` | `E0` | not required |
+| `DOCUMENT-004` | Document Conversion & Text Extraction Engine | `document` | `verified` | `E0` | not required |
+| `DOCUMENT-005` | Document Library UI Navigation | `document` | `verified` | `E0` | not required |
+| `EMAIL-001` | Email Account Setup, IMAP/SMTP Connection & Polling | `email` | `verified` | `E1` | pending |
+| `EMAIL-002` | Email Searching, Threading & Message Operations | `email` | `verified` | `E0` | not required |
+| `EMAIL-003` | Email Composition, Draft Management & Sending | `email` | `verified` | `E0` | pending |
+| `EMAIL-004` | Email MCP Server & Codex Integration Bridge | `email` | `verified` | `E0` | not required |
+| `FRONTEND-001` | Single Page Application Shell & Client Router | `frontend` | `verified` | `E0` | not required |
+| `FRONTEND-002` | Dynamic Theme, Color System & Custom Fonts | `frontend` | `verified` | `E0` | not required |
+| `FRONTEND-003` | Window Manager, Tile Layout & Modal Control System | `frontend` | `verified` | `E0` | not required |
+| `FRONTEND-004` | Global Keyboard Shortcuts & Accessibility Controls | `frontend` | `verified` | `E0` | not required |
+| `FRONTEND-005` | Markdown, LaTeX & Code Block Streaming Renderer | `frontend` | `verified` | `E0` | not required |
+| `FRONTEND-006` | Interactive Tour & Guided Onboarding System | `frontend` | `verified` | `E0` | not required |
+| `FRONTEND-007` | Background Effects Prototyping Sandbox | `frontend` | `dead-code-candidate` | `E1` | not required |
+| `MEDIA-001` | Gallery Image Library & Album Operations | `media` | `verified` | `E0` | not required |
+| `MEDIA-002` | Image Processing, AI Upscaling & Style Transfer | `media` | `verified` | `E0` | not required |
+| `MEDIA-003` | Interactive Image Canvas Editor & Persisted Drafts | `media` | `verified` | `E0` | not required |
+| `MEDIA-004` | Text-to-Speech (TTS) Synthesis Service | `media` | `verified` | `E0` | not required |
+| `MEDIA-005` | Speech-to-Text (STT) Audio Transcription Service | `media` | `verified` | `E0` | not required |
+| `MEDIA-006` | Digital Signature Stamp Storage & Placement | `media` | `verified` | `E0` | not required |
+| `MEDIA-007` | Generated Image Artifact Route & MCP Integration | `media` | `verified` | `E0` | not required |
+| `MEDIA-008` | Native MLX Image Bridge (macOS Apple Silicon) | `media` | `experimental` | `E0` | pending |
+| `MEMORY-001` | Persistent Long-Term Memory & Vector Indexing | `memory` | `verified` | `E0` | not required |
+| `MODEL-001` | Multi-Provider LLM Model Discovery & Metadata Management | `model` | `verified` | `E0` | not required |
+| `MODEL-002` | Model Capability & Context Limits Detection | `model` | `verified` | `E0` | not required |
+| `MODEL-003` | LLM Core Provider Communication & Fallback Routing | `model` | `verified` | `E0` | pending |
+| `MODEL-004` | Model Selection & Display Ordering Preferences | `model` | `verified` | `E0` | not required |
+| `MODEL-005` | Side-by-Side Model Comparison (A/B Testing) | `model` | `verified` | `E0` | not required |
+| `MODEL-006` | GitHub Copilot Device Flow Authentication | `model` | `verified` | `E2` | pending |
+| `MODEL-007` | ChatGPT Subscription Device Flow Authentication | `model` | `verified` | `E2` | pending |
+| `MODEL-008` | Embedding Model Lane & Vector Provider Setup | `model` | `verified` | `E0` | not required |
+| `NOTE-001` | Interactive Notes & Checklist Management | `note` | `verified` | `E0` | not required |
+| `PLATFORM-001` | Application Initialization & Lifespan Management | `platform` | `verified` | `E0` | not required |
+| `PLATFORM-002` | System Health, Readiness & Version Monitoring APIs | `platform` | `verified` | `E0` | not required |
+| `PLATFORM-003` | Database Schema, Migrations & SQLite Persistence | `platform` | `verified` | `E0` | not required |
+| `PLATFORM-004` | User Data Export & Import Backup Infrastructure | `platform` | `verified` | `E0` | not required |
+| `PLATFORM-005` | File Cleanup & Storage Maintenance Engine | `platform` | `verified` | `E0` | not required |
+| `PLATFORM-006` | System Health & RAG Diagnostic Suite | `platform` | `verified` | `E0` | not required |
+| `PLATFORM-007` | Desktop CLI Utilities & Shell Integration Tools | `platform` | `verified` | `E0` | not required |
+| `PLATFORM-008` | Desktop Companion App Integration | `platform` | `verified` | `E0` | not required |
+| `PLATFORM-009` | Docker Containerization & GPU Hardware Manifests | `platform` | `verified` | `E0` | pending |
+| `PLATFORM-010` | Legacy FAISS Vector Index Migration Script | `platform` | `legacy` | `E0` | not required |
+| `RESEARCH-001` | Deep Research Execution Engine & SSE Progress Streaming | `research` | `verified` | `E0` | not required |
+| `RESEARCH-002` | Research Library, Detail View & Image Controls | `research` | `verified` | `E0` | not required |
+| `RESEARCH-003` | Web Search Engine Integration (SearXNG & Multi-Provider) | `research` | `verified` | `E1` | pending |
+| `RESEARCH-004` | Research Result Peeking & Topic Spinoff Generation | `research` | `verified` | `E0` | not required |
+| `SECURITY-001` | Authentication, Session Cookies & User Management | `security` | `verified` | `E0` | not required |
+| `SECURITY-002` | System Vault Encrypted Secret Storage | `security` | `verified` | `E1` | pending |
+| `SECURITY-003` | API Token Management & Scope Access Control | `security` | `verified` | `E0` | not required |
+| `SECURITY-004` | Prompt Security & Injection Defense Engine | `security` | `verified` | `E1` | not required |
+| `SECURITY-005` | URL & Path Confinement Security Guards | `security` | `verified` | `E1` | not required |
+| `SECURITY-006` | HTTP Security Headers Middleware | `security` | `verified` | `E0` | not required |
+| `SECURITY-007` | Admin System Data Wipe ('Danger Zone') | `security` | `verified` | `E0` | not required |
+| `SKILL-001` | Dynamic Skill Management & Code Execution Engine | `skill` | `verified` | `E0` | not required |

docs/discovery/references/source-provenance.md

@@ -0,0 +1,28 @@
+# Source Provenance & Audit Baseline
+
+## Target Repository & Snapshot
+
+- **Repository**: `odysseus-dev/odysseus`
+- **Branch**: `discovery`
+- **Pinned Commit SHA**: `d8a2059df8e53bc7275c45339849d14c8651e73c`
+- **Snapshot Date**: `2026-07-23T14:49:02Z`
+
+## Discovery Package Organization
+
+The public discovery documentation package under `docs/discovery/` is structured as follows:
+
+- `feature-catalog.json`: Canonical machine-readable JSON catalog containing 79 feature records.
+- `feature-catalog.md`: Human-readable summary derived from `feature-catalog.json`.
+- `BASELINE-STATUS.md`: Publication status, evidence-validation snapshot, and durable maintainer guidance.
+- `audit-method.md`: Audit rules, scope, and evidence maturity definitions (E0 to E4).
+- `domains/`: 16 functional domain markdown files detailing feature implementations.
+- `references/`: Audit provenance and repository snapshot metadata.
+- `tools/`: Structural, consistency, and evidence validators with focused evidence-validator tests.
+
+## Exclusion Principles
+
+This public documentation package explicitly excludes:
+- Internal planning artifacts or private meeting notes.
+- Machine-specific filesystem paths or user environments.
+- API keys, credentials, or private service endpoints.
+- Application code or automated test mutations.

docs/discovery/tools/test_validate_discovery_evidence.py

@@ -0,0 +1,224 @@
+#!/usr/bin/env python3
+"""Focused negative tests for validate_discovery_evidence.py."""
+
+from __future__ import annotations
+
+import importlib.util
+import shutil
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+
+MODULE_PATH = Path(__file__).with_name("validate_discovery_evidence.py")
+SPEC = importlib.util.spec_from_file_location("validate_discovery_evidence", MODULE_PATH)
+assert SPEC and SPEC.loader
+validator = importlib.util.module_from_spec(SPEC)
+sys.modules[SPEC.name] = validator
+SPEC.loader.exec_module(validator)
+
+
+class FakeJavascriptParser:
+    supported = True
+    reason = "test parser"
+
+    def __init__(self, symbols: list[validator.Located] | None = None) -> None:
+        self.symbols = symbols or []
+
+    def parse(self, path: Path) -> list[validator.Located]:
+        return self.symbols
+
+
+class UnsupportedJavascriptParser:
+    supported = False
+    reason = "no repository-local parser"
+
+
+class EvidenceNegativeTests(unittest.TestCase):
+    def setUp(self) -> None:
+        self.temp = tempfile.TemporaryDirectory()
+        self.root = Path(self.temp.name)
+        (self.root / "sample.py").write_text(
+            "from fastapi import APIRouter\n"
+            "router = APIRouter(prefix='/api')\n"
+            "\n"
+            "class ChatHandler:\n"
+            "    def preprocess_message(self):\n"
+            "        return True\n"
+            "\n"
+            "@router.post('/chat')\n"
+            "def chat_stream():\n"
+            "    return True\n",
+            encoding="utf-8",
+        )
+        (self.root / "sample.js").write_text(
+            "export const present = () => true;\n", encoding="utf-8"
+        )
+        (self.root / "sample.sh").write_text(
+            "#!/usr/bin/env bash\nreal_function() {\n  return 0\n}\n",
+            encoding="utf-8",
+        )
+        self.backup = self.root / "copied-backups"
+        self.backup.mkdir()
+        for path in self.root.glob("sample.*"):
+            shutil.copy2(path, self.backup / path.name)
+
+    def tearDown(self) -> None:
+        for backup in self.backup.iterdir():
+            target = self.root / backup.name
+            shutil.copy2(backup, target)
+            self.assertEqual(target.read_bytes(), backup.read_bytes())
+        self.temp.cleanup()
+
+    def validate(
+        self,
+        evidence: dict[str, str],
+        javascript_parser: object | None = None,
+    ) -> validator.Validation:
+        return validator.validate_evidence(
+            self.root,
+            "TEST-001",
+            0,
+            evidence,
+            javascript_parser or FakeJavascriptParser(),
+        )
+
+    def test_missing_python_symbol(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.py",
+                "kind": "python-function",
+                "locator": "fabricated",
+                "line_range": "L1-L1",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "invalid-locator")
+
+    def test_incorrect_qualified_method(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.py",
+                "kind": "python-method",
+                "locator": "WrongHandler.preprocess_message",
+                "line_range": "L5-L6",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "invalid-locator")
+
+    def test_symbol_outside_cited_range(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.py",
+                "kind": "python-method",
+                "locator": "ChatHandler.preprocess_message",
+                "line_range": "L1-L2",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "locator-outside-range")
+
+    def test_fabricated_test_function(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.py",
+                "kind": "test-function",
+                "locator": "test_fabricated",
+                "line_range": "L1-L2",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "invalid-locator")
+
+    def test_nonexistent_javascript_symbol_with_parser(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.js",
+                "kind": "javascript-function",
+                "locator": "missing",
+                "line_range": "L1-L1",
+                "explanation": "negative fixture",
+            },
+            FakeJavascriptParser(
+                [validator.Located("present", "javascript-export", 1, 1)]
+            ),
+        )
+        self.assertEqual(result.result, "invalid-locator")
+
+    def test_unsupported_javascript_parser(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.js",
+                "kind": "javascript-function",
+                "locator": "present",
+                "line_range": "L1-L1",
+                "explanation": "negative fixture",
+            },
+            UnsupportedJavascriptParser(),
+        )
+        self.assertEqual(result.result, "unsupported")
+
+    def test_route_path_mismatch(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.py",
+                "kind": "python-route",
+                "locator": "POST /api/wrong -> chat_stream",
+                "line_range": "L9-L10",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "invalid-locator")
+        self.assertIn("path", result.problem or "")
+
+    def test_http_method_mismatch(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.py",
+                "kind": "python-route",
+                "locator": "GET /api/chat -> chat_stream",
+                "line_range": "L9-L10",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "invalid-locator")
+        self.assertIn("method", result.problem or "")
+
+    def test_shell_function_mismatch(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.sh",
+                "kind": "shell-function",
+                "locator": "fabricated",
+                "line_range": "L1-L4",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "invalid-locator")
+
+    def test_invalid_file_level_evidence(self) -> None:
+        result = self.validate(
+            {
+                "path": "missing.file",
+                "kind": "file",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "invalid-path")
+
+    def test_file_level_evidence_rejects_fake_symbol(self) -> None:
+        result = self.validate(
+            {
+                "path": "sample.sh",
+                "kind": "file",
+                "locator": "whole-script",
+                "explanation": "negative fixture",
+            }
+        )
+        self.assertEqual(result.result, "invalid-locator")
+
+
+if __name__ == "__main__":
+    unittest.main()

docs/discovery/tools/validate_discovery_consistency.py

@@ -0,0 +1,592 @@
+#!/usr/bin/env python3
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+from collections import Counter, defaultdict
+from pathlib import Path
+from typing import Any
+
+
+ROOT = Path(__file__).resolve().parents[1]
+CATALOG_PATH = ROOT / "feature-catalog.json"
+CATALOG_MD_PATH = ROOT / "feature-catalog.md"
+DOMAINS_DIR = ROOT / "domains"
+REVIEWS_DIR = ROOT / "reviews"
+
+EXPECTED_COMMIT = "d8a2059df8e53bc7275c45339849d14c8651e73c"
+EXPECTED_FEATURES = 79
+EXPECTED_DOMAINS = 16
+
+VALID_STATUSES = {
+    "verified",
+    "partial",
+    "disabled",
+    "experimental",
+    "legacy",
+    "dead-code-candidate",
+    "unverified",
+}
+
+VALID_MATURITY = {"E0", "E1", "E2", "E3", "E4"}
+
+VALID_RUNTIME = {
+    "not-required",
+    "pending",
+    "blocked",
+    "completed",
+}
+
+REQUIRED_FIELDS = {
+    "id",
+    "domain",
+    "name",
+    "purpose",
+    "status",
+    "evidence_maturity",
+    "verified_at_commit",
+    "evidence",
+    "runtime_validation",
+}
+
+FEATURE_ID_RE = re.compile(r"^[A-Z][A-Z0-9]*-\d{3}$")
+
+DOMAIN_HEADING_RE = re.compile(
+    r"^##\s+`?([A-Z][A-Z0-9]*-\d{3})`?"
+    r"\s+(?:—|-)\s+(.+?)\s*$"
+)
+
+DOMAIN_FIELD_RE = re.compile(
+    r"^-\s+\*\*"
+    r"(Domain|Status|Evidence Maturity|Commit Verified)"
+    r"\*\*:\s*(.*?)\s*$"
+)
+
+DOMAIN_RUNTIME_RE = re.compile(
+    r"^-\s+\*\*Runtime Validation\*\*:\s*(.*?)\s*$"
+)
+
+REVIEW_HEADING_RE = re.compile(
+    r"^###\s+(?:\d+\.\s+)?"
+    r"([A-Z][A-Z0-9]*-\d{3})"
+    r"\s+(?:—|-)\s+.+$"
+)
+
+REVIEW_RESULT_RE = re.compile(
+    r"^-\s+\*\*Resulting Status(?:\s+and|/)\s+Maturity\*\*:"
+    r"\s*`([^`]+)`\s*/\s*`([^`]+)`\s*$"
+)
+
+
+def clean(value: str) -> str:
+    value = value.strip()
+
+    if (
+        len(value) >= 2
+        and value.startswith("`")
+        and value.endswith("`")
+    ):
+        return value[1:-1].strip()
+
+    return value
+
+
+def load_catalog(errors: list[str]) -> list[dict[str, Any]]:
+    try:
+        data = json.loads(
+            CATALOG_PATH.read_text(encoding="utf-8")
+        )
+    except Exception as exc:
+        errors.append(f"Unable to read catalog: {exc}")
+        return []
+
+    if isinstance(data, list):
+        features = data
+    elif isinstance(data, dict) and isinstance(data.get("features"), list):
+        features = data["features"]
+    else:
+        errors.append(
+            "Catalog must be an array or contain a features array"
+        )
+        return []
+
+    if not all(isinstance(feature, dict) for feature in features):
+        errors.append("Every catalog feature must be an object")
+        return []
+
+    return features
+
+
+def validate_catalog(
+    features: list[dict[str, Any]],
+    errors: list[str],
+) -> None:
+    if len(features) != EXPECTED_FEATURES:
+        errors.append(
+            f"Expected {EXPECTED_FEATURES} features, "
+            f"found {len(features)}"
+        )
+
+    ids = [feature.get("id") for feature in features]
+
+    duplicates = sorted(
+        feature_id
+        for feature_id, count in Counter(ids).items()
+        if feature_id and count > 1
+    )
+
+    if duplicates:
+        errors.append(
+            "Duplicate feature IDs: " + ", ".join(duplicates)
+        )
+
+    for index, feature in enumerate(features):
+        feature_id = feature.get("id")
+        label = (
+            feature_id
+            if isinstance(feature_id, str)
+            else f"<index:{index}>"
+        )
+
+        missing = sorted(
+            field
+            for field in REQUIRED_FIELDS
+            if feature.get(field) in (None, "", [])
+        )
+
+        if missing:
+            errors.append(
+                f"{label}: missing fields: {', '.join(missing)}"
+            )
+
+        if (
+            not isinstance(feature_id, str)
+            or not FEATURE_ID_RE.fullmatch(feature_id)
+        ):
+            errors.append(f"{label}: invalid feature ID")
+
+        if feature.get("status") not in VALID_STATUSES:
+            errors.append(
+                f"{label}: invalid status "
+                f"{feature.get('status')!r}"
+            )
+
+        if feature.get("evidence_maturity") not in VALID_MATURITY:
+            errors.append(
+                f"{label}: invalid maturity "
+                f"{feature.get('evidence_maturity')!r}"
+            )
+
+        if feature.get("verified_at_commit") != EXPECTED_COMMIT:
+            errors.append(
+                f"{label}: incorrect verified_at_commit"
+            )
+
+        runtime = feature.get("runtime_validation")
+
+        if not isinstance(runtime, dict):
+            errors.append(
+                f"{label}: runtime_validation must be an object"
+            )
+            continue
+
+        required = runtime.get("required")
+        runtime_status = runtime.get("status")
+        reason = runtime.get("reason")
+
+        if not isinstance(required, bool):
+            errors.append(
+                f"{label}: runtime required must be boolean"
+            )
+
+        if runtime_status not in VALID_RUNTIME:
+            errors.append(
+                f"{label}: invalid runtime status "
+                f"{runtime_status!r}"
+            )
+
+        if not isinstance(reason, str) or not reason.strip():
+            errors.append(
+                f"{label}: runtime reason is blank"
+            )
+
+        if required is False and runtime_status != "not-required":
+            errors.append(
+                f"{label}: required=false requires not-required"
+            )
+
+        if required is True and runtime_status == "not-required":
+            errors.append(
+                f"{label}: required=true cannot be not-required"
+            )
+
+
+def parse_domain(
+    path: Path,
+    errors: list[str],
+) -> dict[str, dict[str, str]]:
+    records: dict[str, dict[str, str]] = {}
+    current_id: str | None = None
+
+    for line_number, line in enumerate(
+        path.read_text(encoding="utf-8").splitlines(),
+        start=1,
+    ):
+        heading = DOMAIN_HEADING_RE.match(line)
+
+        if heading:
+            current_id = heading.group(1)
+
+            if current_id in records:
+                errors.append(
+                    f"{path.relative_to(ROOT)}:{line_number}: "
+                    f"duplicate heading {current_id}"
+                )
+
+            records[current_id] = {
+                "Name": heading.group(2).strip(),
+            }
+            continue
+
+        field = DOMAIN_FIELD_RE.match(line)
+
+        if field and current_id:
+            value = clean(field.group(2))
+
+            if not value:
+                errors.append(
+                    f"{path.relative_to(ROOT)}:{line_number}: "
+                    f"blank {field.group(1)}"
+                )
+
+            records[current_id][field.group(1)] = value
+            continue
+
+        runtime = DOMAIN_RUNTIME_RE.match(line)
+
+        if runtime and current_id:
+            value = clean(runtime.group(1))
+            runtime_status = re.split(
+                r"\s+(?:—|-)\s+",
+                value,
+                maxsplit=1,
+            )[0]
+            records[current_id]["Runtime Validation"] = (
+                runtime_status.strip("` ")
+            )
+
+    return records
+
+
+def validate_domains(
+    features: list[dict[str, Any]],
+    errors: list[str],
+) -> None:
+    catalog = {
+        feature["id"]: feature
+        for feature in features
+        if feature.get("id")
+    }
+
+    expected_by_domain: defaultdict[str, set[str]] = defaultdict(set)
+
+    for feature in features:
+        expected_by_domain[feature["domain"]].add(feature["id"])
+
+    paths = sorted(DOMAINS_DIR.glob("*.md"))
+
+    if len(paths) != EXPECTED_DOMAINS:
+        errors.append(
+            f"Expected {EXPECTED_DOMAINS} domain files, "
+            f"found {len(paths)}"
+        )
+
+    all_found: set[str] = set()
+
+    for path in paths:
+        domain = path.stem
+        records = parse_domain(path, errors)
+        found = set(records)
+        expected = expected_by_domain.get(domain, set())
+        all_found.update(found)
+
+        if found != expected:
+            missing = sorted(expected - found)
+            extra = sorted(found - expected)
+
+            errors.append(
+                f"{domain}: missing={missing}, unexpected={extra}"
+            )
+
+        for feature_id, record in records.items():
+            feature = catalog.get(feature_id)
+
+            if feature is None:
+                continue
+
+            expected_values = {
+                "Name": feature["name"],
+                "Domain": feature["domain"],
+                "Status": feature["status"],
+                "Evidence Maturity": feature["evidence_maturity"],
+                "Commit Verified": feature["verified_at_commit"],
+                "Runtime Validation": (
+                    feature["runtime_validation"]["status"]
+                ),
+            }
+
+            for field, expected_value in expected_values.items():
+                actual = record.get(field)
+
+                if actual != expected_value:
+                    errors.append(
+                        f"{path.relative_to(ROOT)}: "
+                        f"{feature_id} {field}: "
+                        f"{actual!r} != {expected_value!r}"
+                    )
+
+    if all_found != set(catalog):
+        errors.append(
+            "Domain Markdown IDs do not match catalog JSON"
+        )
+
+
+def validate_catalog_markdown(
+    features: list[dict[str, Any]],
+    errors: list[str],
+) -> None:
+    expected = {
+        feature["id"]: feature
+        for feature in features
+    }
+    found: dict[str, list[str]] = {}
+
+    for line in CATALOG_MD_PATH.read_text(
+        encoding="utf-8"
+    ).splitlines():
+        if not line.startswith("|"):
+            continue
+
+        cells = [
+            cell.strip()
+            for cell in line.strip().strip("|").split("|")
+        ]
+
+        if len(cells) < 6:
+            continue
+
+        feature_id = clean(cells[0])
+
+        if FEATURE_ID_RE.fullmatch(feature_id):
+            found[feature_id] = cells
+
+    if set(found) != set(expected):
+        errors.append(
+            "feature-catalog.md IDs do not match JSON"
+        )
+
+    for feature_id, cells in found.items():
+        feature = expected[feature_id]
+        runtime = feature["runtime_validation"]
+        runtime_display = (
+            runtime["status"]
+            if runtime["required"]
+            else "not required"
+        )
+
+        actual = {
+            "name": cells[1].replace("\\|", "|"),
+            "domain": clean(cells[2]),
+            "status": clean(cells[3]),
+            "maturity": clean(cells[4]),
+            "runtime": clean(cells[5]),
+        }
+
+        wanted = {
+            "name": feature["name"],
+            "domain": feature["domain"],
+            "status": feature["status"],
+            "maturity": feature["evidence_maturity"],
+            "runtime": runtime_display,
+        }
+
+        for field, expected_value in wanted.items():
+            if actual[field] != expected_value:
+                errors.append(
+                    f"feature-catalog.md: {feature_id} "
+                    f"{field}: {actual[field]!r} "
+                    f"!= {expected_value!r}"
+                )
+
+
+def validate_reviews(
+    features: list[dict[str, Any]],
+    errors: list[str],
+) -> int:
+    catalog = {
+        feature["id"]: feature
+        for feature in features
+    }
+    checked = 0
+
+    for path in sorted(
+        REVIEWS_DIR.glob("evidence-sample-*.md")
+    ):
+        current_id: str | None = None
+        results: set[str] = set()
+
+        for line_number, line in enumerate(
+            path.read_text(encoding="utf-8").splitlines(),
+            start=1,
+        ):
+            heading = REVIEW_HEADING_RE.match(line)
+
+            if heading:
+                current_id = heading.group(1)
+                continue
+
+            result = REVIEW_RESULT_RE.match(line)
+
+            if not result or current_id is None:
+                continue
+
+            status, maturity = result.groups()
+            results.add(current_id)
+            checked += 1
+
+            feature = catalog.get(current_id)
+
+            if feature is None:
+                errors.append(
+                    f"{path.relative_to(ROOT)}:{line_number}: "
+                    f"unknown feature {current_id}"
+                )
+                continue
+
+            if feature["status"] != status:
+                errors.append(
+                    f"{current_id}: review status {status!r} "
+                    f"!= catalog {feature['status']!r}"
+                )
+
+            if feature["evidence_maturity"] != maturity:
+                errors.append(
+                    f"{current_id}: review maturity {maturity!r} "
+                    f"!= catalog "
+                    f"{feature['evidence_maturity']!r}"
+                )
+
+        if path.name == "evidence-sample-01.md" and len(results) != 12:
+            errors.append(
+                f"{path.relative_to(ROOT)}: expected 12 "
+                f"review results, found {len(results)}"
+            )
+
+    return checked
+
+
+def validate_whitespace(errors: list[str]) -> None:
+    for path in sorted(ROOT.rglob("*")):
+        if not path.is_file():
+            continue
+
+        if path.suffix not in {".md", ".json", ".py", ".txt"}:
+            continue
+
+        for line_number, line in enumerate(
+            path.read_text(
+                encoding="utf-8",
+                errors="replace",
+            ).splitlines(),
+            start=1,
+        ):
+            if line != line.rstrip(" \t"):
+                errors.append(
+                    f"{path.relative_to(ROOT)}:{line_number}: "
+                    "trailing whitespace"
+                )
+
+    if any(ROOT.rglob("*.pyc")):
+        errors.append("Generated .pyc files exist")
+
+    if any(
+        path.is_dir()
+        for path in ROOT.rglob("__pycache__")
+    ):
+        errors.append("__pycache__ exists")
+
+
+def main() -> int:
+    errors: list[str] = []
+    features = load_catalog(errors)
+    reviewed = 0
+
+    if features:
+        validate_catalog(features, errors)
+        validate_domains(features, errors)
+        validate_catalog_markdown(features, errors)
+        reviewed = validate_reviews(features, errors)
+
+    validate_whitespace(errors)
+
+    print(f"Catalog Features: {len(features)}")
+    print(
+        "Unique Feature IDs:",
+        len({feature.get("id") for feature in features}),
+    )
+    print(
+        "Domain Files:",
+        len(list(DOMAINS_DIR.glob("*.md"))),
+    )
+
+    if features:
+        print(
+            "Statuses:",
+            dict(
+                Counter(
+                    feature.get("status")
+                    for feature in features
+                )
+            ),
+        )
+        print(
+            "Evidence Maturity:",
+            dict(
+                Counter(
+                    feature.get("evidence_maturity")
+                    for feature in features
+                )
+            ),
+        )
+        print(
+            "Runtime Validation:",
+            dict(
+                Counter(
+                    (
+                        feature.get(
+                            "runtime_validation",
+                            {},
+                        ).get("required"),
+                        feature.get(
+                            "runtime_validation",
+                            {},
+                        ).get("status"),
+                    )
+                    for feature in features
+                )
+            ),
+        )
+
+    print(f"Review Results Checked: {reviewed}")
+    print(f"Consistency Errors: {len(errors)}")
+
+    for error in errors:
+        print(f"ERROR: {error}")
+
+    return 1 if errors else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

docs/discovery/tools/validate_discovery_docs.py

@@ -0,0 +1,172 @@
+#!/usr/bin/env python3
+"""Validate the Odysseus public discovery documentation package."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+REPO_ROOT = ROOT.parents[1]
+CATALOG_PATH = ROOT / "feature-catalog.json"
+DOMAINS_DIR = ROOT / "domains"
+
+EXPECTED_COMMIT = "d8a2059df8e53bc7275c45339849d14c8651e73c"
+ALLOWED_STATUSES = {
+    "verified", "partial", "disabled", "experimental", "legacy",
+    "dead-code-candidate", "unverified"
+}
+
+LINE_RANGE_RE = re.compile(r"^L([1-9]\d*)-L([1-9]\d*)$")
+FEATURE_HEADING_RE = re.compile(
+    r"^##\s+`?([A-Z][A-Z0-9]*-\d{3})`?\s+(?:—|-)\s+.+$",
+    re.MULTILINE,
+)
+LINK_RE = re.compile(r"(?<!!)\[[^\]]*\]\(([^)]+)\)")
+FORBIDDEN_TERMS_RE = re.compile(
+    r"(roadforge|kanban|matrix|owner link|editor link|github support|"
+    r"private maintainer|OD-AUD-|[A-Z]{2,10}-AUD-\d+|TASK-\d+)",
+    re.IGNORECASE,
+)
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="Validate Odysseus discovery docs.")
+    parser.add_argument(
+        "--repo-root",
+        type=Path,
+        default=REPO_ROOT,
+        help="Path to Odysseus repository root.",
+    )
+    return parser.parse_args()
+
+def validate() -> int:
+    args = parse_args()
+    repo_root = args.repo_root.resolve()
+    errors: list[str] = []
+
+    # 1. Validate Catalog JSON existence and content
+    if not CATALOG_PATH.is_file():
+        errors.append(f"Missing catalog file: {CATALOG_PATH}")
+        print(f"Errors: {len(errors)}")
+        for e in errors:
+            print(f"ERROR: {e}")
+        return 1
+
+    try:
+        catalog = json.loads(CATALOG_PATH.read_text(encoding="utf-8"))
+    except Exception as e:
+        errors.append(f"Failed to parse catalog JSON: {e}")
+        print(f"Errors: {len(errors)}")
+        for err in errors:
+            print(f"ERROR: {err}")
+        return 1
+
+    if not isinstance(catalog, list):
+        errors.append("feature-catalog.json must be a JSON array")
+        return 1
+
+    feature_ids = [item.get("id") for item in catalog if isinstance(item, dict)]
+    if len(feature_ids) != 79:
+        errors.append(f"Expected 79 unique feature IDs, found {len(feature_ids)}")
+    if len(feature_ids) != len(set(feature_ids)):
+        errors.append("Duplicate feature IDs found in catalog JSON")
+
+    # Domain catalog counts
+    catalog_domain_counts: dict[str, int] = {}
+    for item in catalog:
+        if not isinstance(item, dict):
+            errors.append("Catalog item is not an object")
+            continue
+        fid = item.get("id", "<missing>")
+        status = item.get("status")
+        domain = item.get("domain", "").lower()
+        catalog_domain_counts[domain] = catalog_domain_counts.get(domain, 0) + 1
+
+        if status not in ALLOWED_STATUSES:
+            errors.append(f"{fid}: Invalid status '{status}'")
+
+        evidence_list = item.get("evidence")
+        if not isinstance(evidence_list, list) or not evidence_list:
+            errors.append(f"{fid}: Missing or empty evidence list")
+            continue
+
+        for ev in evidence_list:
+            if not isinstance(ev, dict):
+                errors.append(f"{fid}: Evidence item is not an object")
+                continue
+            path_str = ev.get("path")
+            lr_str = str(ev.get("line_range", ""))
+            if not path_str or Path(path_str).is_absolute() or ".." in Path(path_str).parts:
+                errors.append(f"{fid}: Unsafe or invalid path '{path_str}'")
+                continue
+
+            # Check path exists in repo
+            target_path = repo_root / path_str
+            if not target_path.is_file():
+                errors.append(f"{fid}: Referenced path '{path_str}' does not exist on disk")
+                continue
+
+            # Check line range format & bounds
+            m = LINE_RANGE_RE.fullmatch(lr_str)
+            if not m:
+                errors.append(f"{fid}: Invalid line range format '{lr_str}' for path '{path_str}'")
+                continue
+
+            start, end = int(m.group(1)), int(m.group(2))
+            lines_cnt = len(target_path.read_text(encoding="utf-8", errors="ignore").splitlines())
+            if start > end or end > lines_cnt or start < 1:
+                errors.append(
+                    f"{fid}: Line range '{lr_str}' exceeds file length ({lines_cnt} lines) in '{path_str}'"
+                )
+
+    # 2. Check Domain Markdown files
+    md_feature_ids: list[str] = []
+    domain_files = sorted(DOMAINS_DIR.glob("*.md"))
+    for df in domain_files:
+        domain_name = df.stem.lower()
+        content = df.read_text(encoding="utf-8")
+        found_ids = FEATURE_HEADING_RE.findall(content)
+        md_feature_ids.extend(found_ids)
+        if len(found_ids) != catalog_domain_counts.get(domain_name, 0):
+            errors.append(
+                f"Domain '{domain_name}' count mismatch: catalog has {catalog_domain_counts.get(domain_name, 0)}, Markdown has {len(found_ids)}"
+            )
+
+    if sorted(md_feature_ids) != sorted(feature_ids):
+        errors.append("Markdown domain feature IDs do not match catalog JSON feature IDs")
+
+    # 3. Check for forbidden/private terms, sensitive credentials, and broken links across all docs
+    for md_file in ROOT.rglob("*.md"):
+        rel_md = md_file.relative_to(ROOT)
+        content = md_file.read_text(encoding="utf-8")
+
+        # Forbidden terms scan
+        forbidden_matches = FORBIDDEN_TERMS_RE.findall(content)
+        if forbidden_matches:
+            errors.append(
+                f"{rel_md}: Found forbidden/internal terms: {set(forbidden_matches)}"
+            )
+
+        # Broken local link check
+        for target in LINK_RE.findall(content):
+            target = target.strip().strip("<>")
+            if not target or target.startswith(("#", "http://", "https://", "mailto:")):
+                continue
+            target_path = target.split("#", 1)[0]
+            resolved = (md_file.parent / target_path).resolve()
+            if not resolved.exists():
+                errors.append(f"{rel_md}: Broken local link '{target}'")
+
+    print(f"Catalog Features: {len(catalog)}")
+    print(f"Domain Files: {len(domain_files)}")
+    print(f"Validation Errors: {len(errors)}")
+    for err in errors:
+        print(f"ERROR: {err}")
+
+    return 1 if errors else 0
+
+if __name__ == "__main__":
+    sys.exit(validate())

docs/index.html

@@ -409,7 +409,7 @@
         <a href="#testimonials">Testimonials</a>
         <a href="#how">How it started</a>
         <a href="#start">Get started</a>
-        <a class="btn" href="https://github.com/pewdiepie-archdaemon/odysseus" target="_blank">
+        <a class="btn" href="https://github.com/odysseus-dev/odysseus" target="_blank">
           <svg width="16" height="16" viewBox="0 0 24 24" fill="currentColor"><path d="M12 .5C5.7.5.5 5.7.5 12c0 5.1 3.3 9.4 7.9 10.9.6.1.8-.2.8-.6v-2c-3.2.7-3.9-1.5-3.9-1.5-.5-1.3-1.3-1.7-1.3-1.7-1-.7.1-.7.1-.7 1.2.1 1.8 1.2 1.8 1.2 1 1.8 2.7 1.3 3.4 1 .1-.8.4-1.3.7-1.6-2.6-.3-5.3-1.3-5.3-5.7 0-1.3.5-2.3 1.2-3.1-.1-.3-.5-1.5.1-3.1 0 0 1-.3 3.3 1.2a11.5 11.5 0 0 1 6 0C17.3 4.7 18.3 5 18.3 5c.6 1.6.2 2.8.1 3.1.8.8 1.2 1.8 1.2 3.1 0 4.4-2.7 5.4-5.3 5.7.4.4.8 1.1.8 2.2v3.3c0 .4.2.7.8.6 4.6-1.5 7.9-5.8 7.9-10.9C23.5 5.7 18.3.5 12 .5z"/></svg>
           GitHub
         </a>
@@ -437,7 +437,7 @@
       </p>
       <div class="hero-cta">
         <a class="btn primary" href="#start">Get started</a>
-        <a class="btn" href="https://github.com/pewdiepie-archdaemon/odysseus" target="_blank">View on GitHub</a>
+        <a class="btn" href="https://github.com/odysseus-dev/odysseus" target="_blank">View on GitHub</a>
       </div>
 
     </div>
@@ -678,9 +678,9 @@
         <div class="eyebrow"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M4 17l6-5-6-5"/><path d="M12 19h8"/></svg>Get started</div>
         <h2 class="h" style="margin-bottom:6px;">Odysseus is yours.</h2>
         <p class="sub center" style="margin:0 auto;">It's open source and free. No sales team, no demo request, no Trojan horse.</p>
-        <div class="codeblock"><span><span class="prompt">$</span> git clone https://github.com/pewdiepie-archdaemon/odysseus.git &amp;&amp; cd odysseus</span><button class="copy-btn" data-copy="git clone https://github.com/pewdiepie-archdaemon/odysseus.git && cd odysseus">Copy</button></div>
+        <div class="codeblock"><span><span class="prompt">$</span> git clone https://github.com/odysseus-dev/odysseus.git &amp;&amp; cd odysseus</span><button class="copy-btn" data-copy="git clone https://github.com/odysseus-dev/odysseus.git && cd odysseus">Copy</button></div>
         <div>
-          <a class="btn primary" href="https://github.com/pewdiepie-archdaemon/odysseus" target="_blank" style="margin-top:14px;">View on GitHub</a>
+          <a class="btn primary" href="https://github.com/odysseus-dev/odysseus" target="_blank" style="margin-top:14px;">View on GitHub</a>
         </div>
         <div class="pill-row">
           <span class="pill">Self-hosted</span>

docs/security-ci.md

@@ -7,10 +7,9 @@ benefit.
 
 ## What runs, and why
 
-Most checks live in files under `.github/workflows/`. CodeQL is configured
-through GitHub's code scanning default setup, so it appears as a dynamic GitHub
-workflow instead of a checked-in workflow file. They run automatically; you do
-not start them.
+Most checks live in files under `.github/workflows/`. CodeQL uses the
+checked-in advanced configuration in `.github/workflows/codeql.yml`. They run
+automatically; you do not start them.
 
 | Check | What it protects against | Blocks a merge? |
 |---|---|---|
@@ -90,14 +89,9 @@ let the workflows run on one pull request first, then add them here.
 2. Turn on **Dependency graph** (usually on by default for public repos) -- this
    powers Dependency review and Dependabot.
 3. Turn on **Dependabot alerts** and **Dependabot security updates**.
-4. Under **Code scanning**, use **Set up -> Default** for CodeQL. GitHub then
-   runs CodeQL as a dynamic workflow without the fork-token limitations that
-   affect checked-in advanced workflows.
-
-   Do not also add a checked-in CodeQL workflow while default setup is enabled:
-   GitHub rejects advanced CodeQL uploads when default setup is active. If the
-   project later needs an advanced CodeQL workflow, disable default setup first
-   and keep only one CodeQL publishing path active.
+4. Under **Code scanning**, keep **Default setup** disabled. CodeQL is
+   configured by `.github/workflows/codeql.yml`; enabling default setup at the
+   same time causes GitHub to reject uploads from the checked-in workflow.
 
 ## Keeping it current
 

docs/setup.md

@@ -4,7 +4,7 @@ This page keeps the detailed install, deployment, troubleshooting, and configura
 
 ## Quick Start
 
-> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main).
+> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/odysseus-dev/odysseus/tree/main).
 
 Defaults work out of the box: clone, run, then configure models/search/email
 inside **Settings**. Only edit `.env` for deployment-level overrides like
@@ -20,7 +20,7 @@ pull request guidelines.
 
 ### Docker (recommended)
 ```bash
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 cp .env.example .env       # optional, but recommended for explicit defaults
 docker compose up -d --build
@@ -38,7 +38,7 @@ only when you intentionally want LAN/reverse-proxy access.
 
 ### Native Linux / macOS
 ```bash
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 python3 -m venv venv
 source venv/bin/activate
@@ -56,7 +56,7 @@ Docker on macOS cannot use the Metal GPU. For GPU-accelerated Cookbook on an
 M-series Mac, run Odysseus natively:
 
 ```bash
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 ./start-macos.sh
 ```
@@ -150,6 +150,79 @@ scripts/check-docker-gpu.sh --enable-nvidia-overlay
 # Full assisted setup — install toolkit, then enable overlay if passthrough works:
 scripts/check-docker-gpu.sh --install-nvidia-toolkit --enable-nvidia-overlay
 ```
+#### Arch Linux NVIDIA Docker notes
+
+On Arch Linux, verify the host NVIDIA driver and Docker GPU passthrough before enabling the Odysseus NVIDIA overlay.
+
+Install the required packages:
+
+```bash
+sudo pacman -Syu
+sudo pacman -S docker docker-compose nvidia-container-toolkit nvidia-utils
+sudo systemctl enable --now docker
+```
+
+Configure Docker to use the NVIDIA container runtime:
+
+```bash
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+```
+
+Verify the host GPU:
+
+```bash
+nvidia-smi
+```
+
+Verify Docker GPU passthrough:
+
+```bash
+docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi
+```
+
+Then enable the Odysseus NVIDIA compose overlay:
+
+```env
+COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
+```
+
+Rebuild and verify the GPU inside the Odysseus container:
+
+```bash
+docker compose up -d --build
+docker compose exec odysseus nvidia-smi -L
+```
+
+For first-time local model testing on 8 GB laptop GPUs, start with GGUF/Q4 models on llama.cpp before trying GPTQ/AWQ models on vLLM or SGLang. This keeps the first run simpler while confirming GPU passthrough works.
+
+**WSL2 + snap Docker.** If the NVIDIA check fails with this error, Docker may be
+installed via snap:
+
+```text
+failed to fulfil mount request: open /usr/lib/wsl/lib/libdxcore.so: no such file or directory
+```
+
+Check with `snap list docker` or:
+
+```bash
+docker info --format '{{.DockerRootDir}}'
+```
+
+A Docker root under `/var/snap/docker/` means snap confinement can prevent
+Docker from seeing WSL2's `/usr/lib/wsl/lib` GPU libraries even when the files
+exist on the host. Reinstalling or reconfiguring `nvidia-container-toolkit` will
+not fix that. Remove snap Docker, install the official apt-based Docker Engine
+([Docker docs](https://docs.docker.com/engine/install/ubuntu/)), then configure
+the NVIDIA runtime again:
+
+```bash
+sudo snap remove docker
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+```
+
+Then re-run `scripts/check-docker-gpu.sh`.
 
 Safety notes:
 - The app never installs host GPU runtime automatically.
@@ -257,7 +330,7 @@ do not run on macOS. MLX-only models are not served by Odysseus.
 server; safe to re-run):
 
 ```powershell
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
 ```
@@ -265,7 +338,7 @@ powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
 Or do it by hand:
 
 ```powershell
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 py -3.11 -m venv venv
 venv\Scripts\Activate.ps1

mcp_servers/email_server.py

@@ -24,6 +24,7 @@ from pathlib import Path
 from datetime import datetime, timedelta
 import uuid
 from contextvars import ContextVar
+from urllib.parse import parse_qs, unquote, urlparse
 
 from mcp.server import Server
 from mcp.server.stdio import stdio_server
@@ -129,20 +130,36 @@ def _mcp_owner_required(rows: list[dict] | None = None) -> bool:
     return _has_owner_scoped_accounts(rows)
 
 
-def _load_email_writing_style() -> str:
-    """Return the existing Settings > Email > Writing Style value."""
+def _load_email_writing_style(account: str | None = None) -> str:
+    """Return the saved Settings > Email > Writing Style value.
+
+    Prefer the selected account's style when one exists; fall back to the
+    legacy global style so older installs keep behaving as before.
+    """
     try:
         settings_path = DATA_DIR / "settings.json"
         if not settings_path.exists():
             return ""
         settings = json.loads(settings_path.read_text(encoding="utf-8"))
+        account_id = ""
+        if account:
+            try:
+                cfg = _load_config(account)
+                account_id = str(cfg.get("account_id") or account or "").strip()
+            except Exception:
+                account_id = str(account or "").strip()
+        by_account = settings.get("email_writing_styles_by_account") or {}
+        if account_id and isinstance(by_account, dict):
+            style = by_account.get(account_id)
+            if isinstance(style, str) and style.strip():
+                return style.strip()
         return str(settings.get("email_writing_style") or "").strip()
     except Exception:
         return ""
 
 
-def _writing_style_guidance() -> str:
-    style = _load_email_writing_style()
+def _writing_style_guidance(account: str | None = None) -> str:
+    style = _load_email_writing_style(account)
     if not style:
         return (
             "No saved writing style is configured in Settings > Email > Writing Style. "
@@ -509,6 +526,251 @@ def _decode_header(raw):
         return "".join(decoded)
 
 
+def _uid_from_fetch_meta(meta_b: bytes) -> str:
+    m = re.search(rb"UID\s+(\d+)", meta_b or b"")
+    return m.group(1).decode("ascii", errors="ignore") if m else ""
+
+
+def _parse_list_unsubscribe_header(value: str | None) -> list[dict]:
+    raw = str(value or "").strip()
+    if not raw:
+        return []
+    pieces = re.findall(r"<([^>]+)>", raw)
+    if not pieces:
+        pieces = [p.strip() for p in raw.split(",") if p.strip()]
+    out: list[dict] = []
+    seen = set()
+    for piece in pieces:
+        target = piece.strip().strip("<>").strip()
+        if not target:
+            continue
+        parsed = urlparse(target)
+        scheme = parsed.scheme.lower()
+        key = target.lower()
+        if key in seen:
+            continue
+        seen.add(key)
+        if scheme == "mailto":
+            addr = unquote(parsed.path or "").strip()
+            if not addr or "\r" in addr or "\n" in addr:
+                continue
+            query = parse_qs(parsed.query or "", keep_blank_values=True)
+            subject = unquote((query.get("subject") or ["unsubscribe"])[0] or "unsubscribe")
+            body = unquote((query.get("body") or ["unsubscribe"])[0] or "unsubscribe")
+            subject = re.sub(r"[\r\n]+", " ", subject).strip() or "unsubscribe"
+            body = re.sub(r"[\r\n]+", "\n", body).strip() or "unsubscribe"
+            out.append({
+                "kind": "mailto",
+                "target": addr,
+                "subject": subject[:200],
+                "body": body[:1000],
+                "executable": True,
+            })
+        elif scheme in {"http", "https"}:
+            out.append({
+                "kind": "url",
+                "target": target,
+                "executable": False,
+            })
+    return out
+
+
+def _email_unsubscribe_candidate_from_msg(msg, uid: str, folder: str) -> dict | None:
+    sender = _decode_header(msg.get("From", ""))
+    sender_name, sender_addr = email.utils.parseaddr(sender)
+    subject = _decode_header(msg.get("Subject", "(no subject)"))
+    list_id = _decode_header(msg.get("List-Id", ""))
+    precedence = (msg.get("Precedence") or "").strip().lower()
+    auto_submitted = (msg.get("Auto-Submitted") or "").strip().lower()
+    methods = _parse_list_unsubscribe_header(msg.get("List-Unsubscribe"))
+    if not methods:
+        return None
+    reasons: list[str] = ["has unsubscribe header"]
+    score = 45
+    if list_id:
+        score += 20
+        reasons.append("mailing-list header")
+    if precedence in {"bulk", "junk", "list"}:
+        score += 20
+        reasons.append(f"precedence={precedence}")
+    if auto_submitted and auto_submitted != "no":
+        score += 10
+        reasons.append(f"auto-submitted={auto_submitted}")
+    if re.search(r"\b(unsubscribe|newsletter|sale|discount|offer|promo|limited time)\b", (subject or "").lower()):
+        score += 10
+        reasons.append("promotional subject")
+    executable = [m for m in methods if m.get("executable")]
+    return {
+        "uid": str(uid),
+        "folder": folder,
+        "message_id": (msg.get("Message-ID") or "").strip(),
+        "subject": subject,
+        "from_name": sender_name or sender_addr,
+        "from_address": sender_addr,
+        "list_id": list_id,
+        "score": min(score, 100),
+        "reasons": reasons[:5],
+        "methods": methods,
+        "can_execute": bool(executable),
+        "recommended_method": executable[0] if executable else methods[0],
+    }
+
+
+def _unsubscribe_candidate_dedupe_key(candidate: dict) -> tuple[str, str, str]:
+    list_id = str(candidate.get("list_id") or "").strip().lower()
+    method = candidate.get("recommended_method") or {}
+    method_kind = str(method.get("kind") or "").strip().lower()
+    method_target = str(method.get("target") or "").strip().lower()
+    sender = str(candidate.get("from_address") or "").strip().lower()
+    if list_id:
+        return ("list", list_id, method_target or sender)
+    if method_target:
+        return ("method", method_kind, method_target)
+    return ("sender", sender, str(candidate.get("subject") or "").strip().lower())
+
+
+def _dedupe_unsubscribe_candidates(candidates: list[dict]) -> list[dict]:
+    deduped: dict[tuple[str, str, str], dict] = {}
+    for candidate in candidates or []:
+        key = _unsubscribe_candidate_dedupe_key(candidate)
+        existing = deduped.get(key)
+        if not existing:
+            copy = dict(candidate)
+            copy["duplicate_count"] = 1
+            copy["duplicate_uids"] = [str(candidate.get("uid") or "")]
+            deduped[key] = copy
+            continue
+        existing["duplicate_count"] = int(existing.get("duplicate_count") or 1) + 1
+        uid = str(candidate.get("uid") or "")
+        if uid:
+            existing.setdefault("duplicate_uids", []).append(uid)
+        if int(candidate.get("score") or 0) > int(existing.get("score") or 0):
+            keep_count = existing.get("duplicate_count")
+            keep_uids = existing.get("duplicate_uids")
+            replacement = dict(candidate)
+            replacement["duplicate_count"] = keep_count
+            replacement["duplicate_uids"] = keep_uids
+            deduped[key] = replacement
+    return list(deduped.values())
+
+
+def _scan_unsubscribe_candidates(folder="INBOX", account=None, limit=25, max_scan=150) -> dict:
+    limit = max(1, min(int(limit or 25), 100))
+    max_scan = max(limit, min(int(max_scan or 150), 500))
+    folder = folder or "INBOX"
+    candidates: list[dict] = []
+    conn = _imap_connect(account)
+    try:
+        status, _ = conn.select(_q(folder), readonly=True)
+        if status != "OK":
+            return {"success": False, "error": f"Folder not found: {folder}", "candidates": []}
+        status, data = conn.uid("SEARCH", None, "ALL")
+        if status != "OK" or not data or not data[0]:
+            return {"success": True, "candidates": [], "total": 0, "scanned": 0, "folder": folder}
+        uids = []
+        for raw_uid in data[0].split():
+            try:
+                uids.append(int(raw_uid))
+            except Exception:
+                continue
+        uids = sorted(uids, reverse=True)[:max_scan]
+        if not uids:
+            return {"success": True, "candidates": [], "total": 0, "scanned": 0, "folder": folder}
+        status, msg_data = conn.uid("FETCH", _b(",".join(str(u) for u in uids)), "(UID RFC822.HEADER)")
+    finally:
+        try:
+            conn.logout()
+        except Exception:
+            pass
+    if status != "OK":
+        return {"success": False, "error": "Failed to fetch email headers", "candidates": []}
+    for item in msg_data or []:
+        if not isinstance(item, tuple) or len(item) < 2:
+            continue
+        meta_b = item[0] if isinstance(item[0], bytes) else str(item[0]).encode()
+        uid = _uid_from_fetch_meta(meta_b)
+        if not uid:
+            continue
+        try:
+            msg = email.message_from_bytes(item[1] or b"")
+        except Exception:
+            continue
+        candidate = _email_unsubscribe_candidate_from_msg(msg, uid, folder)
+        if candidate:
+            candidates.append(candidate)
+    raw_total = len(candidates)
+    candidates = _dedupe_unsubscribe_candidates(candidates)
+    candidates.sort(key=lambda c: (int(c.get("score") or 0), int(c.get("duplicate_count") or 1), int(c.get("uid") or 0)), reverse=True)
+    return {
+        "success": True,
+        "candidates": candidates[:limit],
+        "total": len(candidates),
+        "raw_total": raw_total,
+        "scanned": len(uids),
+        "folder": folder,
+        "account": account or "",
+    }
+
+
+def _unsubscribe_email(uid, folder="INBOX", account=None, method_index=0, allow_web=False) -> dict:
+    uid = str(uid or "").strip()
+    if not uid:
+        return {"success": False, "error": "uid is required"}
+    conn = _imap_connect(account)
+    try:
+        status, _ = conn.select(_q(folder), readonly=True)
+        if status != "OK":
+            return {"success": False, "error": f"Folder not found: {folder}"}
+        status, msg_data = conn.uid("FETCH", _b(uid), "(UID RFC822.HEADER)")
+    finally:
+        try:
+            conn.logout()
+        except Exception:
+            pass
+    if status != "OK" or not msg_data:
+        return {"success": False, "error": f"Email not found: {uid}"}
+    raw_header = b""
+    for item in msg_data or []:
+        if isinstance(item, tuple) and len(item) >= 2:
+            raw_header = item[1] or b""
+            break
+    msg = email.message_from_bytes(raw_header)
+    candidate = _email_unsubscribe_candidate_from_msg(msg, uid, folder)
+    if not candidate:
+        return {"success": False, "error": "No List-Unsubscribe header found"}
+    methods = candidate.get("methods") or []
+    method_index = int(method_index or 0)
+    method = methods[method_index] if 0 <= method_index < len(methods) else (candidate.get("recommended_method") or methods[0])
+    if method.get("kind") == "url":
+        return {
+            "success": False,
+            "requires_browser": True,
+            "url": method.get("target"),
+            "candidate": candidate,
+            "instructions": (
+                "This unsubscribe is a web link. Ask the user for approval, then use the browser/web tool "
+                "to open the exact URL and complete the unsubscribe page. Do not fetch unrelated links."
+            ),
+        }
+    if method.get("kind") != "mailto" or not method.get("executable"):
+        return {"success": False, "error": "Unsupported unsubscribe method", "candidate": candidate}
+    result = _send_email(
+        to=method.get("target"),
+        subject=method.get("subject") or "unsubscribe",
+        body=method.get("body") or "unsubscribe",
+        account=account,
+    )
+    if "error" in result:
+        return {"success": False, "error": result["error"], "candidate": candidate}
+    return {
+        "success": True,
+        "method": method,
+        "candidate": candidate,
+        "send_result": result,
+        "pending": bool(result.get("pending")),
+    }
+
+
 def _extract_text(msg):
     """Extract plain text body from email message."""
     if msg.is_multipart():
@@ -1546,8 +1808,7 @@ async def _ai_draft_reply_to_email(uid, folder="INBOX", reply_all=False, account
     except Exception as exc:
         return {"error": f"AI reply helpers unavailable: {exc}"}
 
-    settings = _load_settings()
-    style = settings.get("email_writing_style", "")
+    style = _load_email_writing_style(account)
     system_prompt = _EMAIL_REPLY_SYS_PROMPT_BASE
     if style:
         system_prompt += f"\n\nWRITING STYLE TO MATCH:\n{style}"
@@ -1890,6 +2151,45 @@ async def list_tools() -> list[Tool]:
                 "required": [],
             },
         ),
+        Tool(
+            name="scan_email_unsubscribes",
+            description=(
+                "Scan recent email headers for likely spam/newsletter unsubscribe candidates. "
+                "Returns reviewable candidates with UID, sender, subject, score, reasons, and "
+                "List-Unsubscribe methods. This does not unsubscribe anything. For mailto "
+                "methods, use unsubscribe_email after user approval. For web URL methods, use "
+                "browser/web tools after user approval to open the exact URL and complete the page."
+            ),
+            inputSchema={
+                "type": "object",
+                "properties": {
+                    "folder": {"type": "string", "description": "IMAP folder to scan", "default": "INBOX"},
+                    "limit": {"type": "integer", "description": "Maximum candidates to return", "default": 25},
+                    "max_scan": {"type": "integer", "description": "How many newest messages to inspect", "default": 150},
+                    **ACCOUNT_PROP,
+                },
+                "required": [],
+            },
+        ),
+        Tool(
+            name="unsubscribe_email",
+            description=(
+                "Execute one approved unsubscribe action for an email UID. Supports safe mailto "
+                "List-Unsubscribe directly. If the selected method is a web URL, this returns "
+                "requires_browser with the exact URL; use browser/web tools only after user approval."
+            ),
+            inputSchema={
+                "type": "object",
+                "properties": {
+                    "uid": {"type": "string", "description": "Email UID from scan_email_unsubscribes/list_emails"},
+                    "folder": {"type": "string", "description": "IMAP folder", "default": "INBOX"},
+                    "method_index": {"type": "integer", "description": "Unsubscribe method index from scan_email_unsubscribes", "default": 0},
+                    "allow_web": {"type": "boolean", "description": "Return web unsubscribe URL instructions when the method is URL", "default": False},
+                    **ACCOUNT_PROP,
+                },
+                "required": ["uid"],
+            },
+        ),
         Tool(
             name="download_attachment",
             description=(
@@ -2264,6 +2564,69 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
                 lines.append(line)
             return [TextContent(type="text", text="\n\n".join(lines))]
 
+        elif name == "scan_email_unsubscribes":
+            try:
+                result = _scan_unsubscribe_candidates(
+                    folder=arguments.get("folder", "INBOX"),
+                    account=acct,
+                    limit=arguments.get("limit", 25),
+                    max_scan=arguments.get("max_scan", 150),
+                )
+            except Exception as e:
+                return [TextContent(type="text", text=f"Unsubscribe scan failed: {e}")]
+            if not result.get("success"):
+                return [TextContent(type="text", text=f"Unsubscribe scan failed: {result.get('error', 'unknown error')}")]
+            candidates = result.get("candidates") or []
+            if not candidates:
+                return [TextContent(type="text", text=f"No unsubscribe candidates found in {result.get('scanned', 0)} recent emails.")]
+            lines = [
+                f"Found {len(candidates)} unsubscribe candidate(s) from {result.get('scanned', 0)} recent emails.",
+                "Review these with the user before executing. Mailto methods can use unsubscribe_email; URL methods require browser/web tools after approval.\n",
+            ]
+            for i, cand in enumerate(candidates, 1):
+                lines.append(
+                    f"{i}. **{cand.get('subject') or '(no subject)'}**\n"
+                    f"   From: {cand.get('from_name') or cand.get('from_address') or ''} ({cand.get('from_address') or ''})\n"
+                    f"   UID: {cand.get('uid')}  Folder: {cand.get('folder')}\n"
+                    f"   Score: {cand.get('score')}  Matching emails: {cand.get('duplicate_count', 1)}  Reasons: {', '.join(cand.get('reasons') or [])}"
+                )
+                for j, method in enumerate(cand.get("methods") or []):
+                    if method.get("kind") == "mailto":
+                        lines.append(f"   Method {j}: mailto {method.get('target')} (executable via unsubscribe_email)")
+                    elif method.get("kind") == "url":
+                        lines.append(f"   Method {j}: web URL {method.get('target')} (use browser/web tools after approval)")
+            return [TextContent(type="text", text="\n".join(lines))]
+
+        elif name == "unsubscribe_email":
+            result = _unsubscribe_email(
+                uid=arguments.get("uid"),
+                folder=arguments.get("folder", "INBOX"),
+                account=acct,
+                method_index=arguments.get("method_index", 0),
+                allow_web=bool(arguments.get("allow_web", False)),
+            )
+            if result.get("requires_browser"):
+                return [TextContent(
+                    type="text",
+                    text=(
+                        "Web unsubscribe requires browser/web navigation.\n"
+                        f"URL: {result.get('url')}\n"
+                        f"{result.get('instructions')}"
+                    ),
+                )]
+            if not result.get("success"):
+                return [TextContent(type="text", text=f"Unsubscribe failed: {result.get('error', 'unknown error')}")]
+            method = result.get("method") or {}
+            if result.get("pending"):
+                return [TextContent(
+                    type="text",
+                    text=(
+                        f"Unsubscribe email staged for approval to {method.get('target')}. "
+                        "Nothing has been sent until the user approves the pending email."
+                    ),
+                )]
+            return [TextContent(type="text", text=f"Unsubscribe email sent to {method.get('target')}.")]
+
         elif name == "download_attachment":
             uid = arguments.get("uid")
             index = arguments.get("index")

mcp_servers/image_gen_server.py

@@ -81,7 +81,13 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
             if not model_spec:
                 return [TextContent(type="text", text="Error: No image model found. Configure one in Admin.")]
 
-        url, model_id, headers = await asyncio.to_thread(_resolve_model, model_spec)
+        try:
+            url, model_id, headers = await asyncio.to_thread(_resolve_model, model_spec, model_type="image")
+        except ValueError:
+            _lower_model_spec = model_spec.lower()
+            if not any(_name in _lower_model_spec for _name in ("gpt-image", "dall-e")):
+                raise
+            url, model_id, headers = await asyncio.to_thread(_resolve_model, model_spec)
 
         is_gpt_image = "gpt-image" in model_id.lower()
         base_url = url.replace("/chat/completions", "").replace("/v1/messages", "").rstrip("/")

package.json

@@ -1,7 +1,7 @@
 {
   "repository": {
     "type": "git",
-    "url": "https://github.com/pewdiepie-archdaemon/odysseus.git"
+    "url": "https://github.com/odysseus-dev/odysseus.git"
   },
   "devDependencies": {
     "@antithesishq/bombadil": "^0.6.1"

routes/admin_wipe/__init__.py

@@ -0,0 +1,5 @@
+"""Admin wipe route domain package (slice 2h, #4082/#4071).
+
+Contains admin_wipe_routes.py, migrated from the flat routes/ directory.
+Backward-compat shim at routes/admin_wipe_routes.py re-exports from here.
+"""

routes/admin_wipe/admin_wipe_routes.py

@@ -0,0 +1,176 @@
+"""Admin Danger Zone — per-category wipes.
+
+Each endpoint is admin-only and truncates exactly one domain so the
+user can selectively reset memory / skills / notes / etc. without
+nuking everything. The catch-all `chats` endpoint mirrors the
+existing /api/sessions/all so the Danger Zone speaks one URL pattern.
+
+URL shape: DELETE /api/admin/wipe/{kind}
+Kinds: chats, memory, skills, notes, tasks, documents, gallery, calendar.
+"""
+
+import json
+import logging
+import os
+import shutil
+from fastapi import APIRouter, HTTPException, Request
+
+from core.middleware import require_admin
+from core.database import (
+    SessionLocal,
+    Session as DbSession,
+    ChatMessage as DbChatMessage,
+    Memory,
+    Note,
+    ScheduledTask,
+    TaskRun,
+    Document,
+    DocumentVersion,
+    GalleryImage,
+    GalleryAlbum,
+    CalendarEvent,
+    CalendarCal,
+)
+from src.constants import DATA_DIR, SKILLS_DIR, SKILLS_FILE, GALLERY_DIR, GALLERY_UPLOADS_DIR
+
+logger = logging.getLogger(__name__)
+
+
+def _wipe_memory_files():
+    """Blank memory.json + drop the per-owner tidy-state sidecar so the
+    next audit doesn't try to diff against gone memories."""
+    for name in ("memory.json", "memory_tidy_state.json"):
+        p = os.path.join(DATA_DIR, name)
+        if not os.path.exists(p):
+            continue
+        try:
+            if name == "memory.json":
+                with open(p, "w", encoding="utf-8") as f:
+                    json.dump([], f)
+            else:
+                os.remove(p)
+        except OSError as e:
+            logger.warning(f"Could not reset {name}: {e}")
+
+
+def _rmtree_quiet(path: str):
+    """rmtree that doesn't crash if the path doesn't exist."""
+    if os.path.isdir(path):
+        try:
+            shutil.rmtree(path)
+        except OSError as e:
+            logger.warning(f"Could not remove {path}: {e}")
+
+
+def setup_admin_wipe_routes(session_manager):
+    """The session_manager is passed in so we can also clear its
+    in-memory cache when wiping chats — without it the DB is empty
+    but the next /api/sessions returns stale entries."""
+    router = APIRouter(prefix="/api/admin")
+
+    @router.delete("/wipe/{kind}")
+    def wipe(kind: str, request: Request):
+        require_admin(request)
+        kind = (kind or "").strip().lower()
+
+        db = SessionLocal()
+        try:
+            if kind == "chats":
+                count = db.query(DbSession).count()
+                db.query(DbChatMessage).delete()
+                db.query(DbSession).delete()
+                db.commit()
+                try:
+                    session_manager.sessions.clear()
+                except Exception:
+                    pass
+                return {"status": "deleted", "kind": kind, "count": count}
+
+            if kind == "memory":
+                count = db.query(Memory).count()
+                db.query(Memory).delete()
+                db.commit()
+                _wipe_memory_files()
+                # Drop the vector store too so semantic search doesn't
+                # return ghosts. Lazy import — chromadb may not be
+                # initialised in every deployment.
+                try:
+                    from src.memory_vector import get_memory_vector_store
+                    mv = get_memory_vector_store()
+                    if mv and hasattr(mv, "clear"):
+                        mv.clear()
+                except Exception as e:
+                    logger.info(f"Memory vector clear skipped: {e}")
+                return {"status": "deleted", "kind": kind, "count": count}
+
+            if kind == "skills":
+                # Skills live as SKILL.md files under data/skills/. Drop
+                # the entire directory; the SkillsManager re-creates the
+                # tree on next write.
+                skills_dir = SKILLS_DIR
+                count = 0
+                if os.path.isdir(skills_dir):
+                    # Count SKILL.md files for the response — quick walk.
+                    for _, _, files in os.walk(skills_dir):
+                        count += sum(1 for f in files if f == "SKILL.md")
+                    _rmtree_quiet(skills_dir)
+                # Legacy fallback file
+                legacy = SKILLS_FILE
+                if os.path.exists(legacy):
+                    try:
+                        os.remove(legacy)
+                    except OSError:
+                        pass
+                return {"status": "deleted", "kind": kind, "count": count}
+
+            if kind == "notes":
+                count = db.query(Note).count()
+                db.query(Note).delete()
+                db.commit()
+                return {"status": "deleted", "kind": kind, "count": count}
+
+            if kind == "tasks":
+                # TaskRun rows reference tasks via FK — clear them first.
+                db.query(TaskRun).delete()
+                count = db.query(ScheduledTask).count()
+                db.query(ScheduledTask).delete()
+                db.commit()
+                return {"status": "deleted", "kind": kind, "count": count}
+
+            if kind == "documents":
+                # DocumentVersion FKs Document — clear children first.
+                db.query(DocumentVersion).delete()
+                count = db.query(Document).count()
+                db.query(Document).delete()
+                db.commit()
+                return {"status": "deleted", "kind": kind, "count": count}
+
+            if kind == "gallery":
+                count = db.query(GalleryImage).count() + db.query(GalleryAlbum).count()
+                db.query(GalleryImage).delete()
+                db.query(GalleryAlbum).delete()
+                db.commit()
+                # Also drop the upload dir so disk doesn't keep orphans.
+                _rmtree_quiet(GALLERY_DIR)
+                _rmtree_quiet(GALLERY_UPLOADS_DIR)
+                return {"status": "deleted", "kind": kind, "count": count}
+
+            if kind == "calendar":
+                # Events FK calendars — clear children first, then both.
+                db.query(CalendarEvent).delete()
+                count = db.query(CalendarCal).count()
+                db.query(CalendarCal).delete()
+                db.commit()
+                return {"status": "deleted", "kind": kind, "count": count}
+
+            raise HTTPException(400, f"Unknown wipe kind: {kind!r}")
+        except HTTPException:
+            raise
+        except Exception as e:
+            db.rollback()
+            logger.exception(f"Wipe {kind} failed")
+            raise HTTPException(500, f"Wipe {kind} failed: {e}")
+        finally:
+            db.close()
+
+    return router

routes/admin_wipe_routes.py

@@ -1,176 +1,17 @@
-"""Admin Danger Zone — per-category wipes.
+"""Backward-compat shim — canonical location is routes/admin_wipe/admin_wipe_routes.py.
 
-Each endpoint is admin-only and truncates exactly one domain so the
-user can selectively reset memory / skills / notes / etc. without
-nuking everything. The catch-all `chats` endpoint mirrors the
-existing /api/sessions/all so the Danger Zone speaks one URL pattern.
-
-URL shape: DELETE /api/admin/wipe/{kind}
-Kinds: chats, memory, skills, notes, tasks, documents, gallery, calendar.
+This module is replaced in ``sys.modules`` by the canonical module object so
+that ``import routes.admin_wipe_routes``, ``from routes.admin_wipe_routes
+import X``, ``importlib.import_module("routes.admin_wipe_routes")``, and the
+``import ... as admin_wipe_routes`` + ``monkeypatch.setattr(admin_wipe_routes,
+"SessionLocal", ...)`` / ``"require_admin"`` pattern used by
+test_admin_wipe_gallery.py all operate on the *same* object the application
+actually uses. Keeps existing import paths working after slice 2h
+(#4082/#4071).
 """
 
-import json
-import logging
-import os
-import shutil
-from fastapi import APIRouter, HTTPException, Request
+import sys as _sys
 
-from core.middleware import require_admin
-from core.database import (
-    SessionLocal,
-    Session as DbSession,
-    ChatMessage as DbChatMessage,
-    Memory,
-    Note,
-    ScheduledTask,
-    TaskRun,
-    Document,
-    DocumentVersion,
-    GalleryImage,
-    GalleryAlbum,
-    CalendarEvent,
-    CalendarCal,
-)
-from src.constants import DATA_DIR, SKILLS_DIR, SKILLS_FILE, GALLERY_DIR, GALLERY_UPLOADS_DIR
+from routes.admin_wipe import admin_wipe_routes as _canonical  # noqa: F401
 
-logger = logging.getLogger(__name__)
-
-
-def _wipe_memory_files():
-    """Blank memory.json + drop the per-owner tidy-state sidecar so the
-    next audit doesn't try to diff against gone memories."""
-    for name in ("memory.json", "memory_tidy_state.json"):
-        p = os.path.join(DATA_DIR, name)
-        if not os.path.exists(p):
-            continue
-        try:
-            if name == "memory.json":
-                with open(p, "w", encoding="utf-8") as f:
-                    json.dump([], f)
-            else:
-                os.remove(p)
-        except OSError as e:
-            logger.warning(f"Could not reset {name}: {e}")
-
-
-def _rmtree_quiet(path: str):
-    """rmtree that doesn't crash if the path doesn't exist."""
-    if os.path.isdir(path):
-        try:
-            shutil.rmtree(path)
-        except OSError as e:
-            logger.warning(f"Could not remove {path}: {e}")
-
-
-def setup_admin_wipe_routes(session_manager):
-    """The session_manager is passed in so we can also clear its
-    in-memory cache when wiping chats — without it the DB is empty
-    but the next /api/sessions returns stale entries."""
-    router = APIRouter(prefix="/api/admin")
-
-    @router.delete("/wipe/{kind}")
-    def wipe(kind: str, request: Request):
-        require_admin(request)
-        kind = (kind or "").strip().lower()
-
-        db = SessionLocal()
-        try:
-            if kind == "chats":
-                count = db.query(DbSession).count()
-                db.query(DbChatMessage).delete()
-                db.query(DbSession).delete()
-                db.commit()
-                try:
-                    session_manager.sessions.clear()
-                except Exception:
-                    pass
-                return {"status": "deleted", "kind": kind, "count": count}
-
-            if kind == "memory":
-                count = db.query(Memory).count()
-                db.query(Memory).delete()
-                db.commit()
-                _wipe_memory_files()
-                # Drop the vector store too so semantic search doesn't
-                # return ghosts. Lazy import — chromadb may not be
-                # initialised in every deployment.
-                try:
-                    from src.memory_vector import get_memory_vector_store
-                    mv = get_memory_vector_store()
-                    if mv and hasattr(mv, "clear"):
-                        mv.clear()
-                except Exception as e:
-                    logger.info(f"Memory vector clear skipped: {e}")
-                return {"status": "deleted", "kind": kind, "count": count}
-
-            if kind == "skills":
-                # Skills live as SKILL.md files under data/skills/. Drop
-                # the entire directory; the SkillsManager re-creates the
-                # tree on next write.
-                skills_dir = SKILLS_DIR
-                count = 0
-                if os.path.isdir(skills_dir):
-                    # Count SKILL.md files for the response — quick walk.
-                    for _, _, files in os.walk(skills_dir):
-                        count += sum(1 for f in files if f == "SKILL.md")
-                    _rmtree_quiet(skills_dir)
-                # Legacy fallback file
-                legacy = SKILLS_FILE
-                if os.path.exists(legacy):
-                    try:
-                        os.remove(legacy)
-                    except OSError:
-                        pass
-                return {"status": "deleted", "kind": kind, "count": count}
-
-            if kind == "notes":
-                count = db.query(Note).count()
-                db.query(Note).delete()
-                db.commit()
-                return {"status": "deleted", "kind": kind, "count": count}
-
-            if kind == "tasks":
-                # TaskRun rows reference tasks via FK — clear them first.
-                db.query(TaskRun).delete()
-                count = db.query(ScheduledTask).count()
-                db.query(ScheduledTask).delete()
-                db.commit()
-                return {"status": "deleted", "kind": kind, "count": count}
-
-            if kind == "documents":
-                # DocumentVersion FKs Document — clear children first.
-                db.query(DocumentVersion).delete()
-                count = db.query(Document).count()
-                db.query(Document).delete()
-                db.commit()
-                return {"status": "deleted", "kind": kind, "count": count}
-
-            if kind == "gallery":
-                count = db.query(GalleryImage).count() + db.query(GalleryAlbum).count()
-                db.query(GalleryImage).delete()
-                db.query(GalleryAlbum).delete()
-                db.commit()
-                # Also drop the upload dir so disk doesn't keep orphans.
-                _rmtree_quiet(GALLERY_DIR)
-                _rmtree_quiet(GALLERY_UPLOADS_DIR)
-                return {"status": "deleted", "kind": kind, "count": count}
-
-            if kind == "calendar":
-                # Events FK calendars — clear children first, then both.
-                db.query(CalendarEvent).delete()
-                count = db.query(CalendarCal).count()
-                db.query(CalendarCal).delete()
-                db.commit()
-                return {"status": "deleted", "kind": kind, "count": count}
-
-            raise HTTPException(400, f"Unknown wipe kind: {kind!r}")
-        except HTTPException:
-            raise
-        except Exception as e:
-            db.rollback()
-            logger.exception(f"Wipe {kind} failed")
-            raise HTTPException(500, f"Wipe {kind} failed: {e}")
-        finally:
-            db.close()
-
-    return router
+_sys.modules[__name__] = _canonical

routes/calendar_routes.py

@@ -13,8 +13,9 @@ from sqlalchemy import or_, and_
 from dateutil.rrule import rrulestr
 
 from core.database import SessionLocal, CalendarCal, CalendarDeletedEvent, CalendarEvent
-from src.auth_helpers import require_user
+from src.auth_helpers import effective_user, require_user
 from src.upload_limits import read_upload_limited, ICS_MAX_BYTES
+from src.upload_handler import reserve_upload_references
 
 logger = logging.getLogger(__name__)
 
@@ -697,9 +698,18 @@ def _expand_rrule(
 
 # ── Routes ──
 
-def setup_calendar_routes() -> APIRouter:
+def setup_calendar_routes(upload_handler=None) -> APIRouter:
     router = APIRouter(prefix="/api/calendar", tags=["calendar"])
 
+    def _reserve_calendar_uploads(request: Request, *values) -> None:
+        missing_id = reserve_upload_references(
+            upload_handler,
+            effective_user(request),
+            *values,
+        )
+        if missing_id:
+            raise HTTPException(409, f"Referenced upload is no longer available: {missing_id}")
+
     # ── CalDAV multi-account helpers ─────────────────────────────────────────
 
     def _get_caldav_accounts(owner: str) -> list:
@@ -913,7 +923,24 @@ def setup_calendar_routes() -> APIRouter:
             '</d:prop></d:propfind>'
         )
         try:
-            async with httpx.AsyncClient(timeout=8.0, follow_redirects=False, trust_env=False) as cx:
+            # Build an SSL context that trusts the operator's custom CA bundle
+            # (SSL_CERT_FILE / REQUESTS_CA_BUNDLE) so self-signed CalDAV servers
+            # pass the pre-flight the same way they pass the real sync.
+            # trust_env=False is kept to block proxy/auth env leakage; the CA
+            # bundle is loaded explicitly instead.
+            import ssl as _ssl
+            _ssl_ctx = _ssl.create_default_context()
+            # Disable VERIFY_X509_STRICT so certs without a keyUsage extension
+            # (common in self-signed setups) are accepted, matching the
+            # requests/urllib3 behavior used by the CalDAV sync path.
+            _ssl_ctx.verify_flags &= ~_ssl.VERIFY_X509_STRICT
+            _ca_bundle = _os.environ.get("SSL_CERT_FILE") or _os.environ.get("REQUESTS_CA_BUNDLE")
+            if _ca_bundle:
+                if _os.path.isfile(_ca_bundle):
+                    _ssl_ctx.load_verify_locations(_ca_bundle)
+                else:
+                    logger.warning("CalDAV test: CA bundle %s not found, using system CAs", _ca_bundle)
+            async with httpx.AsyncClient(timeout=8.0, follow_redirects=False, trust_env=False, verify=_ssl_ctx) as cx:
                 r = await cx.request(
                     "PROPFIND", url,
                     auth=(user, pw),
@@ -1070,6 +1097,7 @@ def setup_calendar_routes() -> APIRouter:
     @router.post("/events")
     async def create_event(request: Request, data: EventCreate):
         owner = _require_user(request)
+        _reserve_calendar_uploads(request, data.color, data.description, data.location)
         db = SessionLocal()
         try:
             cal = None
@@ -1131,6 +1159,7 @@ def setup_calendar_routes() -> APIRouter:
     @router.put("/events/{uid}")
     async def update_event(request: Request, uid: str, data: EventUpdate):
         owner = _require_user(request)
+        _reserve_calendar_uploads(request, data.color, data.description, data.location)
         try:
             base_uid = _resolve_base_uid(uid)
         except ValueError as e:
@@ -1224,6 +1253,7 @@ def setup_calendar_routes() -> APIRouter:
     @router.post("/calendars")
     async def create_calendar(request: Request, name: str = "Imported", color: str = "#5b8abf"):
         owner = _require_user(request)
+        _reserve_calendar_uploads(request, color)
         db = SessionLocal()
         try:
             cal = CalendarCal(
@@ -1246,6 +1276,7 @@ def setup_calendar_routes() -> APIRouter:
     @router.put("/calendars/{cal_id}")
     async def update_calendar(request: Request, cal_id: str, name: str = None, color: str = None):
         owner = _require_user(request)
+        _reserve_calendar_uploads(request, color)
         db = SessionLocal()
         try:
             cal = _get_or_404_calendar(db, cal_id, owner)

routes/chat_helpers.py

@@ -5,6 +5,7 @@ import json
 import logging
 import os
 import re
+import time
 from dataclasses import dataclass, field
 from typing import Any, Optional
 
@@ -17,6 +18,7 @@ from src.context_compactor import maybe_compact, trim_for_context
 from src.model_context import estimate_tokens
 from src.auth_helpers import effective_user
 from src.prompt_security import untrusted_context_message
+from src.attachment_refs import attachment_ref
 from routes.prefs_routes import _load_for_user as load_prefs_for_user
 
 from fastapi import HTTPException
@@ -55,6 +57,9 @@ def _is_casual_low_signal(text: str) -> bool:
 # the background work (extraction, auto-naming) silently never runs.
 # Mirrors WebhookManager._spawn_tracked from src/webhook_manager.py.
 _BG_TASKS: set[asyncio.Task] = set()
+_INCOGNITO_CONTEXTS: dict[str, dict[str, Any]] = {}
+_INCOGNITO_CONTEXT_TTL_SECONDS = 6 * 60 * 60
+_INCOGNITO_CONTEXT_MAX_MESSAGES = 80
 
 
 def _spawn_bg(coro) -> asyncio.Task:
@@ -65,6 +70,40 @@ def _spawn_bg(coro) -> asyncio.Task:
     return task
 
 
+def _prune_incognito_contexts(now: float | None = None):
+    now = now or time.time()
+    stale = [
+        sid for sid, bundle in _INCOGNITO_CONTEXTS.items()
+        if now - float(bundle.get("updated_at") or 0) > _INCOGNITO_CONTEXT_TTL_SECONDS
+    ]
+    for sid in stale:
+        _INCOGNITO_CONTEXTS.pop(sid, None)
+
+
+def _incognito_messages(session_id: str) -> list[dict[str, Any]]:
+    _prune_incognito_contexts()
+    bundle = _INCOGNITO_CONTEXTS.get(str(session_id or ""))
+    if not bundle:
+        return []
+    return [dict(m) for m in bundle.get("messages", []) if isinstance(m, dict)]
+
+
+def _append_incognito_message(session_id: str, role: str, content: Any, metadata: dict | None = None):
+    sid = str(session_id or "").strip()
+    if not sid:
+        return
+    _prune_incognito_contexts()
+    bundle = _INCOGNITO_CONTEXTS.setdefault(sid, {"messages": [], "updated_at": time.time()})
+    msg: dict[str, Any] = {"role": role, "content": content}
+    if metadata:
+        msg["metadata"] = dict(metadata)
+    messages = bundle.setdefault("messages", [])
+    messages.append(msg)
+    if len(messages) > _INCOGNITO_CONTEXT_MAX_MESSAGES:
+        del messages[:-_INCOGNITO_CONTEXT_MAX_MESSAGES]
+    bundle["updated_at"] = time.time()
+
+
 # ── Data containers ────────────────────────────────────────────────────── #
 
 @dataclass
@@ -418,24 +457,28 @@ def build_uploaded_file_manifest(att_ids: list, upload_handler, owner: Optional[
             except Exception:
                 path = None
 
-        manifest.append({
-            "id": info.get("id") or str(att_id),
-            "name": info.get("name") or info.get("original_name") or str(att_id),
-            "mime": info.get("mime", ""),
-            "size": info.get("size", 0),
+        ref = attachment_ref({**info, "id": info.get("id") or str(att_id)})
+        ref.update({
+            "id": ref["attachment_id"],
+            "uri": f"odysseus://attachment/{ref['attachment_id']}",
+            "read_policy": "owner_checked_upload",
+            # Transitional compatibility: existing built-in tools can still use
+            # this path, but only after owner, upload-root, and tool-root checks.
             "path": path,
         })
+        manifest.append(ref)
     return manifest
 
 
 def add_user_message(sess, chat_handler, preprocessed: PreprocessedMessage, incognito: bool = False):
     """Add user message to session history and update session name.
-    In incognito mode, still add to in-memory history (for conversation context)
-    but skip session name update (which would persist)."""
+    Incognito messages must not mutate persistent session history, even in
+    memory, because a later normal turn can persist the same session object."""
+    if incognito:
+        return
     user_meta = {"attachments": preprocessed.attachment_meta} if preprocessed.attachment_meta else None
     sess.add_message(ChatMessage("user", preprocessed.user_content, metadata=user_meta))
-    if not incognito:
-        chat_handler.update_session_name_if_needed(sess, preprocessed.text_for_context)
+    chat_handler.update_session_name_if_needed(sess, preprocessed.text_for_context)
 
 
 def fire_message_event(request, webhook_manager, session_id: str, sess, message: str, compare_mode: bool = False):
@@ -664,8 +707,14 @@ async def build_chat_context(
         allow_tool_preprocessing=allow_tool_preprocessing,
     )
 
-    # Add user message to history
-    add_user_message(sess, chat_handler, preprocessed, incognito=incognito)
+    # Add user message to history. Nobody/incognito uses a request-local
+    # transcript store instead of session history so stale saved chats cannot
+    # bleed into context and the turn is not persisted.
+    if incognito:
+        user_meta = {"attachments": preprocessed.attachment_meta} if preprocessed.attachment_meta else None
+        _append_incognito_message(session_id, "user", preprocessed.user_content, user_meta)
+    else:
+        add_user_message(sess, chat_handler, preprocessed, incognito=False)
 
     # Fire events
     if not incognito:
@@ -756,8 +805,10 @@ async def build_chat_context(
     if norm:
         sess.model = norm
 
-    # Build messages
-    messages = preface + sess.get_context_messages()
+    # Build messages. In Nobody/incognito mode, never read saved session
+    # history: the session id may be a temporary wrapper or, in buggy clients, a
+    # stale normal session id. Only the ephemeral incognito transcript is safe.
+    messages = preface + (_incognito_messages(session_id) if incognito else sess.get_context_messages())
 
     # Current date/time — injected as a standalone *user*-role context message
     # placed immediately before the latest user turn, NOT folded into the
@@ -1023,7 +1074,12 @@ def save_assistant_response(
     tool_events: list = None,
     incognito: bool = False,
 ):
-    """Add assistant response to session history. In incognito mode, keeps in-memory context but skips DB persistence."""
+    """Add assistant response to session history.
+
+    Incognito responses are intentionally not added to the session object. The
+    session may later be saved by a normal turn, so "in-memory only" is not
+    private enough.
+    """
     md = dict(last_metrics) if last_metrics else {}
     def _model_value(value) -> str:
         if value is None:
@@ -1063,19 +1119,18 @@ def save_assistant_response(
         _content = _think_info["reply"]
     else:
         _content = full_response
+    if incognito:
+        _append_incognito_message(session_id, "assistant", _content, md)
+        return None
     sess.add_message(ChatMessage("assistant", _content, metadata=md))
 
-    if not incognito:
-        from core.database import update_session_last_accessed
-        update_session_last_accessed(session_id)
-        session_manager.save_sessions()
+    from core.database import update_session_last_accessed
+    update_session_last_accessed(session_id)
+    session_manager.save_sessions()
 
     # Return the persisted message's DB id so the stream can wire it onto the
     # freshly-rendered bubble — lets the user edit/delete a just-streamed reply
-    # without reloading. Incognito returns None: those messages are ephemeral,
-    # so we don't hand out an edit/delete handle for them.
-    if incognito:
-        return None
+    # without reloading.
     try:
         _last = sess.history[-1]
         _meta = getattr(_last, "metadata", None)

routes/chat_routes.py

@@ -42,6 +42,7 @@ from routes.chat_helpers import (
     _enforce_chat_privileges,
 )
 from src.action_intents import ToolIntent, classify_tool_intent as _classify_tool_intent
+from src.image_model_ids import looks_like_image_generation_model
 from src.tool_policy import (
     WEB_TOOL_NAMES,
     build_effective_tool_policy,
@@ -53,7 +54,6 @@ logger = logging.getLogger(__name__)
 
 # Track active streams for partial-save safety net
 _active_streams: Dict[str, dict] = {}
-_IMAGE_MODEL_PREFIXES = ("gpt-image", "dall-e", "chatgpt-image")
 
 
 def _stream_set(session_id: str, **fields) -> None:
@@ -111,7 +111,8 @@ def _ensure_current_request_is_latest_user(messages: List[Dict[str, Any]], curre
 _WEB_FOLLOWUP_RE = re.compile(
     r"^\s*(?:(?:can|could|would|will)\s+you\s+)?"
     r"(?:check|try\s+again|look(?:\s+now|\s+it\s+up)?|search(?:\s+now|\s+online|\s+it)?|"
-    r"do\s+it|again)\??\s*$",
+    r"do\s+it|again|approved|approve(?:d)?|yes|ok(?:ay)?|proceed|go\s+ahead|"
+    r"send(?:\s+it)?|submit(?:\s+it)?|email(?:\s+them|\s+it)?)\??\s*$",
     re.I,
 )
 _RECENT_WEB_CONTEXT_RE = re.compile(
@@ -119,6 +120,26 @@ _RECENT_WEB_CONTEXT_RE = re.compile(
     r"price|current|latest|search|look\s+up|online)\b",
     re.I,
 )
+_RECENT_BROWSER_CONTEXT_RE = re.compile(
+    r"\b(?:browser|browse|open\s+(?:the\s+)?(?:site|page|url|link)|click|"
+    r"fill(?:\s+out)?|submit|send\s+(?:the\s+)?form|contact\s+form|web\s*form|"
+    r"form\s+submission|playwright|automation)\b",
+    re.I,
+)
+_BROWSER_MCP_TOOLS = {
+    "mcp__builtin_browser__browser_navigate",
+    "mcp__builtin_browser__browser_snapshot",
+    "mcp__builtin_browser__browser_click",
+    "mcp__builtin_browser__browser_type",
+    "mcp__builtin_browser__browser_fill_form",
+    "mcp__builtin_browser__browser_select_option",
+    "mcp__builtin_browser__browser_press_key",
+    "mcp__builtin_browser__browser_wait_for",
+    "mcp__builtin_browser__browser_take_screenshot",
+    "mcp__builtin_browser__browser_drag",
+    "mcp__builtin_browser__browser_navigate_back",
+    "mcp__builtin_browser__browser_close",
+}
 
 
 def _recent_session_text(sess, limit: int = 8, max_chars: int = 2000) -> str:
@@ -141,6 +162,13 @@ def _is_contextual_web_followup(message: str, sess) -> bool:
     return bool(_RECENT_WEB_CONTEXT_RE.search(_recent_session_text(sess)))
 
 
+def _is_contextual_browser_followup(message: str, sess) -> bool:
+    """Treat short retry replies as browser tasks when recent context was forms/browser automation."""
+    if not message or not _WEB_FOLLOWUP_RE.search(message):
+        return False
+    return bool(_RECENT_BROWSER_CONTEXT_RE.search(_recent_session_text(sess, limit=12, max_chars=4000)))
+
+
 def _resolve_request_workspace(request, raw_value) -> tuple:
     """Resolve the posted workspace for this request: (workspace, rejected).
 
@@ -168,6 +196,46 @@ def _resolve_request_workspace(request, raw_value) -> tuple:
     return workspace, (requested if not workspace else "")
 
 
+_ABS_PATH_RE = re.compile(r"(?<!\S)(~?/[^\"'\s`<>]+)")
+_LOCAL_FILE_TASK_RE = re.compile(
+    r"\b(?:file|folder|directory|path|workspace|repo|project|movie|video|"
+    r"subtitle|subtitles|srt|vtt|ass|download|save|rename|move|copy|extract|"
+    r"convert|ffmpeg|run|execute|open|read|inspect|fix|debug|test|build)\b",
+    re.IGNORECASE,
+)
+
+
+def _resolve_workspace_from_message_path(request, message: str) -> tuple[str, str]:
+    """Auto-bind a workspace only when the user names an explicit safe path.
+
+    This is intentionally deterministic rather than LLM/RAG-driven: RAG can
+    choose the tool family, but filesystem binding must not let a prompt infer
+    or probe arbitrary host paths. For a file path, bind its parent directory.
+    For a directory path, bind that directory.
+    """
+    text = str(message or "")
+    if not text or not _LOCAL_FILE_TASK_RE.search(text):
+        return "", ""
+
+    from src.tool_security import owner_is_admin_or_single_user
+    if not owner_is_admin_or_single_user(get_current_user(request)):
+        return "", ""
+
+    from src.tool_execution import vet_workspace
+
+    for match in _ABS_PATH_RE.finditer(text):
+        raw = match.group(1).rstrip(".,;:)]}")
+        expanded = os.path.realpath(os.path.expanduser(raw))
+        candidates = [expanded]
+        if os.path.isfile(expanded):
+            candidates.insert(0, os.path.dirname(expanded))
+        for candidate in candidates:
+            workspace = vet_workspace(candidate) or ""
+            if workspace:
+                return workspace, ""
+    return "", ""
+
+
 def _session_url_matches_endpoint(session_url: str, endpoint_base: str) -> bool:
     if not session_url or not endpoint_base:
         return False
@@ -243,7 +311,7 @@ def _is_image_generation_session(sess, owner: str | None = None) -> bool:
     models into the image-generation path.
     """
     model = (getattr(sess, "model", "") or "").strip()
-    if any(model.lower().startswith(prefix) for prefix in _IMAGE_MODEL_PREFIXES):
+    if looks_like_image_generation_model(model):
         return True
 
     endpoint_url = (getattr(sess, "endpoint_url", "") or "").strip()
@@ -271,6 +339,29 @@ def _is_image_generation_session(sess, owner: str | None = None) -> bool:
     return False
 
 
+def _first_image_attachment(chat_handler, att_ids: List[str], owner: str | None = None) -> Optional[Dict[str, Any]]:
+    """Return the first attached image file that this owner can read."""
+    upload_handler = getattr(chat_handler, "upload_handler", None)
+    if not upload_handler:
+        return None
+    for att_id in att_ids or []:
+        try:
+            info = upload_handler.resolve_upload(att_id, owner=owner)
+        except Exception as e:
+            logger.warning("Failed to resolve image edit upload %s", att_id, exc_info=e)
+            continue
+        if not info:
+            continue
+        name = info.get("name") or info.get("original_name") or info.get("id") or ""
+        mime = info.get("mime", "")
+        try:
+            if upload_handler.is_image_file(name, mime):
+                return info
+        except Exception:
+            continue
+    return None
+
+
 def _recover_empty_session_model(sess, session_id: str, owner: str | None = None) -> bool:
     """Re-populate sess.model from the matching endpoint's cached models.
 
@@ -381,9 +472,85 @@ def _recover_empty_session_model(sess, session_id: str, owner: str | None = None
     except Exception as e:
         db.rollback()
         logger.warning("Failed to recover empty session model for %s: %s", session_id, e)
+    return False
+
+
+def _reconcile_selected_route_from_request(
+    request: Request,
+    sess,
+    session_id: str,
+    form_data,
+    owner: str | None = None,
+) -> bool:
+    """Apply the model route the browser selected before streaming.
+
+    The frontend creates a pending chat first and only materializes it on first
+    send. Startup/default-model refreshes can race with that UI state, so the
+    stream request includes the route that was selected at click/send time.
+    Trust only registered endpoint ids, or the session's existing endpoint URL.
+    """
+    selected_model = str(form_data.get("selected_model") or "").strip()
+    selected_endpoint_id = str(form_data.get("selected_endpoint_id") or "").strip()
+    selected_endpoint_url = str(form_data.get("selected_endpoint_url") or "").strip()
+    if not selected_model:
         return False
+
+    endpoint_url = ""
+    headers = None
+    if selected_endpoint_id or selected_endpoint_url:
+        try:
+            from src.auth_helpers import owner_filter
+            from src.endpoint_resolver import build_headers, normalize_base
+            db = SessionLocal()
+            try:
+                q = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True)
+                if selected_endpoint_id:
+                    q = q.filter(ModelEndpoint.id == selected_endpoint_id)
+                if owner:
+                    q = owner_filter(q, ModelEndpoint, owner)
+                candidates = q.all() if selected_endpoint_url and not selected_endpoint_id else [q.first()]
+                ep = None
+                for cand in candidates:
+                    if not cand:
+                        continue
+                    if selected_endpoint_id or _session_url_matches_endpoint(selected_endpoint_url, cand.base_url or ""):
+                        ep = cand
+                        break
+                if not ep:
+                    return False
+                endpoint_url = build_chat_url(normalize_base(ep.base_url or ""))
+                headers = build_headers(ep.api_key or "", ep.base_url or "") if ep.api_key else {}
+            finally:
+                db.close()
+        except Exception as e:
+            logger.warning("Failed to resolve selected endpoint %s/%s for %s: %s", selected_endpoint_id, selected_endpoint_url, session_id, e)
+            return False
+
+    if not endpoint_url:
+        return False
+
+    if (
+        selected_model == (getattr(sess, "model", "") or "")
+        and endpoint_url == (getattr(sess, "endpoint_url", "") or "")
+    ):
+        return False
+
+    sess.model = selected_model
+    sess.endpoint_url = endpoint_url
+    sess.headers = headers or {}
+    db = SessionLocal()
+    try:
+        db_session = db.query(DBSession).filter(DBSession.id == session_id).first()
+        if db_session:
+            db_session.model = selected_model
+            db_session.endpoint_url = endpoint_url
+            db_session.headers = sess.headers or {}
+            db_session.updated_at = datetime.utcnow()
+            db.commit()
     finally:
         db.close()
+    logger.info("Reconciled selected route for %s: model=%r endpoint=%s", session_id, selected_model, redact_url(endpoint_url))
+    return True
 
 
 def _set_user_time_from_request(request: Request) -> None:
@@ -565,9 +732,7 @@ def setup_chat_routes(
         search_context = form_data.get("search_context")  # pre-fetched web search results (compare mode)
         compare_mode = str(form_data.get("compare_mode", "")).lower() == "true"
         incognito = str(form_data.get("incognito", "")).lower() == "true"
-        # Plan mode is not part of the merge-ready UI. Ignore stale clients or
-        # manual form posts that still send plan_mode=true.
-        plan_mode = False
+        plan_mode = str(form_data.get("plan_mode") or (body or {}).get("plan_mode") or "").lower() == "true"
         chat_mode = str(form_data.get("mode", "")).lower()  # 'chat' or 'agent'
         # Workspace: confine the agent's file/shell tools to this folder.
         workspace, workspace_rejected = _resolve_request_workspace(
@@ -589,6 +754,25 @@ def setup_chat_routes(
         # not chats we quietly promoted for a notes/calendar intent.
         user_requested_agent = (chat_mode == "agent")
         _search_enabled = web_search_enabled_for_turn(allow_web_search, use_web)
+        _explicit_web_intent = False
+        _explicit_browser_intent = False
+        if isinstance(message, str):
+            _msg_l = message.lower()
+            _explicit_web_intent = bool(re.search(
+                r"\b(search|look\s*up|lookup|google|browse|web|online|latest|current|today|news|weather|forecast|rate|exchange\s+rate)\b",
+                _msg_l,
+            ))
+            _explicit_browser_intent = bool(re.search(
+                r"\b(browser|browse|open\s+(?:the\s+)?(?:site|page|url|link)|"
+                r"click|fill(?:\s+out)?|submit|send\s+(?:the\s+)?form|"
+                r"contact\s+form|web\s*form|form\s+submission)\b",
+                _msg_l,
+            ))
+        _allow_browser_for_web_turn = bool(
+            _explicit_browser_intent
+            or _explicit_web_intent
+            or _search_enabled
+        )
         # Intent auto-escalation: if the user is clearly asking the assistant
         # to create a todo, reminder, or calendar event, promote chat → agent
         # for this turn so the LLM has access to manage_notes / manage_calendar.
@@ -598,9 +782,13 @@ def setup_chat_routes(
         # shell disabled).
         auto_escalated = False
         _tool_intent = _classify_tool_intent(message) if isinstance(message, str) else None
+        _workspace_agent_intent = False
         if chat_mode == "chat" and _tool_intent and _tool_intent.needs_tools:
             chat_mode = "agent"
             auto_escalated = True
+            _workspace_agent_intent = _tool_intent.category in {"shell", "workspace"}
+            if _workspace_agent_intent:
+                allow_bash = "true"
             logger.info(
                 "chat→agent auto-escalation: category=%s reason=%s",
                 _tool_intent.category,
@@ -610,6 +798,10 @@ def setup_chat_routes(
             chat_mode = "agent"
             auto_escalated = True
             logger.info("chat→agent auto-escalation: search enabled")
+        elif chat_mode == "chat" and _explicit_web_intent:
+            chat_mode = "agent"
+            auto_escalated = True
+            logger.info("chat→agent auto-escalation: explicit web intent")
         active_doc_id = form_data.get("active_doc_id", "").strip()
         logger.info(f"[doc-inject] chat_mode={chat_mode}, active_doc_id={active_doc_id!r}")
 
@@ -688,6 +880,7 @@ def setup_chat_routes(
             _verify_session_owner(request, session)
             sess = session_manager.get_session(session)
             owner = effective_user(request)
+            _reconcile_selected_route_from_request(request, sess, session, form_data, owner=owner)
             if _clear_orphaned_session_endpoint(sess, owner=owner):
                 raise HTTPException(400, "Selected model endpoint was removed. Pick another model in Settings.")
             # Issue #587: picker shows a model from the endpoint cache but
@@ -711,11 +904,28 @@ def setup_chat_routes(
                 _tool_intent = ToolIntent(True, "web", "contextual web lookup follow-up")
                 chat_mode = "agent"
                 auto_escalated = True
+                _workspace_agent_intent = False
                 logger.info(
                     "chat→agent auto-escalation: category=%s reason=%s",
                     _tool_intent.category,
                     _tool_intent.reason,
                 )
+            if isinstance(message, str) and _is_contextual_browser_followup(message, sess):
+                _explicit_browser_intent = True
+                if chat_mode == "chat":
+                    chat_mode = "agent"
+                    auto_escalated = True
+                    _workspace_agent_intent = False
+                    logger.info("chat→agent auto-escalation: contextual browser/form follow-up")
+            if not workspace and isinstance(message, str):
+                _auto_workspace, _ = _resolve_workspace_from_message_path(request, message)
+                if _auto_workspace:
+                    workspace = _auto_workspace
+                    chat_mode = "agent"
+                    auto_escalated = True
+                    _workspace_agent_intent = True
+                    allow_bash = "true"
+                    logger.info("chat→agent auto-escalation: explicit path workspace=%s", workspace)
         except SessionNotFoundError as e:
             raise HTTPException(404, str(e))
         except (ValueError, ValidationError):
@@ -750,7 +960,12 @@ def setup_chat_routes(
             except Exception as e:
                 logger.warning("Failed to parse attachments JSON, ignoring attachments", exc_info=e)
 
+        image_generation_session = _is_image_generation_session(sess, owner=effective_user(request))
         no_memory = str(form_data.get("no_memory", "")).lower() == "true"
+        if image_generation_session:
+            no_memory = True
+            use_rag = "false"
+            search_context = None
         pre_context_tool_policy = build_effective_tool_policy(
             last_user_message=message,
         )
@@ -879,7 +1094,7 @@ def setup_chat_routes(
         # explicitly enable it.
         if allow_bash is not None and str(allow_bash).lower() != "true":
             disabled_tools.add("bash")
-        _explicit_web_intent = bool(_tool_intent and _tool_intent.category == "web")
+        _explicit_web_intent = _explicit_web_intent or bool(_tool_intent and _tool_intent.category == "web")
         if is_web_search_explicitly_denied(allow_web_search) or not _search_enabled:
             disabled_tools.update(WEB_TOOL_NAMES)
         if _explicit_web_intent:
@@ -893,7 +1108,7 @@ def setup_chat_routes(
                 "create_document", "edit_document", "update_document",
                 "send_email", "reply_to_email",
                 "manage_notes", "manage_calendar", "manage_tasks",
-                "api_call", "builtin_browser",
+                "api_call",
             })
             if _search_enabled:
                 disabled_tools.difference_update(WEB_TOOL_NAMES)
@@ -909,6 +1124,11 @@ def setup_chat_routes(
                 "manage_memory",      # persistent memory store
                 "search_chats",       # past chat history
                 "manage_skills",      # skill presets tied to user
+                "create_session",
+                "list_sessions",
+                "manage_session",
+                "send_to_session",
+                "chat_with_model",
             })
 
         # Active email reader open → strip the tools that let the agent drift
@@ -935,7 +1155,7 @@ def setup_chat_routes(
             if not _privs.get("can_use_bash", True):
                 disabled_tools.update({"bash", "python", "read_file", "write_file"})
             if not _privs.get("can_use_browser", True):
-                disabled_tools.add("builtin_browser")
+                disabled_tools.update(_BROWSER_MCP_TOOLS)
             if not _privs.get("can_use_documents", True):
                 disabled_tools.update({"create_document", "edit_document", "update_document", "suggest_document"})
             if not _privs.get("can_generate_images", True):
@@ -958,10 +1178,12 @@ def setup_chat_routes(
         # the heavy "do things on the computer" tools — otherwise the model
         # tries to shell out for a request that never needed it, then fails
         # (and looks broken when the shell is disabled).
-        if auto_escalated:
+        if auto_escalated and not _workspace_agent_intent:
             disabled_tools.update({
-                "bash", "python", "read_file", "write_file", "builtin_browser",
+                "bash", "python", "read_file", "write_file",
             })
+            if not _allow_browser_for_web_turn:
+                disabled_tools.update(_BROWSER_MCP_TOOLS)
 
         # Disable document tools in compare sessions — they break the pane UI
         if sess.name and sess.name.startswith("[CMP]"):
@@ -1195,7 +1417,7 @@ def setup_chat_routes(
                 _model_info["character_name"] = ctx.preset.character_name
             yield f'data: {json.dumps(_model_info)}\n\n'
 
-            if _is_image_generation_session(sess, owner=_user):
+            if image_generation_session:
                 from src.settings import get_setting
                 if tool_policy.blocks("generate_image"):
                     _blocked_msg = tool_policy.reason_for("generate_image")
@@ -1208,26 +1430,85 @@ def setup_chat_routes(
                     yield "data: [DONE]\n\n"
                     _active_streams.pop(session, None)
                     return
-                from src.ai_interaction import do_generate_image
+                from src.ai_interaction import do_edit_image, do_generate_image
                 _user_msg = message or ""
-                yield f'data: {json.dumps({"type": "tool_start", "tool": "generate_image", "command": _user_msg[:100]})}\n\n'
+                _image_upload = _first_image_attachment(chat_handler, att_ids, owner=_user)
+                _image_tool_name = "edit_image" if _image_upload else "generate_image"
+                yield f'data: {json.dumps({"type": "tool_start", "tool": _image_tool_name, "command": _user_msg[:100]})}\n\n'
                 yield ": heartbeat\n\n"
-                _img_result = await do_generate_image(f"{_user_msg}\n{sess.model}", session, owner=_user)
+                _progress_queue: asyncio.Queue = asyncio.Queue()
+
+                async def _image_progress_callback(progress: Dict[str, Any]):
+                    try:
+                        _progress_queue.put_nowait(progress)
+                    except Exception:
+                        pass
+
+                if _image_upload:
+                    _img_task = asyncio.create_task(do_edit_image(
+                        _user_msg,
+                        _image_upload.get("path", ""),
+                        model_spec=sess.model,
+                        session_id=session,
+                        owner=_user,
+                        size="1024x1024",
+                        progress_callback=_image_progress_callback,
+                    ))
+                else:
+                    _img_task = asyncio.create_task(do_generate_image(f"{_user_msg}\n{sess.model}\n512x512", session, owner=_user))
+                _img_started = time.time()
+                _img_tick = 0
+                while not _img_task.done():
+                    try:
+                        _progress = await asyncio.wait_for(_progress_queue.get(), timeout=2.0)
+                    except asyncio.TimeoutError:
+                        _progress = None
+                    _img_tick += 1
+                    _elapsed = int(time.time() - _img_started)
+                    _label = "Editing image" if _image_upload else "Generating image"
+                    yield ": image generation still running\n\n"
+                    _progress_data = {"type": "tool_progress", "tool": _image_tool_name, "message": f"{_label}… {_elapsed}s", "elapsed": _elapsed, "tick": _img_tick}
+                    if isinstance(_progress, dict) and _progress.get("total"):
+                        _step = int(_progress.get("step") or 0)
+                        _total = int(_progress.get("total") or 0)
+                        _percent = _progress.get("percent")
+                        _progress_data.update({
+                            "step": _step,
+                            "total": _total,
+                            "percent": _percent,
+                            "message": f"{_label}… {_step}/{_total}",
+                        })
+                    yield f'data: {json.dumps(_progress_data)}\n\n'
+                _img_result = await _img_task
                 _img_output = _img_result.get("results", _img_result.get("error", ""))
-                _img_tool_data = {"type": "tool_output", "tool": "generate_image", "command": _user_msg[:100], "output": _img_output, "exit_code": 0 if "error" not in _img_result else 1}
+                _img_tool_data = {"type": "tool_output", "tool": _image_tool_name, "command": _user_msg[:100], "output": _img_output, "exit_code": 0 if "error" not in _img_result else 1}
                 for _k in ("image_url", "image_id", "image_prompt", "image_model", "image_size", "image_quality"):
                     if _k in _img_result:
                         _img_tool_data[_k] = _img_result[_k]
+                if _image_upload:
+                    _img_tool_data["source_image"] = {
+                        "id": _image_upload.get("id"),
+                        "name": _image_upload.get("name") or _image_upload.get("original_name"),
+                    }
                 yield f'data: {json.dumps(_img_tool_data)}\n\n'
+                if _img_result.get("image_url"):
+                    _img_event = {"type": "generated_image", "url": _img_result.get("image_url")}
+                    for _k in ("image_url", "image_id", "image_prompt", "image_model", "image_size", "image_quality"):
+                        if _img_result.get(_k):
+                            _img_event[_k] = _img_result[_k]
+                    yield f'data: {json.dumps(_img_event)}\n\n'
                 _desc = _img_result.get("results", _img_result.get("error", "Image generation complete"))
                 full_response = _desc
                 yield f'data: {json.dumps({"delta": _desc})}\n\n'
                 # Save to session history
                 if not incognito:
-                    _ev = {"round": 1, "tool": "generate_image", "command": _user_msg[:100], "output": _img_output, "exit_code": 0 if "error" not in _img_result else 1}
+                    _ev = {"round": 1, "tool": _image_tool_name, "command": _user_msg[:100], "output": _img_output, "exit_code": 0 if "error" not in _img_result else 1}
                     for _ek in ("image_url", "image_id", "image_prompt", "image_model", "image_size", "image_quality"):
                         if _img_result.get(_ek):
                             _ev[_ek] = _img_result[_ek]
+                    if _image_upload:
+                        _ev["source_image_id"] = _image_upload.get("id")
+                        _ev["source_image_name"] = _image_upload.get("name") or _image_upload.get("original_name")
                     sess.add_message(ChatMessage("assistant", full_response, metadata={"tool_events": [_ev], "model": sess.model}))
                     session_manager.save_sessions()
                 yield f'data: {json.dumps({"type": "metrics", "data": {"total_time": 0}})}\n\n'
@@ -1292,8 +1573,10 @@ def setup_chat_routes(
                                         last_metrics["context_messages_after_trim"] = ctx.context_messages_after_trim
                                         last_metrics["context_tokens_before_trim"] = ctx.context_tokens_before_trim
                                         last_metrics["context_tokens_after_trim"] = ctx.context_tokens_after_trim
-                                    if ctx.context_length and last_metrics.get("input_tokens"):
-                                        pct = min(round((last_metrics["input_tokens"] / ctx.context_length) * 100, 1), 100.0)
+                                    request_context_tokens = ctx.context_tokens_after_trim or estimate_tokens(messages)
+                                    last_metrics["request_context_tokens"] = request_context_tokens
+                                    if ctx.context_length and request_context_tokens:
+                                        pct = min(round((request_context_tokens / ctx.context_length) * 100, 1), 100.0)
                                         last_metrics["context_percent"] = pct
                                         last_metrics["context_length"] = ctx.context_length
                                     # The frontend reads `tokens_per_second`; the raw usage event
@@ -1326,6 +1609,7 @@ def setup_chat_routes(
                                     "input_tokens": _est_in,
                                     "output_tokens": _est_out,
                                     "tokens_per_second": _tps,
+                                    "request_context_tokens": _est_in,
                                     "context_percent": _ctx_pct,
                                     "context_length": ctx.context_length,
                                     "model": _actual_model or _answered_by or _requested_model,
@@ -1360,7 +1644,7 @@ def setup_chat_routes(
                             _stream_set(session, status="done")
                             yield chunk
                 except (asyncio.CancelledError, GeneratorExit):
-                    if full_response:
+                    if full_response and not incognito:
                         logger.info("Client disconnected mid-stream (chat mode) for session %s, saving partial (%d chars)", session, len(full_response))
                         _stopped_content, _stopped_md = clean_thinking_for_save(
                             full_response,
@@ -1371,8 +1655,7 @@ def setup_chat_routes(
                             },
                         )
                         sess.add_message(ChatMessage("assistant", _stopped_content, metadata=_stopped_md))
-                        if not incognito:
-                            session_manager.save_sessions()
+                        session_manager.save_sessions()
                     raise
                 finally:
                     _active_streams.pop(session, None)
@@ -1405,6 +1688,10 @@ def setup_chat_routes(
                     _forced_tools = None
                     if _search_enabled:
                         _forced_tools = set(WEB_TOOL_NAMES)
+                        if _explicit_browser_intent:
+                            _forced_tools |= set(_BROWSER_MCP_TOOLS)
+                    elif _explicit_browser_intent:
+                        _forced_tools = set(_BROWSER_MCP_TOOLS)
 
                     async for chunk in stream_agent_loop(
                         sess.endpoint_url,
@@ -1450,7 +1737,9 @@ def setup_chat_routes(
                                     "tool_start", "tool_output", "agent_step",
                                     "doc_stream_open", "doc_stream_delta",
                                     "doc_update", "doc_suggestions", "ui_control",
-                                    "rounds_exhausted",
+                                    "rounds_exhausted", "budget_exceeded",
+                                    "loop_breaker_triggered",
+                                    "intent_nudge_exhausted",
                                     "ask_user",
                                     "plan_update",
                                 ):
@@ -1527,7 +1816,7 @@ def setup_chat_routes(
                     # outer finally from running and left _active_streams
                     # with a stale entry).
                     try:
-                        if full_response:
+                        if full_response and not incognito:
                             logger.info("Client disconnected mid-stream for session %s, saving partial response (%d chars)", session, len(full_response))
                             _stopped_content2, _stopped_md2 = clean_thinking_for_save(
                                 full_response,
@@ -1538,8 +1827,7 @@ def setup_chat_routes(
                                 },
                             )
                             sess.add_message(ChatMessage("assistant", _stopped_content2, metadata=_stopped_md2))
-                            if not incognito:
-                                session_manager.save_sessions()
+                            session_manager.save_sessions()
                     except Exception:
                         logger.exception("Failed to save partial response on disconnect (session %s)", session)
                     raise

routes/cleanup/__init__.py

@@ -0,0 +1,5 @@
+"""Cleanup route domain package (slice 2g, #4082/#4071).
+
+Contains cleanup_routes.py, migrated from the flat routes/ directory.
+Backward-compat shim at routes/cleanup_routes.py re-exports from here.
+"""

routes/cleanup/cleanup_routes.py

@@ -0,0 +1,60 @@
+# routes/cleanup_routes.py
+"""Routes for cleanup operations."""
+import logging
+from fastapi import APIRouter, HTTPException, Request
+from src.cleanup_service import get_cleanup_preview, cleanup_sessions
+from src.auth_helpers import get_current_user
+
+logger = logging.getLogger(__name__)
+
+def setup_cleanup_routes(session_manager):
+    """
+    Setup cleanup-related routes.
+
+    Args:
+        session_manager: SessionManager instance
+
+    Returns:
+        APIRouter instance with cleanup routes
+    """
+    router = APIRouter(prefix="/api/cleanup")
+
+    @router.get("/preview")
+    async def cleanup_preview(request: Request):
+        """
+        Preview what would be cleaned up without making any changes.
+
+        Returns:
+            JSON response with lists of sessions that would be archived/deleted and estimated space savings
+        """
+        user = get_current_user(request)
+        try:
+            preview = await get_cleanup_preview(owner=user)
+            return preview
+        except Exception as e:
+            logger.error(f"Cleanup preview failed: {e}")
+            raise HTTPException(500, "Cleanup preview generation failed")
+
+    @router.post("")
+    async def cleanup_endpoint(request: Request):
+        """
+        Perform cleanup operations:
+        1. Archive inactive sessions (not accessed for 7 days)
+        2. Delete old sessions (archived, not important, not accessed for 14+ days, with fewer than 10 messages)
+
+        Returns:
+            JSON response with counts of deleted and archived sessions, and space freed
+        """
+        user = get_current_user(request)
+        try:
+            archived_count, deleted_count, space_freed_mb = await cleanup_sessions(session_manager, owner=user)
+            return {
+                "archived_count": archived_count,
+                "deleted_count": deleted_count,
+                "space_freed_mb": round(space_freed_mb, 2)
+            }
+        except Exception as e:
+            logger.error(f"Cleanup failed: {e}")
+            raise HTTPException(500, "Cleanup operation failed")
+
+    return router

routes/cleanup_routes.py

@@ -1,60 +1,17 @@
-# routes/cleanup_routes.py
-"""Routes for cleanup operations."""
-import logging
-from fastapi import APIRouter, HTTPException, Request
-from src.cleanup_service import get_cleanup_preview, cleanup_sessions
-from src.auth_helpers import get_current_user
+"""Backward-compat shim — canonical location is routes/cleanup/cleanup_routes.py.
 
-logger = logging.getLogger(__name__)
+This module is replaced in ``sys.modules`` by the canonical module object so
+that ``import routes.cleanup_routes``, ``from routes.cleanup_routes import X``,
+``importlib.import_module("routes.cleanup_routes")``, and the string-targeted
+``monkeypatch.setattr("routes.cleanup_routes.get_cleanup_preview", ...)`` /
+``"routes.cleanup_routes.get_current_user"`` / ``"routes.cleanup_routes.
+cleanup_sessions"`` pattern used by test_cleanup_owner_scope.py all operate
+on the *same* object the application actually uses. Keeps existing import
+paths working after slice 2g (#4082/#4071).
+"""
 
-def setup_cleanup_routes(session_manager):
-    """
-    Setup cleanup-related routes.
+import sys as _sys
 
-    Args:
-        session_manager: SessionManager instance
+from routes.cleanup import cleanup_routes as _canonical  # noqa: F401
 
-    Returns:
-        APIRouter instance with cleanup routes
-    """
-    router = APIRouter(prefix="/api/cleanup")
-
-    @router.get("/preview")
-    async def cleanup_preview(request: Request):
-        """
-        Preview what would be cleaned up without making any changes.
-
-        Returns:
-            JSON response with lists of sessions that would be archived/deleted and estimated space savings
-        """
-        user = get_current_user(request)
-        try:
-            preview = await get_cleanup_preview(owner=user)
-            return preview
-        except Exception as e:
-            logger.error(f"Cleanup preview failed: {e}")
-            raise HTTPException(500, "Cleanup preview generation failed")
-
-    @router.post("")
-    async def cleanup_endpoint(request: Request):
-        """
-        Perform cleanup operations:
-        1. Archive inactive sessions (not accessed for 7 days)
-        2. Delete old sessions (archived, not important, not accessed for 14+ days, with fewer than 10 messages)
-
-        Returns:
-            JSON response with counts of deleted and archived sessions, and space freed
-        """
-        user = get_current_user(request)
-        try:
-            archived_count, deleted_count, space_freed_mb = await cleanup_sessions(session_manager, owner=user)
-            return {
-                "archived_count": archived_count,
-                "deleted_count": deleted_count,
-                "space_freed_mb": round(space_freed_mb, 2)
-            }
-        except Exception as e:
-            logger.error(f"Cleanup failed: {e}")
-            raise HTTPException(500, "Cleanup operation failed")
-
-    return router
+_sys.modules[__name__] = _canonical

routes/compare/__init__.py

@@ -0,0 +1,5 @@
+"""Compare route domain package (slice 2i, #4082/#4071).
+
+Contains compare_routes.py, migrated from the flat routes/ directory.
+Backward-compat shim at routes/compare_routes.py re-exports from here.
+"""

routes/compare/compare_routes.py

@@ -0,0 +1,365 @@
+# routes/compare_routes.py
+"""Model A/B comparison routes."""
+import json
+import uuid
+import random
+from datetime import datetime
+from fastapi import APIRouter, Form, HTTPException, Request
+from typing import List
+from pydantic import BaseModel
+import logging
+
+from core.database import Comparison, SessionLocal
+from core.session_manager import SessionManager
+from src.auth_helpers import get_current_user
+from routes.session_routes import _reject_raw_endpoint_url_for_non_admin
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/compare", tags=["compare"])
+
+
+def _owned_endpoint_by_url(db, base_url, owner):
+    """ModelEndpoint whose base_url == `base_url` and is VISIBLE to `owner`
+    (their own rows + legacy null-owner "shared" rows); None otherwise.
+
+    Owner-scoped on purpose. ModelEndpoint is per-user (core/database.py: non-null
+    owner = private, "the model picker only shows the endpoint to that user") and
+    holds a decrypted `api_key`. start_comparison copies the matched row's api_key
+    into the caller-owned [CMP] session's headers, which then drives that session's
+    /api/chat_stream calls — so an UNSCOPED base_url match would let a user mint a
+    comparison bound to ANOTHER user's private endpoint and spend that owner's
+    api_key / reach whatever base_url they configured. Mirrors
+    session_routes._owned_endpoint. A null/empty owner is a no-op (single-user /
+    legacy mode).
+    """
+    from core.database import ModelEndpoint
+    from src.auth_helpers import owner_filter
+    q = db.query(ModelEndpoint).filter(ModelEndpoint.base_url == base_url)
+    return owner_filter(q, ModelEndpoint, owner).first()
+
+
+def _owned_endpoint_by_id(db, endpoint_id, owner):
+    """ModelEndpoint whose id == `endpoint_id` and is VISIBLE to `owner` (their
+    own rows + legacy null-owner "shared" rows); None otherwise.
+
+    Preferred over _owned_endpoint_by_url for credential resolution: two visible
+    endpoints can share the same base_url but hold DIFFERENT api_keys (e.g. two
+    accounts on the same provider). A base_url-only match returns whichever row
+    sorts first, so it can copy the WRONG owner-scoped key into the [CMP] session.
+    An id pins the exact registered endpoint, so /api/compare/start prefers it and
+    only falls back to URL matching for legacy / admin raw-URL callers. Owner
+    scoping is identical to _owned_endpoint_by_url (a null/empty owner is a no-op).
+    """
+    from core.database import ModelEndpoint
+    from src.auth_helpers import owner_filter
+    q = db.query(ModelEndpoint).filter(ModelEndpoint.id == endpoint_id)
+    return owner_filter(q, ModelEndpoint, owner).first()
+
+
+class RecordVoteRequest(BaseModel):
+    prompt: str
+    models: List[str]
+    winner: str           # model name or "tie"
+    is_blind: bool = True
+
+
+def setup_compare_routes(session_manager: SessionManager):
+    """Setup comparison routes."""
+
+    @router.post("/start")
+    def start_comparison(
+        request: Request,
+        prompt: str = Form(...),
+        model_a: str = Form(...),
+        model_b: str = Form(...),
+        endpoint_a: str = Form(""),
+        endpoint_b: str = Form(""),
+        endpoint_a_id: str = Form(""),
+        endpoint_b_id: str = Form(""),
+        is_blind: str = Form("true"),
+    ):
+        """Create two ephemeral sessions and a comparison record.
+
+        Returns the comparison ID and the two session IDs so the client
+        can fire two independent SSE streams to /api/chat_stream.
+        """
+        user = getattr(request.state, 'current_user', None)
+        comp_id = str(uuid.uuid4())
+        sid_a = str(uuid.uuid4())
+        sid_b = str(uuid.uuid4())
+
+        # Blind mapping: randomly assign left/right
+        blind = str(is_blind).lower() == "true"
+        if blind:
+            mapping = {"left": "a", "right": "b"}
+            if random.random() > 0.5:
+                mapping = {"left": "b", "right": "a"}
+        else:
+            mapping = {"left": "a", "right": "b"}
+
+        # Map session IDs to left/right based on blind mapping
+        session_left = sid_a if mapping["left"] == "a" else sid_b
+        session_right = sid_a if mapping["right"] == "a" else sid_b
+
+        # In blind mode, name the helper sessions by their neutral slot
+        # ("Model A" / "Model B") instead of the real model. Otherwise the
+        # session name leaks the model in the sidebar and GET /api/sessions,
+        # de-anonymizing the comparison before the user votes (issue #1285).
+        slot_name = {session_left: "Model A", session_right: "Model B"}
+
+        # SECURITY: resolve and validate BOTH endpoints before creating any
+        # session. Compare copies a registered endpoint's Authorization header
+        # into the [CMP] session, so validating one endpoint while creating its
+        # session, then rejecting the other, would leave a partial compare
+        # session behind with that header attached. Doing all the owner-scope
+        # resolution + raw-URL rejection up front means a 403 on either endpoint
+        # aborts the whole request with nothing created and no header copied.
+        from src.endpoint_resolver import build_chat_url, build_headers, normalize_base
+        resolved = []
+        db = SessionLocal()
+        try:
+            for sid, model, endpoint, endpoint_id in [
+                (sid_a, model_a, endpoint_a, endpoint_a_id),
+                (sid_b, model_b, endpoint_b, endpoint_b_id),
+            ]:
+                # Prefer an explicit endpoint id: it pins the EXACT registered
+                # endpoint (and its api_key), even when two endpoints visible to
+                # the caller share a base_url with different keys — a URL-only
+                # match would copy whichever row sorts first, i.e. possibly the
+                # wrong key. Fall back to URL resolution only for legacy / admin
+                # raw-URL callers that don't send an id.
+                eid = endpoint_id.strip() if isinstance(endpoint_id, str) else ""
+                if eid:
+                    ep = _owned_endpoint_by_id(db, eid, user)
+                    if ep is None:
+                        # An id the caller can't see (wrong owner / deleted) must
+                        # NOT silently fall back to a same-URL row with a different
+                        # key — that's exactly the mix-up ids exist to prevent.
+                        raise HTTPException(404, "Model endpoint not found")
+                    # The id already resolved the endpoint; ignore any raw URL the
+                    # caller also sent and dial the stored config instead.
+                    endpoint = ep.base_url
+                elif not endpoint:
+                    raise HTTPException(
+                        422, "endpoint_a/endpoint_b or endpoint_a_id/endpoint_b_id is required"
+                    )
+                else:
+                    # Resolve the supplied URL to a ModelEndpoint the caller owns
+                    # (their own rows + legacy null-owner shared rows), scoped so a
+                    # comparison can't borrow another user's private endpoint key.
+                    base = normalize_base(endpoint)
+                    ep = _owned_endpoint_by_url(db, base, user)
+                # Reject *unregistered* raw URLs for signed-in non-admins; a
+                # matched registered endpoint supplies an id so the caller can
+                # still compare endpoints they own. Blanket-rejecting here (the
+                # earlier `endpoint_id=None` call) locked non-admins out of
+                # compare entirely, since compare resolves endpoints by URL with
+                # no endpoint_id. Mirrors the gallery inpaint/harmonize checks.
+                # Raised here (phase 1), before any session exists.
+                _reject_raw_endpoint_url_for_non_admin(
+                    request, user, str(ep.id) if ep is not None else None, endpoint
+                )
+                # Bind the [CMP] session to the RESOLVED endpoint, not the raw
+                # caller-supplied string. When the URL matches a registered
+                # endpoint visible to the caller, use that row's own normalized
+                # base URL (the same value owner scoping + endpoint validation
+                # already vetted) so the session dials exactly where the stored
+                # config points. The raw `endpoint` only survives for callers
+                # allowed to pass one — admins / single-user mode, where
+                # `_reject_raw_endpoint_url_for_non_admin` is a no-op and `ep`
+                # is None. Mirrors the registered-endpoint path in session_routes.
+                session_endpoint_url = (
+                    build_chat_url(normalize_base(ep.base_url)) if ep is not None else endpoint
+                )
+                # Headers come only from a matched endpoint's key; None when
+                # `ep` is None (raw admin URL or no match), so a comparison can
+                # never inherit another user's key/headers.
+                headers = build_headers(ep.api_key, ep.base_url) if (ep and ep.api_key) else None
+                resolved.append((sid, model, session_endpoint_url, headers))
+        finally:
+            db.close()
+
+        # Both endpoints validated — only now create the ephemeral [CMP]
+        # sessions and copy any resolved headers.
+        for sid, model, session_endpoint_url, headers in resolved:
+            name = f"[CMP] {slot_name[sid]}" if blind else f"[CMP] {model.split('/')[-1]}"
+            session_manager.create_session(
+                session_id=sid,
+                name=name,
+                endpoint_url=session_endpoint_url,
+                model=model,
+                rag=False,
+                owner=user,
+            )
+            if headers:
+                s = session_manager.sessions.get(sid)
+                if s:
+                    s.headers = headers
+
+        # Store comparison record
+        db = SessionLocal()
+        try:
+            comp = Comparison(
+                id=comp_id,
+                prompt=prompt,
+                model_a=model_a,
+                model_b=model_b,
+                # Record the URL the session actually dials. For URL callers this
+                # is their raw input; for id-only callers (empty endpoint_a/_b)
+                # fall back to the resolved endpoint URL so the column stays
+                # meaningful and non-null. resolved is in [a, b] order.
+                endpoint_a=endpoint_a or resolved[0][2],
+                endpoint_b=endpoint_b or resolved[1][2],
+                is_blind=blind,
+                blind_mapping=json.dumps(mapping),
+                owner=user,
+            )
+            db.add(comp)
+            db.commit()
+        finally:
+            db.close()
+
+        # In blind mode, withhold the model identities AND the left/right
+        # mapping from the response. The client already knows model_a/model_b
+        # (it sent them), so returning either would defeat blind mode. They are
+        # revealed by POST /api/compare/{id}/vote once the user has voted (#1285).
+        return {
+            "id": comp_id,
+            "session_left": session_left,
+            "session_right": session_right,
+            "model_left": None if blind else (model_a if mapping["left"] == "a" else model_b),
+            "model_right": None if blind else (model_a if mapping["right"] == "a" else model_b),
+            "is_blind": blind,
+            "mapping": None if blind else mapping,
+        }
+
+    @router.post("/{comp_id}/vote")
+    def vote_comparison(
+        request: Request,
+        comp_id: str,
+        winner: str = Form(...),  # "left", "right", or "tie"
+    ):
+        """Record the user's vote and reveal model names if blind."""
+        user = get_current_user(request)
+        db = SessionLocal()
+        try:
+            comp = db.query(Comparison).filter(Comparison.id == comp_id).first()
+            if not comp:
+                raise HTTPException(404, "Comparison not found")
+            # SECURITY: strict ownership — null-owner Comparisons were
+            # accessible to every user.
+            if user and comp.owner != user:
+                raise HTTPException(404, "Comparison not found")
+            if comp.winner:
+                raise HTTPException(400, "Already voted")
+
+            mapping = json.loads(comp.blind_mapping) if comp.blind_mapping else {"left": "a", "right": "b"}
+
+            if winner == "tie":
+                comp.winner = "tie"
+            elif winner == "left":
+                comp.winner = mapping["left"]
+            elif winner == "right":
+                comp.winner = mapping["right"]
+            else:
+                raise HTTPException(400, "winner must be 'left', 'right', or 'tie'")
+
+            comp.voted_at = datetime.utcnow()
+            db.commit()
+
+            return {
+                "winner": comp.winner,
+                "model_a": comp.model_a,
+                "model_b": comp.model_b,
+                "revealed": {
+                    "left": comp.model_a if mapping["left"] == "a" else comp.model_b,
+                    "right": comp.model_a if mapping["right"] == "a" else comp.model_b,
+                },
+            }
+        finally:
+            db.close()
+
+    @router.post("/record")
+    def record_comparison(request: Request, body: RecordVoteRequest):
+        """Lightweight endpoint to record a comparison vote from the frontend."""
+        user = get_current_user(request)
+        comp_id = str(uuid.uuid4())
+
+        model_a = body.models[0] if len(body.models) > 0 else ""
+        model_b = body.models[1] if len(body.models) > 1 else ""
+
+        # For N>2 models, store the full list as JSON in blind_mapping
+        if len(body.models) > 2:
+            blind_mapping = json.dumps({"models": body.models})
+        else:
+            blind_mapping = None
+
+        db = SessionLocal()
+        try:
+            comp = Comparison(
+                id=comp_id,
+                prompt=body.prompt[:500],
+                model_a=model_a,
+                model_b=model_b,
+                endpoint_a="",
+                endpoint_b="",
+                winner=body.winner,
+                is_blind=body.is_blind,
+                blind_mapping=blind_mapping,
+                voted_at=datetime.utcnow(),
+                owner=user,
+            )
+            db.add(comp)
+            db.commit()
+        finally:
+            db.close()
+
+        return {"status": "ok", "id": comp_id}
+
+    @router.get("/history")
+    def list_comparisons(request: Request):
+        """List past comparisons."""
+        user = get_current_user(request)
+        db = SessionLocal()
+        try:
+            q = db.query(Comparison)
+            if user:
+                q = q.filter(Comparison.owner == user)
+            comps = q.order_by(Comparison.created_at.desc()).limit(50).all()
+            return [
+                {
+                    "id": c.id,
+                    "prompt": c.prompt[:100],
+                    "model_a": c.model_a,
+                    "model_b": c.model_b,
+                    "winner": c.winner,
+                    "is_blind": c.is_blind,
+                    "voted_at": c.voted_at.isoformat() if c.voted_at else None,
+                    "created_at": c.created_at.isoformat() if c.created_at else None,
+                }
+                for c in comps
+            ]
+        finally:
+            db.close()
+
+    @router.delete("/{comp_id}")
+    def delete_comparison(request: Request, comp_id: str):
+        """Delete a comparison and its ephemeral sessions."""
+        user = get_current_user(request)
+        db = SessionLocal()
+        try:
+            comp = db.query(Comparison).filter(Comparison.id == comp_id).first()
+            if not comp:
+                raise HTTPException(404, "Comparison not found")
+            # SECURITY: strict ownership — null-owner Comparisons were
+            # accessible to every user.
+            if user and comp.owner != user:
+                raise HTTPException(404, "Comparison not found")
+            db.delete(comp)
+            db.commit()
+            return {"status": "deleted"}
+        finally:
+            db.close()
+
+    return router

routes/compare_routes.py

@@ -1,365 +1,18 @@
-# routes/compare_routes.py
-"""Model A/B comparison routes."""
-import json
-import uuid
-import random
-from datetime import datetime
-from fastapi import APIRouter, Form, HTTPException, Request
-from typing import List
-from pydantic import BaseModel
-import logging
+"""Backward-compat shim — canonical location is routes/compare/compare_routes.py.
 
-from core.database import Comparison, SessionLocal
-from core.session_manager import SessionManager
-from src.auth_helpers import get_current_user
-from routes.session_routes import _reject_raw_endpoint_url_for_non_admin
+This module is replaced in ``sys.modules`` by the canonical module object so
+that ``import routes.compare_routes``, ``from routes.compare_routes import X``,
+``importlib.import_module("routes.compare_routes")``, and the
+``import ... as cr`` + ``monkeypatch.setattr(cr, "SessionLocal", ...)`` /
+``"_owned_endpoint_by_url"`` / ``"_owned_endpoint_by_id"`` pattern used by
+test_endpoint_owner_scope_followup.py all operate on the *same* object the
+application actually uses. Keeps existing import paths working after
+slice 2i (#4082/#4071). Source-introspection tests read the canonical file
+by path.
+"""
 
-logger = logging.getLogger(__name__)
+import sys as _sys
 
-router = APIRouter(prefix="/api/compare", tags=["compare"])
+from routes.compare import compare_routes as _canonical  # noqa: F401
 
-
-def _owned_endpoint_by_url(db, base_url, owner):
-    """ModelEndpoint whose base_url == `base_url` and is VISIBLE to `owner`
-    (their own rows + legacy null-owner "shared" rows); None otherwise.
-
-    Owner-scoped on purpose. ModelEndpoint is per-user (core/database.py: non-null
-    owner = private, "the model picker only shows the endpoint to that user") and
-    holds a decrypted `api_key`. start_comparison copies the matched row's api_key
-    into the caller-owned [CMP] session's headers, which then drives that session's
-    /api/chat_stream calls — so an UNSCOPED base_url match would let a user mint a
-    comparison bound to ANOTHER user's private endpoint and spend that owner's
-    api_key / reach whatever base_url they configured. Mirrors
-    session_routes._owned_endpoint. A null/empty owner is a no-op (single-user /
-    legacy mode).
-    """
-    from core.database import ModelEndpoint
-    from src.auth_helpers import owner_filter
-    q = db.query(ModelEndpoint).filter(ModelEndpoint.base_url == base_url)
-    return owner_filter(q, ModelEndpoint, owner).first()
-
-
-def _owned_endpoint_by_id(db, endpoint_id, owner):
-    """ModelEndpoint whose id == `endpoint_id` and is VISIBLE to `owner` (their
-    own rows + legacy null-owner "shared" rows); None otherwise.
-
-    Preferred over _owned_endpoint_by_url for credential resolution: two visible
-    endpoints can share the same base_url but hold DIFFERENT api_keys (e.g. two
-    accounts on the same provider). A base_url-only match returns whichever row
-    sorts first, so it can copy the WRONG owner-scoped key into the [CMP] session.
-    An id pins the exact registered endpoint, so /api/compare/start prefers it and
-    only falls back to URL matching for legacy / admin raw-URL callers. Owner
-    scoping is identical to _owned_endpoint_by_url (a null/empty owner is a no-op).
-    """
-    from core.database import ModelEndpoint
-    from src.auth_helpers import owner_filter
-    q = db.query(ModelEndpoint).filter(ModelEndpoint.id == endpoint_id)
-    return owner_filter(q, ModelEndpoint, owner).first()
-
-
-class RecordVoteRequest(BaseModel):
-    prompt: str
-    models: List[str]
-    winner: str           # model name or "tie"
-    is_blind: bool = True
-
-
-def setup_compare_routes(session_manager: SessionManager):
-    """Setup comparison routes."""
-
-    @router.post("/start")
-    def start_comparison(
-        request: Request,
-        prompt: str = Form(...),
-        model_a: str = Form(...),
-        model_b: str = Form(...),
-        endpoint_a: str = Form(""),
-        endpoint_b: str = Form(""),
-        endpoint_a_id: str = Form(""),
-        endpoint_b_id: str = Form(""),
-        is_blind: str = Form("true"),
-    ):
-        """Create two ephemeral sessions and a comparison record.
-
-        Returns the comparison ID and the two session IDs so the client
-        can fire two independent SSE streams to /api/chat_stream.
-        """
-        user = getattr(request.state, 'current_user', None)
-        comp_id = str(uuid.uuid4())
-        sid_a = str(uuid.uuid4())
-        sid_b = str(uuid.uuid4())
-
-        # Blind mapping: randomly assign left/right
-        blind = str(is_blind).lower() == "true"
-        if blind:
-            mapping = {"left": "a", "right": "b"}
-            if random.random() > 0.5:
-                mapping = {"left": "b", "right": "a"}
-        else:
-            mapping = {"left": "a", "right": "b"}
-
-        # Map session IDs to left/right based on blind mapping
-        session_left = sid_a if mapping["left"] == "a" else sid_b
-        session_right = sid_a if mapping["right"] == "a" else sid_b
-
-        # In blind mode, name the helper sessions by their neutral slot
-        # ("Model A" / "Model B") instead of the real model. Otherwise the
-        # session name leaks the model in the sidebar and GET /api/sessions,
-        # de-anonymizing the comparison before the user votes (issue #1285).
-        slot_name = {session_left: "Model A", session_right: "Model B"}
-
-        # SECURITY: resolve and validate BOTH endpoints before creating any
-        # session. Compare copies a registered endpoint's Authorization header
-        # into the [CMP] session, so validating one endpoint while creating its
-        # session, then rejecting the other, would leave a partial compare
-        # session behind with that header attached. Doing all the owner-scope
-        # resolution + raw-URL rejection up front means a 403 on either endpoint
-        # aborts the whole request with nothing created and no header copied.
-        from src.endpoint_resolver import build_chat_url, build_headers, normalize_base
-        resolved = []
-        db = SessionLocal()
-        try:
-            for sid, model, endpoint, endpoint_id in [
-                (sid_a, model_a, endpoint_a, endpoint_a_id),
-                (sid_b, model_b, endpoint_b, endpoint_b_id),
-            ]:
-                # Prefer an explicit endpoint id: it pins the EXACT registered
-                # endpoint (and its api_key), even when two endpoints visible to
-                # the caller share a base_url with different keys — a URL-only
-                # match would copy whichever row sorts first, i.e. possibly the
-                # wrong key. Fall back to URL resolution only for legacy / admin
-                # raw-URL callers that don't send an id.
-                eid = endpoint_id.strip() if isinstance(endpoint_id, str) else ""
-                if eid:
-                    ep = _owned_endpoint_by_id(db, eid, user)
-                    if ep is None:
-                        # An id the caller can't see (wrong owner / deleted) must
-                        # NOT silently fall back to a same-URL row with a different
-                        # key — that's exactly the mix-up ids exist to prevent.
-                        raise HTTPException(404, "Model endpoint not found")
-                    # The id already resolved the endpoint; ignore any raw URL the
-                    # caller also sent and dial the stored config instead.
-                    endpoint = ep.base_url
-                elif not endpoint:
-                    raise HTTPException(
-                        422, "endpoint_a/endpoint_b or endpoint_a_id/endpoint_b_id is required"
-                    )
-                else:
-                    # Resolve the supplied URL to a ModelEndpoint the caller owns
-                    # (their own rows + legacy null-owner shared rows), scoped so a
-                    # comparison can't borrow another user's private endpoint key.
-                    base = normalize_base(endpoint)
-                    ep = _owned_endpoint_by_url(db, base, user)
-                # Reject *unregistered* raw URLs for signed-in non-admins; a
-                # matched registered endpoint supplies an id so the caller can
-                # still compare endpoints they own. Blanket-rejecting here (the
-                # earlier `endpoint_id=None` call) locked non-admins out of
-                # compare entirely, since compare resolves endpoints by URL with
-                # no endpoint_id. Mirrors the gallery inpaint/harmonize checks.
-                # Raised here (phase 1), before any session exists.
-                _reject_raw_endpoint_url_for_non_admin(
-                    request, user, str(ep.id) if ep is not None else None, endpoint
-                )
-                # Bind the [CMP] session to the RESOLVED endpoint, not the raw
-                # caller-supplied string. When the URL matches a registered
-                # endpoint visible to the caller, use that row's own normalized
-                # base URL (the same value owner scoping + endpoint validation
-                # already vetted) so the session dials exactly where the stored
-                # config points. The raw `endpoint` only survives for callers
-                # allowed to pass one — admins / single-user mode, where
-                # `_reject_raw_endpoint_url_for_non_admin` is a no-op and `ep`
-                # is None. Mirrors the registered-endpoint path in session_routes.
-                session_endpoint_url = (
-                    build_chat_url(normalize_base(ep.base_url)) if ep is not None else endpoint
-                )
-                # Headers come only from a matched endpoint's key; None when
-                # `ep` is None (raw admin URL or no match), so a comparison can
-                # never inherit another user's key/headers.
-                headers = build_headers(ep.api_key, ep.base_url) if (ep and ep.api_key) else None
-                resolved.append((sid, model, session_endpoint_url, headers))
-        finally:
-            db.close()
-
-        # Both endpoints validated — only now create the ephemeral [CMP]
-        # sessions and copy any resolved headers.
-        for sid, model, session_endpoint_url, headers in resolved:
-            name = f"[CMP] {slot_name[sid]}" if blind else f"[CMP] {model.split('/')[-1]}"
-            session_manager.create_session(
-                session_id=sid,
-                name=name,
-                endpoint_url=session_endpoint_url,
-                model=model,
-                rag=False,
-                owner=user,
-            )
-            if headers:
-                s = session_manager.sessions.get(sid)
-                if s:
-                    s.headers = headers
-
-        # Store comparison record
-        db = SessionLocal()
-        try:
-            comp = Comparison(
-                id=comp_id,
-                prompt=prompt,
-                model_a=model_a,
-                model_b=model_b,
-                # Record the URL the session actually dials. For URL callers this
-                # is their raw input; for id-only callers (empty endpoint_a/_b)
-                # fall back to the resolved endpoint URL so the column stays
-                # meaningful and non-null. resolved is in [a, b] order.
-                endpoint_a=endpoint_a or resolved[0][2],
-                endpoint_b=endpoint_b or resolved[1][2],
-                is_blind=blind,
-                blind_mapping=json.dumps(mapping),
-                owner=user,
-            )
-            db.add(comp)
-            db.commit()
-        finally:
-            db.close()
-
-        # In blind mode, withhold the model identities AND the left/right
-        # mapping from the response. The client already knows model_a/model_b
-        # (it sent them), so returning either would defeat blind mode. They are
-        # revealed by POST /api/compare/{id}/vote once the user has voted (#1285).
-        return {
-            "id": comp_id,
-            "session_left": session_left,
-            "session_right": session_right,
-            "model_left": None if blind else (model_a if mapping["left"] == "a" else model_b),
-            "model_right": None if blind else (model_a if mapping["right"] == "a" else model_b),
-            "is_blind": blind,
-            "mapping": None if blind else mapping,
-        }
-
-    @router.post("/{comp_id}/vote")
-    def vote_comparison(
-        request: Request,
-        comp_id: str,
-        winner: str = Form(...),  # "left", "right", or "tie"
-    ):
-        """Record the user's vote and reveal model names if blind."""
-        user = get_current_user(request)
-        db = SessionLocal()
-        try:
-            comp = db.query(Comparison).filter(Comparison.id == comp_id).first()
-            if not comp:
-                raise HTTPException(404, "Comparison not found")
-            # SECURITY: strict ownership — null-owner Comparisons were
-            # accessible to every user.
-            if user and comp.owner != user:
-                raise HTTPException(404, "Comparison not found")
-            if comp.winner:
-                raise HTTPException(400, "Already voted")
-
-            mapping = json.loads(comp.blind_mapping) if comp.blind_mapping else {"left": "a", "right": "b"}
-
-            if winner == "tie":
-                comp.winner = "tie"
-            elif winner == "left":
-                comp.winner = mapping["left"]
-            elif winner == "right":
-                comp.winner = mapping["right"]
-            else:
-                raise HTTPException(400, "winner must be 'left', 'right', or 'tie'")
-
-            comp.voted_at = datetime.utcnow()
-            db.commit()
-
-            return {
-                "winner": comp.winner,
-                "model_a": comp.model_a,
-                "model_b": comp.model_b,
-                "revealed": {
-                    "left": comp.model_a if mapping["left"] == "a" else comp.model_b,
-                    "right": comp.model_a if mapping["right"] == "a" else comp.model_b,
-                },
-            }
-        finally:
-            db.close()
-
-    @router.post("/record")
-    def record_comparison(request: Request, body: RecordVoteRequest):
-        """Lightweight endpoint to record a comparison vote from the frontend."""
-        user = get_current_user(request)
-        comp_id = str(uuid.uuid4())
-
-        model_a = body.models[0] if len(body.models) > 0 else ""
-        model_b = body.models[1] if len(body.models) > 1 else ""
-
-        # For N>2 models, store the full list as JSON in blind_mapping
-        if len(body.models) > 2:
-            blind_mapping = json.dumps({"models": body.models})
-        else:
-            blind_mapping = None
-
-        db = SessionLocal()
-        try:
-            comp = Comparison(
-                id=comp_id,
-                prompt=body.prompt[:500],
-                model_a=model_a,
-                model_b=model_b,
-                endpoint_a="",
-                endpoint_b="",
-                winner=body.winner,
-                is_blind=body.is_blind,
-                blind_mapping=blind_mapping,
-                voted_at=datetime.utcnow(),
-                owner=user,
-            )
-            db.add(comp)
-            db.commit()
-        finally:
-            db.close()
-
-        return {"status": "ok", "id": comp_id}
-
-    @router.get("/history")
-    def list_comparisons(request: Request):
-        """List past comparisons."""
-        user = get_current_user(request)
-        db = SessionLocal()
-        try:
-            q = db.query(Comparison)
-            if user:
-                q = q.filter(Comparison.owner == user)
-            comps = q.order_by(Comparison.created_at.desc()).limit(50).all()
-            return [
-                {
-                    "id": c.id,
-                    "prompt": c.prompt[:100],
-                    "model_a": c.model_a,
-                    "model_b": c.model_b,
-                    "winner": c.winner,
-                    "is_blind": c.is_blind,
-                    "voted_at": c.voted_at.isoformat() if c.voted_at else None,
-                    "created_at": c.created_at.isoformat() if c.created_at else None,
-                }
-                for c in comps
-            ]
-        finally:
-            db.close()
-
-    @router.delete("/{comp_id}")
-    def delete_comparison(request: Request, comp_id: str):
-        """Delete a comparison and its ephemeral sessions."""
-        user = get_current_user(request)
-        db = SessionLocal()
-        try:
-            comp = db.query(Comparison).filter(Comparison.id == comp_id).first()
-            if not comp:
-                raise HTTPException(404, "Comparison not found")
-            # SECURITY: strict ownership — null-owner Comparisons were
-            # accessible to every user.
-            if user and comp.owner != user:
-                raise HTTPException(404, "Comparison not found")
-            db.delete(comp)
-            db.commit()
-            return {"status": "deleted"}
-        finally:
-            db.close()
-
-    return router
+_sys.modules[__name__] = _canonical

routes/cookbook_helpers.py

@@ -463,14 +463,22 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
         "        if sz == 0 and os.path.isdir(snap):",
         "            sz2, nf2, ic2 = snapshot_size()",
         "            sz, nf, ic = sz2, nf2, ic or ic2",
-        "        is_diffusion = False; gguf_files = []",
+        "        is_video = bool(re.search(r'(?i)(^|/)Lightricks/LTX-|(^|/)LTX[-_/]|video|text-to-video|image-to-video', rid))",
+        "        is_diffusion = is_video; is_adapter = bool(re.search(r'(?i)(lora|adapter|peft|qlora|control[-_]?lora|diffusion[-_]?lora)', rid)); gguf_files = []",
         "        if os.path.isdir(snap):",
         "            for sd in os.listdir(snap):",
         "                sf = os.path.join(snap, sd)",
         "                if not os.path.isdir(sf): continue",
         "                if os.path.exists(os.path.join(sf, 'model_index.json')): is_diffusion = True",
+        "                if os.path.exists(os.path.join(sf, 'adapter_config.json')) or os.path.exists(os.path.join(sf, 'adapter_model.safetensors')): is_adapter = True",
+        "                for _root, _dirs, _fns in safe_walk(sf):",
+        "                    for _fn in _fns:",
+        "                        _lfn = _fn.lower()",
+        "                        if _lfn.endswith('.safetensors') and re.search(r'(?i)(ltx|video|upscaler)', _lfn): is_video = True; is_diffusion = True",
+        "                        if _lfn in ('adapter_config.json','adapter_model.safetensors','pytorch_lora_weights.safetensors') or 'lora' in _lfn:",
+        "                            is_adapter = True",
         "                for f in collect_ggufs(sf): f['rel_path'] = sd + '/' + f['rel_path']; gguf_files.append(f)",
-        "        models.append({'repo_id':rid,'size_bytes':sz,'nb_files':nf,'has_incomplete':ic,'path':cache,'is_diffusion':is_diffusion,'is_gguf':bool(gguf_files),'gguf_files':gguf_files})",
+        "        models.append({'repo_id':rid,'size_bytes':sz,'nb_files':nf,'has_incomplete':ic,'path':cache,'is_diffusion':is_diffusion,'is_video':is_video,'is_adapter':is_adapter,'is_gguf':bool(gguf_files),'gguf_files':gguf_files})",
         "def hf_cache_paths():",
         "    candidates = []",
         "    def add(p):",
@@ -505,11 +513,12 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
         "        fp = os.path.join(p, d)",
         "        if not os.path.isdir(fp) or os.path.islink(fp) or not safe_path(fp): continue",
         "        if d in seen: continue",
-        "        is_model = False; gguf_files = []",
+        "        is_model = False; is_adapter = bool(re.search(r'(?i)(lora|adapter|peft|qlora|control[-_]?lora|diffusion[-_]?lora)', d)); gguf_files = []",
         "        for root, dirs, fns in safe_walk(fp):",
         "            for fn in fns:",
         "                if fn.lower().endswith('.gguf'): is_model = True",
         "                elif fn == 'config.json' or fn.endswith('.safetensors') or fn.endswith('.bin'): is_model = True",
+        "                if fn in ('adapter_config.json','adapter_model.safetensors','pytorch_lora_weights.safetensors') or 'lora' in fn.lower(): is_adapter = True",
         "            if is_model: break",
         "        if not is_model: continue",
         "        gguf_files = collect_ggufs(fp)",
@@ -520,7 +529,7 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
         "                try: nf += 1; sz += os.path.getsize(os.path.join(dp, fn))",
         "                except Exception: pass",
         "        is_diff = os.path.exists(os.path.join(fp, 'model_index.json'))",
-        "        models.append({'repo_id':d,'size_bytes':sz,'nb_files':nf,'has_incomplete':False,'path':p,'is_local_dir':True,'is_diffusion':is_diff,'is_gguf':bool(gguf_files),'gguf_files':gguf_files})",
+        "        models.append({'repo_id':d,'size_bytes':sz,'nb_files':nf,'has_incomplete':False,'path':p,'is_local_dir':True,'is_diffusion':is_diff,'is_adapter':is_adapter,'is_gguf':bool(gguf_files),'gguf_files':gguf_files})",
         "def parse_size(num, unit):",
         "    try: n = float(num)",
         "    except Exception: return 0",
@@ -1320,6 +1329,26 @@ def _diagnose_serve_output(text: str) -> dict | None:
             "MLX LM is not installed on this server.",
             [{"label": "install mlx-lm in Cookbook Dependencies", "op": "dependency", "package": "mlx-lm"}],
         ),
+        (
+            r"OmniGen2Pipeline|module diffusers has no attribute .*Pipeline|custom_pipeline=.*failed",
+            "This image model uses a custom Diffusers pipeline that the launch environment does not know yet.",
+            [{"label": "update Diffusers image dependencies", "op": "dependency", "package": "diffusers transformers accelerate"}],
+        ),
+        (
+            r"mflux-generate-qwen.*not found|mflux-generate.*not found|MLX image serving requires mflux|No module named ['\"]?mflux",
+            "MLX image serving requires mflux on this Apple Silicon server.",
+            [{"label": "install mflux in Cookbook Dependencies", "op": "dependency", "package": "mflux"}],
+        ),
+        (
+            r"mlx-lama-swift|odysseus-mlx-inpaint|mlx-lama-serve|LaMa / MI-GAN MLX inpainting models require",
+            "LaMa / MI-GAN MLX inpainting requires an Odysseus-compatible mlx-lama-swift bridge on this Apple Silicon server.",
+            [{"label": "build mlx-lama-swift bridge and put odysseus-mlx-inpaint or mlx-lama-serve on PATH", "op": "dependency", "package": "mlx_lama_swift"}],
+        ),
+        (
+            r"mlx-ddcolor-swift|odysseus-mlx-colorize|mlx-ddcolor-serve|DDColor MLX models require",
+            "DDColor MLX colorization requires an Odysseus-compatible mlx-ddcolor-swift bridge on this Apple Silicon server.",
+            [{"label": "build mlx-ddcolor-swift bridge and put odysseus-mlx-colorize or mlx-ddcolor-serve on PATH", "op": "dependency", "package": "mlx_ddcolor_swift"}],
+        ),
         (
             r"Unable to quantize model of type <class ['\"]mlx_lm\.models\.switch_layers\.QuantizedSwitchLinear['\"]>|QuantizedSwitchLinear",
             "MLX-LM tried to quantize an already-quantized DeepSeek switch layer.",
@@ -1358,9 +1387,9 @@ def _diagnose_serve_output(text: str) -> dict | None:
             [{"label": "download a GGUF build of this model (repo name usually ends in -GGUF, file like Q4_K_M.gguf)", "op": "manual"}],
         ),
         (
-            r"No module named 'torch'|No module named torch|No module named 'diffusers'|No module named diffusers",
-            "Diffusion serving requires PyTorch and diffusers.",
-            [{"label": "install diffusers[torch] in Cookbook Dependencies", "op": "dependency", "package": "diffusers[torch]"}],
+            r"No module named 'torch'|No module named torch|No module named 'torchvision'|No module named torchvision|No module named 'diffusers'|No module named diffusers|No module named 'scipy'|No module named scipy|install scipy if you want to use beta sigmas|requires the Torchvision library",
+            "Diffusion serving requires PyTorch, Torchvision, Diffusers, Accelerate, and SciPy.",
+            [{"label": "install Diffusers image deps in Cookbook Dependencies", "op": "dependency", "package": "diffusers[torch] torchvision accelerate scipy python-multipart"}],
         ),
         (
             r"403 Forbidden|401 Unauthorized|Access to model.*is restricted|gated repo|not in the authorized list|awaiting a review",

routes/cookbook_routes.py

@@ -73,6 +73,23 @@ _HF_TOKEN_STATUS_SNIPPET = (
 )
 
 
+def _append_mlx_image_server_script(runner_lines: list[str]) -> None:
+    """Write the MLX image API helper next to the tmux runner on remote hosts."""
+    script_path = Path(__file__).resolve().parents[1] / "scripts" / "mlx_image_server.py"
+    try:
+        script = script_path.read_text(encoding="utf-8")
+    except Exception as e:
+        logger.warning("Failed to read mlx_image_server.py: %s", e)
+        runner_lines.append('echo "ERROR: Odysseus could not prepare the MLX image server helper."')
+        runner_lines.append('ODYSSEUS_PREFLIGHT_EXIT=127')
+        return
+    runner_lines.append('mkdir -p scripts')
+    runner_lines.append("cat > scripts/mlx_image_server.py <<'PY'")
+    runner_lines.extend(script.splitlines())
+    runner_lines.append("PY")
+    runner_lines.append('chmod +x scripts/mlx_image_server.py 2>/dev/null || true')
+
+
 def _venv_root_from_serve_cmd(cmd: str) -> str:
     """Best-effort venv root from an absolute venv python in a serve command."""
     try:
@@ -492,6 +509,11 @@ def setup_cookbook_routes() -> APIRouter:
                 "MLX LM is not installed on this server.",
                 [{"label": "install mlx-lm in Cookbook Dependencies", "op": "dependency", "package": "mlx-lm"}],
             ),
+            (
+                r"OmniGen2Pipeline|module diffusers has no attribute .*Pipeline|custom_pipeline=.*failed",
+                "This image model uses a custom Diffusers pipeline that the launch environment does not know yet.",
+                [{"label": "update Diffusers image dependencies", "op": "dependency", "package": "diffusers transformers accelerate"}],
+            ),
             (
                 r"Unable to quantize model of type <class ['\"]mlx_lm\.models\.switch_layers\.QuantizedSwitchLinear['\"]>|QuantizedSwitchLinear",
                 "MLX-LM tried to quantize an already-quantized DeepSeek switch layer.",
@@ -530,9 +552,9 @@ def setup_cookbook_routes() -> APIRouter:
                 [{"label": "download a GGUF build of this model (repo name usually ends in -GGUF, file like Q4_K_M.gguf)", "op": "manual"}],
             ),
             (
-                r"No module named 'torch'|No module named torch|No module named 'diffusers'|No module named diffusers",
-                "Diffusion serving requires PyTorch and diffusers.",
-                [{"label": "install diffusers[torch] in Cookbook Dependencies", "op": "dependency", "package": "diffusers[torch]"}],
+                r"No module named 'torch'|No module named torch|No module named 'torchvision'|No module named torchvision|No module named 'diffusers'|No module named diffusers|No module named 'scipy'|No module named scipy|install scipy if you want to use beta sigmas|requires the Torchvision library",
+                "Diffusion serving requires PyTorch, Torchvision, Diffusers, Accelerate, and SciPy.",
+                [{"label": "install Diffusers image deps in Cookbook Dependencies", "op": "dependency", "package": "diffusers[torch] torchvision accelerate scipy python-multipart"}],
             ),
             (
                 r"403 Forbidden|401 Unauthorized|Access to model.*is restricted|gated repo|not in the authorized list|awaiting a review",
@@ -1433,9 +1455,11 @@ def setup_cookbook_routes() -> APIRouter:
                     "nb_files": m["nb_files"],
                     "has_incomplete": m["has_incomplete"],
                     "status": "downloading" if m["has_incomplete"] else "ready",
-                    "path": m.get("path", ""),
-                    "is_diffusion": m.get("is_diffusion", False),
-                }
+	                    "path": m.get("path", ""),
+	                    "is_diffusion": m.get("is_diffusion", False),
+	                    "is_video": m.get("is_video", False),
+	                    "is_adapter": m.get("is_adapter", False),
+	                }
                 if m.get("is_local_dir"):
                     entry["is_local_dir"] = True
                 if m.get("is_gguf"):
@@ -1460,6 +1484,7 @@ def setup_cookbook_routes() -> APIRouter:
         """Register a diffusion model as an image endpoint so it appears in the model selector."""
         import re
         from core.database import SessionLocal, ModelEndpoint
+        from src.settings import load_settings, save_settings
 
         # Parse port from command (--port NNNN), default 8100 for diffusion_server
         port_match = re.search(r'--port\s+(\d+)', req.cmd)
@@ -1477,6 +1502,7 @@ def setup_cookbook_routes() -> APIRouter:
         # Friendly display name from repo_id
         short_name = req.repo_id.split("/")[-1] if "/" in req.repo_id else req.repo_id
         display_name = f"{short_name} (image)"
+        pinned_models = [req.repo_id] if req.repo_id else []
 
         db = SessionLocal()
         try:
@@ -1486,7 +1512,16 @@ def setup_cookbook_routes() -> APIRouter:
                 existing.is_enabled = True
                 existing.model_type = "image"
                 existing.name = display_name
+                existing.endpoint_kind = "local"
+                existing.model_refresh_mode = "manual"
+                if pinned_models:
+                    existing.cached_models = json.dumps(pinned_models)
+                    existing.pinned_models = json.dumps(pinned_models)
                 db.commit()
+                settings = load_settings()
+                if settings.get("image_gen_enabled") is not True:
+                    settings["image_gen_enabled"] = True
+                    save_settings(settings)
                 logger.info(f"Updated existing image endpoint: {base_url}")
                 return existing.id
 
@@ -1498,9 +1533,18 @@ def setup_cookbook_routes() -> APIRouter:
                 api_key=None,
                 is_enabled=True,
                 model_type="image",
+                endpoint_kind="local",
+                model_refresh_mode="manual",
+                cached_models=json.dumps(pinned_models) if pinned_models else None,
+                pinned_models=json.dumps(pinned_models) if pinned_models else None,
             )
             db.add(ep)
             db.commit()
+            settings = load_settings()
+            settings["image_gen_enabled"] = True
+            if not settings.get("image_model"):
+                settings["image_model"] = req.repo_id
+            save_settings(settings)
             logger.info(f"Auto-registered image endpoint: {display_name} @ {base_url}")
             return ep_id
         except Exception as e:
@@ -2356,24 +2400,8 @@ def setup_cookbook_routes() -> APIRouter:
                 runner_lines.append('fi')
             elif "sglang.launch_server" in req.cmd:
                 runner_lines.append('export PATH="$HOME/.local/bin:$PATH"')
-                runner_lines.append('if ! command -v sglang &>/dev/null; then')
-                runner_lines.append('  echo "ERROR: SGLang is not installed."')
-                runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
-                runner_lines.append('elif ! ODYSSEUS_SGLANG_IMPORT_ERROR="$(python3 -c "import sglang" 2>&1)"; then')
-                runner_lines.append('  echo "ERROR: SGLang is installed but failed to import."')
-                runner_lines.append('  printf "%s\\n" "$ODYSSEUS_SGLANG_IMPORT_ERROR"')
-                runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
-                runner_lines.append('fi')
-            elif "mlx_lm.server" in req.cmd:
-                runner_lines.append('export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"')
-                runner_lines.append('if ! ODYSSEUS_MLX_IMPORT_ERROR="$(python3 -c "import mlx_lm" 2>&1)"; then')
-                runner_lines.append('  echo "ERROR: MLX LM is not installed."')
-                runner_lines.append('  printf "%s\\n" "$ODYSSEUS_MLX_IMPORT_ERROR"')
-                runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
-                runner_lines.append('fi')
                 runner_lines.append(f"ODYSSEUS_SERVE_CMD='{_bash_squote(req.cmd)}'")
-                runner_lines.append('if [ -z "$ODYSSEUS_PREFLIGHT_EXIT" ]; then')
-                runner_lines.append('  ODYSSEUS_MLX_CMD_PY="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
+                runner_lines.append('ODYSSEUS_SGLANG_CMD_PY="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
                 runner_lines.append('import shlex, sys')
                 runner_lines.append('parts = shlex.split(sys.argv[1])')
                 runner_lines.append('py = "python3"')
@@ -2384,6 +2412,36 @@ def setup_cookbook_routes() -> APIRouter:
                 runner_lines.append('print(py)')
                 runner_lines.append('PY')
                 runner_lines.append(')"')
+                runner_lines.append('if ! "$ODYSSEUS_SGLANG_CMD_PY" -c "import sglang" &>/dev/null; then')
+                runner_lines.append('  if ! command -v sglang &>/dev/null; then')
+                runner_lines.append('    echo "ERROR: SGLang is not installed."')
+                runner_lines.append('  else')
+                runner_lines.append('    echo "ERROR: SGLang is installed but failed to import in the launch Python."')
+                runner_lines.append('  fi')
+                runner_lines.append('  ODYSSEUS_SGLANG_IMPORT_ERROR="$("$ODYSSEUS_SGLANG_CMD_PY" -c "import sglang" 2>&1)"')
+                runner_lines.append('  printf "%s\\n" "$ODYSSEUS_SGLANG_IMPORT_ERROR"')
+                runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('fi')
+            elif "mlx_lm.server" in req.cmd:
+                runner_lines.append('export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"')
+                runner_lines.append(f"ODYSSEUS_SERVE_CMD='{_bash_squote(req.cmd)}'")
+                runner_lines.append('ODYSSEUS_MLX_CMD_PY="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
+                runner_lines.append('import shlex, sys')
+                runner_lines.append('parts = shlex.split(sys.argv[1])')
+                runner_lines.append('py = "python3"')
+                runner_lines.append('for i, part in enumerate(parts):')
+                runner_lines.append('    if part.endswith("/bin/python") or part.endswith("/bin/python3") or "/bin/python3." in part:')
+                runner_lines.append('        py = part')
+                runner_lines.append('        break')
+                runner_lines.append('print(py)')
+                runner_lines.append('PY')
+                runner_lines.append(')"')
+                runner_lines.append('if ! ODYSSEUS_MLX_IMPORT_ERROR="$("$ODYSSEUS_MLX_CMD_PY" -c "import mlx_lm" 2>&1)"; then')
+                runner_lines.append('  echo "ERROR: MLX LM is not installed in the launch Python: $ODYSSEUS_MLX_CMD_PY"')
+                runner_lines.append('  printf "%s\\n" "$ODYSSEUS_MLX_IMPORT_ERROR"')
+                runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('fi')
+                runner_lines.append('if [ -z "$ODYSSEUS_PREFLIGHT_EXIT" ]; then')
                 runner_lines.append('  ODYSSEUS_SERVE_CMD="$("$ODYSSEUS_MLX_CMD_PY" - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
                 runner_lines.append('import json, os, shlex, sys')
                 runner_lines.append('from pathlib import Path')
@@ -2474,10 +2532,111 @@ def setup_cookbook_routes() -> APIRouter:
                 runner_lines.append('PY')
                 runner_lines.append(')"')
                 runner_lines.append('fi')
+            elif "scripts/mlx_image_server.py" in req.cmd or ".mlx_image_server.py" in req.cmd:
+                _append_mlx_image_server_script(runner_lines)
+                runner_lines.append('export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"')
+                runner_lines.append(f"ODYSSEUS_SERVE_CMD='{_bash_squote(req.cmd)}'")
+                runner_lines.append('ODYSSEUS_MLX_IMAGE_CMD_PY="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
+                runner_lines.append('import shlex, sys')
+                runner_lines.append('parts = shlex.split(sys.argv[1])')
+                runner_lines.append('py = "python3"')
+                runner_lines.append('for part in parts:')
+                runner_lines.append('    if part.endswith("/bin/python") or part.endswith("/bin/python3") or "/bin/python3." in part:')
+                runner_lines.append('        py = part')
+                runner_lines.append('        break')
+                runner_lines.append('print(py)')
+                runner_lines.append('PY')
+                runner_lines.append(')"')
+                runner_lines.append('ODYSSEUS_MLX_IMAGE_BIN_DIR="$(dirname "$ODYSSEUS_MLX_IMAGE_CMD_PY" 2>/dev/null || true)"')
+                runner_lines.append('if [ -n "$ODYSSEUS_MLX_IMAGE_BIN_DIR" ]; then export PATH="$ODYSSEUS_MLX_IMAGE_BIN_DIR:$PATH"; fi')
+                runner_lines.append('if ! "$ODYSSEUS_MLX_IMAGE_CMD_PY" -c "import fastapi, uvicorn, multipart" >/dev/null 2>&1; then')
+                runner_lines.append('  echo "ERROR: MLX image serving requires FastAPI + uvicorn + python-multipart in the launch Python: $ODYSSEUS_MLX_IMAGE_CMD_PY. Install the MLX image dependencies in Cookbook Dependencies."')
+                runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('fi')
+                runner_lines.append('ODYSSEUS_MLX_IMAGE_MODEL="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
+                runner_lines.append('import shlex, sys')
+                runner_lines.append('parts = shlex.split(sys.argv[1])')
+                runner_lines.append('model = ""')
+                runner_lines.append('for i, part in enumerate(parts):')
+                runner_lines.append('    if part == "--model" and i + 1 < len(parts):')
+                runner_lines.append('        model = parts[i + 1]')
+                runner_lines.append('        break')
+                runner_lines.append('print(model)')
+                runner_lines.append('PY')
+                runner_lines.append(')"')
+                runner_lines.append('if printf "%s" "$ODYSSEUS_MLX_IMAGE_MODEL" | grep -qi hidream; then')
+                runner_lines.append('  if ! "$ODYSSEUS_MLX_IMAGE_CMD_PY" -c "import mlx, mlx_vlm, transformers, huggingface_hub, safetensors, numpy, PIL" >/dev/null 2>&1; then')
+                runner_lines.append('    echo "ERROR: HiDream MLX serving needs the model requirements in the launch Python: $ODYSSEUS_MLX_IMAGE_CMD_PY."')
+                runner_lines.append('    echo "Install with: $ODYSSEUS_MLX_IMAGE_CMD_PY -m pip install -U fastapi uvicorn python-multipart mlx mlx-vlm \'transformers>=4.57.0,<6.0\' huggingface_hub safetensors numpy pillow tqdm sentencepiece hf_transfer"')
+                runner_lines.append('    ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('  fi')
+                runner_lines.append('elif printf "%s" "$ODYSSEUS_MLX_IMAGE_MODEL" | grep -qi boogu; then')
+                runner_lines.append('  if ! "$ODYSSEUS_MLX_IMAGE_CMD_PY" -c "import boogu_image_mlx, mlx, huggingface_hub, safetensors, numpy, PIL" >/dev/null 2>&1; then')
+                runner_lines.append('    echo "ERROR: Boogu MLX serving needs boogu-image-mlx in the launch Python: $ODYSSEUS_MLX_IMAGE_CMD_PY."')
+                runner_lines.append('    echo "Install with: $ODYSSEUS_MLX_IMAGE_CMD_PY -m pip install -U git+https://github.com/xocialize/boogu-image-mlx.git fastapi uvicorn python-multipart pillow"')
+                runner_lines.append('    ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('  fi')
+                runner_lines.append('elif printf "%s" "$ODYSSEUS_MLX_IMAGE_MODEL" | grep -Eqi "ddcolor"; then')
+                runner_lines.append('  if ! "$ODYSSEUS_MLX_IMAGE_CMD_PY" -c "import PIL" >/dev/null 2>&1; then')
+                runner_lines.append('    echo "ERROR: DDColor MLX serving needs Pillow in the launch Python: $ODYSSEUS_MLX_IMAGE_CMD_PY."')
+                runner_lines.append('    echo "Install with: $ODYSSEUS_MLX_IMAGE_CMD_PY -m pip install -U fastapi uvicorn python-multipart pillow huggingface_hub"')
+                runner_lines.append('    ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('  fi')
+                runner_lines.append('  if ! command -v odysseus-mlx-colorize >/dev/null 2>&1 && ! command -v mlx-ddcolor-serve >/dev/null 2>&1; then')
+                runner_lines.append('    echo "ERROR: DDColor MLX serving requires the Odysseus mlx-ddcolor-swift bridge on PATH: odysseus-mlx-colorize or mlx-ddcolor-serve."')
+                runner_lines.append('    echo "Build it from swift/odysseus-mlx-image-bridge in Cookbook Dependencies."')
+                runner_lines.append('    ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('  fi')
+                runner_lines.append('  ODYSSEUS_DDCOLOR_BIN="$(command -v odysseus-mlx-colorize 2>/dev/null || command -v mlx-ddcolor-serve 2>/dev/null || true)"')
+                runner_lines.append('  if [ -n "$ODYSSEUS_DDCOLOR_BIN" ]; then')
+                runner_lines.append('    ODYSSEUS_DDCOLOR_DIR="$(dirname "$ODYSSEUS_DDCOLOR_BIN")"')
+                runner_lines.append('    if [ ! -f "$ODYSSEUS_DDCOLOR_DIR/mlx.metallib" ] && [ ! -f "$ODYSSEUS_DDCOLOR_DIR/default.metallib" ]; then')
+                runner_lines.append('      echo "ERROR: DDColor MLX serving found the Swift runner, but mlx.metallib/default.metallib is missing next to it."')
+                runner_lines.append('      echo "Run the DDColor MLX image editing dependency install again; it copies mlx.metallib from the launch Python MLX package."')
+                runner_lines.append('      ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('    fi')
+                runner_lines.append('  fi')
+                runner_lines.append('elif printf "%s" "$ODYSSEUS_MLX_IMAGE_MODEL" | grep -Eqi "mi-gan|migan|lama"; then')
+                runner_lines.append('  if ! "$ODYSSEUS_MLX_IMAGE_CMD_PY" -c "import PIL" >/dev/null 2>&1; then')
+                runner_lines.append('    echo "ERROR: LaMa / MI-GAN MLX serving needs Pillow in the launch Python: $ODYSSEUS_MLX_IMAGE_CMD_PY."')
+                runner_lines.append('    echo "Install with: $ODYSSEUS_MLX_IMAGE_CMD_PY -m pip install -U fastapi uvicorn python-multipart pillow huggingface_hub"')
+                runner_lines.append('    ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('  fi')
+                runner_lines.append('  if ! command -v odysseus-mlx-inpaint >/dev/null 2>&1 && ! command -v mlx-lama-serve >/dev/null 2>&1; then')
+                runner_lines.append('    echo "ERROR: LaMa / MI-GAN MLX serving requires the Odysseus mlx-lama-swift bridge on PATH: odysseus-mlx-inpaint or mlx-lama-serve."')
+                runner_lines.append('    echo "Build it from swift/odysseus-mlx-image-bridge in Cookbook Dependencies."')
+                runner_lines.append('    ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('  fi')
+                runner_lines.append('  ODYSSEUS_INPAINT_BIN="$(command -v odysseus-mlx-inpaint 2>/dev/null || command -v mlx-lama-serve 2>/dev/null || true)"')
+                runner_lines.append('  if [ -n "$ODYSSEUS_INPAINT_BIN" ]; then')
+                runner_lines.append('    ODYSSEUS_INPAINT_DIR="$(dirname "$ODYSSEUS_INPAINT_BIN")"')
+                runner_lines.append('    if [ ! -f "$ODYSSEUS_INPAINT_DIR/mlx.metallib" ] && [ ! -f "$ODYSSEUS_INPAINT_DIR/default.metallib" ]; then')
+                runner_lines.append('      echo "ERROR: LaMa / MI-GAN MLX serving found the Swift runner, but mlx.metallib/default.metallib is missing next to it."')
+                runner_lines.append('      echo "Run the LaMa / MI-GAN MLX image editing dependency install again; it copies mlx.metallib from the launch Python MLX package."')
+                runner_lines.append('      ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('    fi')
+                runner_lines.append('  fi')
+                runner_lines.append('elif ! command -v mflux-generate >/dev/null 2>&1 && ! command -v mflux-generate-qwen >/dev/null 2>&1; then')
+                runner_lines.append('  echo "ERROR: mflux-compatible MLX image serving requires mflux-generate or mflux-generate-qwen in PATH for launch Python: $ODYSSEUS_MLX_IMAGE_CMD_PY."')
+                runner_lines.append('  echo "Install with: $ODYSSEUS_MLX_IMAGE_CMD_PY -m pip install -U mflux fastapi uvicorn python-multipart"')
+                runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
+                runner_lines.append('fi')
             elif "scripts/diffusion_server.py" in req.cmd or ".diffusion_server.py" in req.cmd:
                 runner_lines.append('export PATH="$HOME/.local/bin:$PATH"')
-                runner_lines.append('if ! ODYSSEUS_DIFFUSION_IMPORT_ERROR="$(python3 -c "import torch, diffusers" 2>&1)"; then')
-                runner_lines.append('  echo "ERROR: Diffusion serving requires PyTorch + diffusers."')
+                runner_lines.append(f"ODYSSEUS_SERVE_CMD='{_bash_squote(req.cmd)}'")
+                runner_lines.append('ODYSSEUS_DIFFUSION_CMD_PY="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
+                runner_lines.append('import shlex, sys')
+                runner_lines.append('parts = shlex.split(sys.argv[1])')
+                runner_lines.append('py = "python3"')
+                runner_lines.append('for part in parts:')
+                runner_lines.append('    if part.endswith("/bin/python") or part.endswith("/bin/python3") or "/bin/python3." in part:')
+                runner_lines.append('        py = part')
+                runner_lines.append('        break')
+                runner_lines.append('print(py)')
+                runner_lines.append('PY')
+                runner_lines.append(')"')
+                runner_lines.append('if ! ODYSSEUS_DIFFUSION_IMPORT_ERROR="$("$ODYSSEUS_DIFFUSION_CMD_PY" -c "import torch, torchvision, diffusers" 2>&1)"; then')
+                runner_lines.append('  echo "ERROR: Diffusion serving requires PyTorch + Torchvision + diffusers in the launch Python: $ODYSSEUS_DIFFUSION_CMD_PY."')
                 runner_lines.append('  printf "%s\\n" "$ODYSSEUS_DIFFUSION_IMPORT_ERROR"')
                 runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
                 runner_lines.append('fi')
@@ -2588,8 +2747,8 @@ def setup_cookbook_routes() -> APIRouter:
         # endpoint; any other real model serve (i.e. not a pip-install task) gets
         # a local LLM endpoint pointed at its /v1.
         endpoint_id = None
-        is_diffusion = "diffusion_server.py" in req.cmd
-        if is_diffusion:
+        is_image_endpoint = "diffusion_server.py" in req.cmd or "mlx_image_server.py" in req.cmd
+        if is_image_endpoint:
             endpoint_id = _auto_register_image_endpoint(req, remote)
         elif not is_pip_install:
             endpoint_id = _auto_register_llm_endpoint(req, remote)
@@ -2605,7 +2764,7 @@ def setup_cookbook_routes() -> APIRouter:
         # if N != 0 within the watch window, delete the endpoint we just
         # created. Skipped for diffusion (different image-endpoint cleanup
         # path) and pip-install tasks (no endpoint to drop).
-        if endpoint_id and not is_diffusion and not is_pip_install:
+        if endpoint_id and not is_image_endpoint and not is_pip_install:
             asyncio.create_task(_serve_crash_watchdog(
                 endpoint_id=endpoint_id,
                 session_id=session_id,

routes/document_routes.py

@@ -12,6 +12,7 @@ from core.database import SessionLocal, Document, DocumentVersion
 from core.database import Session as DbSession
 from src.auth_helpers import get_current_user, _auth_disabled
 from src.constants import MAIL_ATTACHMENTS_DIR
+from src.upload_handler import reserve_upload_references
 
 logger = logging.getLogger(__name__)
 
@@ -78,6 +79,14 @@ from routes.document_helpers import (
 def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
     router = APIRouter(tags=["documents"])
 
+    def _reserve_document_uploads(user: Optional[str], content: str) -> None:
+        missing_id = reserve_upload_references(upload_handler, user, content)
+        if missing_id:
+            raise HTTPException(
+                409,
+                f"Referenced upload is no longer available: {missing_id}",
+            )
+
     def _locate_current_user_upload(request: Request, upload_id: str, user: Optional[str]):
         if upload_handler is None:
             return None
@@ -124,6 +133,7 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
             if _looks_like_email_document(req.content, req.title):
                 language = "email"
 
+            _reserve_document_uploads(user, req.content)
             _assert_pdf_marker_upload_owned(request, req.content, user, upload_handler)
 
             # Reply drafts are keyed to the source email. If a UI/tool path tries
@@ -636,6 +646,7 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
             if doc.current_content == incoming_content and not req.force_version:
                 return _doc_to_dict(doc)
 
+            _reserve_document_uploads(user, incoming_content)
             _assert_pdf_marker_upload_owned(request, incoming_content, user, upload_handler)
 
             # Check if we can coalesce with the latest version

routes/email_helpers.py

@@ -246,6 +246,7 @@ import re as _re_reply
 # serves replies and summaries (any fenced final-output block).
 _REPLY_OPEN_RE = _re_reply.compile(r"<<<\s*(?:REPLY|SUMMARY|OUTPUT)\s*>>+", _re_reply.I)
 _REPLY_CLOSE_RE = _re_reply.compile(r"<<<\s*END\s*>>+", _re_reply.I)
+_REPLY_ROLE_MARKER_RE = _re_reply.compile(r"</?\|(?:assistant|assistan|user|system|tool)\|>?|</\|end\|>?", _re_reply.I)
 
 
 def _extract_reply(text: str) -> str:
@@ -272,6 +273,7 @@ def _extract_reply(text: str) -> str:
     # Drop any stray/duplicate marker tokens, then strip think markup.
     t = _REPLY_OPEN_RE.sub("", t)
     t = _REPLY_CLOSE_RE.sub("", t)
+    t = _REPLY_ROLE_MARKER_RE.sub("", t)
     return _strip_think(t).strip()
 
 
@@ -1035,13 +1037,23 @@ def _coerce_imap_timeout_seconds(raw: str | None) -> int:
 _IMAP_TIMEOUT_SECONDS = _coerce_imap_timeout_seconds(os.environ.get("ODYSSEUS_IMAP_TIMEOUT_SECONDS"))
 
 
-def _open_imap_connection(host: str, port: int, *, starttls: bool, timeout: int = _IMAP_TIMEOUT_SECONDS):
+def _open_imap_connection(
+    host: str,
+    port: int,
+    *,
+    starttls: bool,
+    timeout: int = _IMAP_TIMEOUT_SECONDS,
+    ssl_context=None,
+):
     """Open an IMAP connection using the configured security mode."""
     port = int(port or 993)
     if starttls:
         conn = imaplib.IMAP4(host, port, timeout=timeout)
         try:
-            conn.starttls()
+            if ssl_context:
+                conn.starttls(ssl_context=ssl_context)
+            else:
+                conn.starttls()
         except Exception:
             # Don't leak the open plain socket if the STARTTLS upgrade is
             # rejected; close it before propagating. (#3174)
@@ -1051,7 +1063,8 @@ def _open_imap_connection(host: str, port: int, *, starttls: bool, timeout: int
                 pass
             raise
     elif port == 993:
-        conn = imaplib.IMAP4_SSL(host, port, timeout=timeout)
+        kwargs = {"ssl_context": ssl_context} if ssl_context else {}
+        conn = imaplib.IMAP4_SSL(host, port, timeout=timeout, **kwargs)
     else:
         conn = imaplib.IMAP4(host, port, timeout=timeout)
     try:

routes/email_pollers.py

@@ -100,6 +100,272 @@ def _owner_for_email_account(account_id: str | None) -> str:
         return ""
 
 
+def _email_date_only(value: str | None):
+    value = (value or "").strip()
+    if not value:
+        return None
+    try:
+        return datetime.strptime(value[:10], "%Y-%m-%d").date()
+    except Exception:
+        return None
+
+
+_AUTO_REPLY_KEYS = {
+    "email_auto_reply",
+    "email_auto_reply_start",
+    "email_auto_reply_end",
+    "email_auto_reply_subject",
+    "email_auto_reply_message",
+    "email_auto_reply_cooldown",
+    "email_auto_reply_scope",
+    "email_auto_reply_account_id",
+    "email_auto_reply_exclude_automated",
+    "email_auto_reply_pause_notifications",
+    "email_auto_reply_enabled_at",
+}
+
+
+def _effective_settings_for_email_account(settings: dict, account_id: str | None) -> dict:
+    """Overlay per-account auto-reply settings onto global settings.
+
+    Other automation toggles remain global. This lets each mailbox have its own
+    away reply while preserving existing installs that only have global keys.
+    """
+    effective = dict(settings or {})
+    key = str(account_id or "").strip()
+    by_account = effective.get("email_auto_reply_by_account") or {}
+    account_cfg = by_account.get(key) if key and isinstance(by_account, dict) else None
+    if isinstance(account_cfg, dict):
+        for k in _AUTO_REPLY_KEYS:
+            if k in account_cfg:
+                effective[k] = account_cfg[k]
+    return effective
+
+
+def _away_reply_active(settings: dict, account_id: str | None) -> bool:
+    if not settings.get("email_auto_reply", False):
+        return False
+
+    scope = str(settings.get("email_auto_reply_scope") or "all").strip().lower()
+    if scope == "account":
+        selected = str(settings.get("email_auto_reply_account_id") or "").strip()
+        if selected and selected != str(account_id or ""):
+            return False
+
+    today = datetime.utcnow().date()
+    start = _email_date_only(settings.get("email_auto_reply_start"))
+    end = _email_date_only(settings.get("email_auto_reply_end"))
+    if start and today < start:
+        return False
+    if end and today > end:
+        return False
+    return True
+
+
+def _message_after_away_enabled(settings: dict, msg) -> bool:
+    enabled_at = (settings.get("email_auto_reply_enabled_at") or "").strip()
+    if not enabled_at:
+        # Existing installs may already have the toggle on before this feature
+        # existed. Do not back-reply old mail until the user saves/toggles it.
+        return False
+    try:
+        enabled_dt = datetime.fromisoformat(enabled_at.replace("Z", "+00:00"))
+    except Exception:
+        return False
+    try:
+        msg_dt = email.utils.parsedate_to_datetime(msg.get("Date", ""))
+    except Exception:
+        return False
+    try:
+        if enabled_dt.tzinfo and not msg_dt.tzinfo:
+            msg_dt = msg_dt.replace(tzinfo=enabled_dt.tzinfo)
+        elif msg_dt.tzinfo and not enabled_dt.tzinfo:
+            enabled_dt = enabled_dt.replace(tzinfo=msg_dt.tzinfo)
+    except Exception:
+        pass
+    return msg_dt >= enabled_dt
+
+
+def _away_reply_period_key(settings: dict) -> str:
+    start = (settings.get("email_auto_reply_start") or "").strip()
+    end = (settings.get("email_auto_reply_end") or "").strip()
+    return f"{start or '*'}..{end or '*'}"
+
+
+def _away_reply_cooldown_seconds(settings: dict) -> int | None:
+    raw = str(settings.get("email_auto_reply_cooldown") or "period").strip().lower()
+    if raw == "1d":
+        return 24 * 60 * 60
+    if raw == "3d":
+        return 3 * 24 * 60 * 60
+    if raw == "7d":
+        return 7 * 24 * 60 * 60
+    return None
+
+
+def _ensure_away_reply_table():
+    import sqlite3 as _sql3
+    conn = _sql3.connect(SCHEDULED_DB)
+    try:
+        conn.execute("""
+            CREATE TABLE IF NOT EXISTS email_away_replies (
+                id INTEGER PRIMARY KEY AUTOINCREMENT,
+                owner TEXT DEFAULT '',
+                account_id TEXT DEFAULT '',
+                message_id TEXT DEFAULT '',
+                sender_addr TEXT DEFAULT '',
+                subject TEXT DEFAULT '',
+                period_key TEXT DEFAULT '',
+                sent_at TEXT DEFAULT ''
+            )
+        """)
+        conn.execute("CREATE INDEX IF NOT EXISTS idx_email_away_msg ON email_away_replies(owner, account_id, message_id)")
+        conn.execute("CREATE INDEX IF NOT EXISTS idx_email_away_sender ON email_away_replies(owner, account_id, sender_addr, sent_at)")
+        conn.commit()
+    finally:
+        conn.close()
+
+
+def _sender_is_automated(msg, sender_addr: str) -> bool:
+    auto_submitted = (msg.get("Auto-Submitted") or "").strip().lower()
+    if auto_submitted and auto_submitted != "no":
+        return True
+    precedence = (msg.get("Precedence") or "").strip().lower()
+    if precedence in {"bulk", "junk", "list"}:
+        return True
+    if msg.get("List-Id") or msg.get("List-Unsubscribe"):
+        return True
+    local = (sender_addr or "").split("@", 1)[0].lower()
+    return local in {
+        "no-reply", "noreply", "do-not-reply", "donotreply",
+        "notification", "notifications", "automated", "mailer-daemon",
+        "postmaster",
+    }
+
+
+def _away_reply_already_sent(settings: dict, account_owner: str, account_id: str | None,
+                             message_id: str, sender_addr: str) -> bool:
+    import sqlite3 as _sql3
+    _ensure_away_reply_table()
+    owner = account_owner or ""
+    aid = account_id or ""
+    sender = (sender_addr or "").strip().lower()
+    conn = _sql3.connect(SCHEDULED_DB)
+    try:
+        row = conn.execute(
+            "SELECT 1 FROM email_away_replies WHERE owner=? AND account_id=? AND message_id=? LIMIT 1",
+            (owner, aid, message_id),
+        ).fetchone()
+        if row:
+            return True
+
+        cooldown = _away_reply_cooldown_seconds(settings)
+        if cooldown is None:
+            period_key = _away_reply_period_key(settings)
+            row = conn.execute(
+                "SELECT 1 FROM email_away_replies WHERE owner=? AND account_id=? AND sender_addr=? AND period_key=? LIMIT 1",
+                (owner, aid, sender, period_key),
+            ).fetchone()
+            return bool(row)
+
+        since = datetime.utcnow().timestamp() - cooldown
+        rows = conn.execute(
+            "SELECT sent_at FROM email_away_replies WHERE owner=? AND account_id=? AND sender_addr=? ORDER BY sent_at DESC LIMIT 5",
+            (owner, aid, sender),
+        ).fetchall()
+        for (sent_at,) in rows:
+            try:
+                if datetime.fromisoformat(sent_at).timestamp() >= since:
+                    return True
+            except Exception:
+                continue
+        return False
+    finally:
+        conn.close()
+
+
+def _record_away_reply(settings: dict, account_owner: str, account_id: str | None,
+                       message_id: str, sender_addr: str, subject: str):
+    import sqlite3 as _sql3
+    _ensure_away_reply_table()
+    conn = _sql3.connect(SCHEDULED_DB)
+    try:
+        conn.execute(
+            """
+            INSERT INTO email_away_replies
+            (owner, account_id, message_id, sender_addr, subject, period_key, sent_at)
+            VALUES (?, ?, ?, ?, ?, ?, ?)
+            """,
+            (
+                account_owner or "",
+                account_id or "",
+                message_id,
+                (sender_addr or "").strip().lower(),
+                subject or "",
+                _away_reply_period_key(settings),
+                datetime.utcnow().isoformat(),
+            ),
+        )
+        conn.commit()
+    finally:
+        conn.close()
+
+
+def _send_away_reply(settings: dict, account_owner: str, account_id: str | None,
+                     msg, message_id: str, sender: str, subject: str):
+    sender_name, sender_addr = email.utils.parseaddr(sender or "")
+    sender_addr = (sender_addr or "").strip()
+    if not sender_addr:
+        return False, "missing sender"
+
+    cfg = _get_email_config(account_id, owner=account_owner)
+    from_addr = (cfg.get("from_address") or cfg.get("smtp_user") or "").strip()
+    if not from_addr:
+        return False, "missing from address"
+    if sender_addr.lower() == from_addr.lower():
+        return False, "self mail"
+    if settings.get("email_auto_reply_exclude_automated", True) and _sender_is_automated(msg, sender_addr):
+        return False, "automated sender"
+    if _away_reply_already_sent(settings, account_owner, account_id, message_id, sender_addr):
+        return False, "already sent"
+
+    body = (settings.get("email_auto_reply_message") or "").strip()
+    if not body:
+        body = "Thanks for your email. I'm away and may be slower to reply."
+
+    subject_template = (settings.get("email_auto_reply_subject") or "(Away) {subject}").strip()
+    if subject_template:
+        original_subject = subject or ""
+        reply_subject = (
+            subject_template
+            .replace("{subject}", original_subject)
+            .replace("{original_subject}", original_subject)
+        ).strip() or "Re:"
+    else:
+        reply_subject = subject or ""
+        if not reply_subject.lower().lstrip().startswith("re:"):
+            reply_subject = f"Re: {reply_subject}" if reply_subject else "Re:"
+
+    outer = MIMEMultipart("alternative")
+    display = cfg.get("display_name") or ""
+    outer["From"] = email.utils.formataddr((display, from_addr)) if display else from_addr
+    outer["To"] = email.utils.formataddr((sender_name, sender_addr)) if sender_name else sender_addr
+    outer["Subject"] = reply_subject
+    outer["Date"] = email.utils.formatdate(localtime=False)
+    outer["Message-ID"] = email.utils.make_msgid()
+    outer["Auto-Submitted"] = "auto-replied"
+    outer["X-Auto-Response-Suppress"] = "All"
+    if message_id:
+        outer["In-Reply-To"] = message_id
+        refs = (msg.get("References") or "").strip()
+        outer["References"] = f"{refs} {message_id}".strip()
+    outer.attach(MIMEText(body, "plain", "utf-8"))
+
+    _send_smtp_message(cfg, from_addr, [sender_addr], outer.as_string())
+    _record_away_reply(settings, account_owner, account_id, message_id, sender_addr, subject)
+    return True, sender_addr
+
+
 # ── Routes ──
 
 async def _emit_progress(progress_cb, message: str):
@@ -125,9 +391,10 @@ async def _run_auto_summarize_once(do_summary: bool = True, do_reply: bool = Tru
     settings = _load_settings()
     prev = {k: settings.get(k, False) for k in
             ("email_auto_summarize", "email_auto_reply", "email_auto_tag",
-             "email_auto_spam", "email_auto_calendar")}
+             "email_auto_spam", "email_auto_calendar", "_email_auto_reply_draft_only")}
     settings["email_auto_summarize"] = bool(do_summary)
     settings["email_auto_reply"] = bool(do_reply)
+    settings["_email_auto_reply_draft_only"] = bool(do_reply)
     settings["email_auto_tag"] = bool(do_tag)
     settings["email_auto_spam"] = bool(do_spam)
     settings["email_auto_calendar"] = bool(do_calendar)
@@ -142,7 +409,10 @@ async def _run_auto_summarize_once(do_summary: bool = True, do_reply: bool = Tru
     finally:
         s2 = _load_settings()
         for k, v in prev.items():
-            s2[k] = v
+            if v is None and k.startswith("_"):
+                s2.pop(k, None)
+            else:
+                s2[k] = v
         _save_settings(s2)
 
 
@@ -176,7 +446,7 @@ def _latest_inbox_fallback_uids(conn, reconnect):
         return [], reconnect()
 
 
-async def _auto_summarize_pass(days_back: int = 1, account_id: str | None = None, max_process: int | None = None, progress_cb=None) -> str:
+async def _auto_summarize_pass(days_back: int = 1, account_id: str | None = None, max_process: int | None = None, progress_cb=None, away_only: bool = False) -> str:
     """Single pass of the auto-summarize/reply scan.
 
     When account_id is None, iterates over every enabled account in
@@ -208,6 +478,7 @@ async def _auto_summarize_pass(days_back: int = 1, account_id: str | None = None
                 account_id=(ids[0] if ids else None),
                 max_process=max_process,
                 progress_cb=progress_cb,
+                away_only=away_only,
             )
         outs = []
         for idx, aid in enumerate(ids, start=1):
@@ -218,6 +489,7 @@ async def _auto_summarize_pass(days_back: int = 1, account_id: str | None = None
                     account_id=aid,
                     max_process=max_process,
                     progress_cb=progress_cb,
+                    away_only=away_only,
                 )
                 outs.append(f"[{names.get(aid, aid[:8])}] {result}")
             except Exception as e:
@@ -229,23 +501,32 @@ async def _auto_summarize_pass(days_back: int = 1, account_id: str | None = None
         account_id=account_id,
         max_process=max_process,
         progress_cb=progress_cb,
+        away_only=away_only,
     )
 
 
-async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None = None, max_process: int | None = None, progress_cb=None) -> str:
+async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None = None, max_process: int | None = None, progress_cb=None, away_only: bool = False) -> str:
     """Single pass of the auto-summarize/reply scan for ONE account.
     Reads current settings flags."""
     import asyncio
     import sqlite3 as _sql3
     from src.llm_core import _uses_max_completion_tokens
 
-    settings = _load_settings()
+    settings = _effective_settings_for_email_account(_load_settings(), account_id)
     auto_sum = settings.get("email_auto_summarize", False)
     auto_reply = settings.get("email_auto_reply", False)
+    auto_reply_draft = bool(auto_reply and settings.get("_email_auto_reply_draft_only", False))
+    auto_reply_away = bool(auto_reply and not auto_reply_draft and _away_reply_active(settings, account_id))
     auto_tag = settings.get("email_auto_tag", False)
     auto_spam = settings.get("email_auto_spam", False)
     auto_cal = settings.get("email_auto_calendar", False)
-    if not auto_sum and not auto_reply and not auto_tag and not auto_spam and not auto_cal:
+    if away_only:
+        auto_sum = False
+        auto_reply_draft = False
+        auto_tag = False
+        auto_spam = False
+        auto_cal = False
+    if not auto_sum and not auto_reply_draft and not auto_reply_away and not auto_tag and not auto_spam and not auto_cal:
         return "Nothing to do"
 
     # Owner of the account being processed. All calendar + mailbox reads/writes
@@ -304,11 +585,11 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
 
         _c = _sql3.connect(SCHEDULED_DB)
         _cache_owner_clause, _cache_owner_params = _email_cache_owner_clause(account_owner)
-        _sum_existing = {r[0] for r in _c.execute(
+        _sum_existing = set() if away_only else {r[0] for r in _c.execute(
             f"SELECT message_id FROM email_summaries WHERE {_cache_owner_clause}",
             _cache_owner_params,
         ).fetchall()}
-        _reply_existing = {r[0] for r in _c.execute(
+        _reply_existing = set() if away_only else {r[0] for r in _c.execute(
             f"SELECT message_id FROM email_ai_replies WHERE {_cache_owner_clause}",
             _cache_owner_params,
         ).fetchall()}
@@ -325,7 +606,7 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
                 ).fetchall()}
         else:
             _tag_existing = set()
-        _cal_existing = {r[0] for r in _c.execute(
+        _cal_existing = set() if away_only else {r[0] for r in _c.execute(
             f"SELECT message_id FROM email_calendar_extractions WHERE {_cache_owner_clause}",
             _cache_owner_params,
         ).fetchall()} if auto_cal else set()
@@ -351,12 +632,21 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
         if auto_spam and not spam_folder:
             logger.warning("Auto-spam enabled but no Junk/Spam folder detected — will classify but not move")
 
-        task_candidates = resolve_task_candidates(owner=account_owner)
-        if not task_candidates:
-            return "No model configured"
-        url, model, headers = task_candidates[0]
+        needs_llm = bool(auto_sum or auto_reply_draft or auto_tag or auto_spam or auto_cal)
+        if needs_llm:
+            task_candidates = resolve_task_candidates(owner=account_owner)
+            if not task_candidates:
+                return "No model configured"
+            url, model, headers = task_candidates[0]
+        else:
+            url, model, headers = None, "", None
 
-        writing_style = settings.get("email_writing_style", "")
+        by_account_styles = settings.get("email_writing_styles_by_account") or {}
+        writing_style = ""
+        if account_id and isinstance(by_account_styles, dict):
+            writing_style = str(by_account_styles.get(str(account_id)) or "")
+        if not writing_style:
+            writing_style = settings.get("email_writing_style", "")
         processed = 0
         already_cached = 0
         too_short = 0
@@ -366,12 +656,15 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
         _events_created = 0
         _replies_drafted = 0
         _reply_failed = 0
+        _away_replies_sent = 0
+        _away_replies_skipped = 0
+        _away_replies_failed = 0
         _detail_lines = []
         _current_folder = "INBOX"
         # Calendar extraction is sequential and each row can involve a model
         # call plus a calendar write. Keep the scheduled calendar-only pass
         # below the 5-minute action budget instead of timing out mid-run.
-        _default_max_process = 3 if (auto_cal and not auto_sum and not auto_reply and not auto_tag and not auto_spam) else 5
+        _default_max_process = 3 if (auto_cal and not auto_sum and not auto_reply_draft and not auto_reply_away and not auto_tag and not auto_spam) else 5
         try:
             _max_process = max(1, int(max_process)) if max_process is not None else _default_max_process
         except Exception:
@@ -402,10 +695,6 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
                     seed = f"{_folder}|{uid_str}|{msg.get('From','')}|{msg.get('Date','')}|{msg.get('Subject','')}"
                     message_id = f"<synth-{_hl.sha256(seed.encode()).hexdigest()[:16]}@local>"
                     no_msgid += 1
-                need_sum = auto_sum and message_id not in _sum_existing
-                need_reply = auto_reply and message_id not in _reply_existing
-                need_class = (auto_tag or auto_spam) and message_id not in _tag_existing
-                need_cal = bool(settings.get("email_auto_calendar", False)) and message_id not in _cal_existing
                 # Only check urgency on INBOX (received mail), not Sent
                 # Skip messages that are themselves urgency alerts, or that
                 # we sent to ourselves — otherwise the alert loop re-flags
@@ -422,17 +711,45 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
                 except Exception:
                     _from_addr_only = ""
                 _is_self_mail = bool(_self_self_addr) and _from_addr_only.lower() == _self_self_addr
+                need_sum = auto_sum and message_id not in _sum_existing
+                need_reply = auto_reply_draft and message_id not in _reply_existing
+                need_away_reply = bool(
+                    auto_reply_away
+                    and _folder.upper() == "INBOX"
+                    and not _is_self_mail
+                    and (away_only or _message_after_away_enabled(settings, msg))
+                    and not _away_reply_already_sent(settings, account_owner, account_id, message_id, _from_addr_only)
+                )
+                need_class = (auto_tag or auto_spam) and message_id not in _tag_existing
+                need_cal = bool(settings.get("email_auto_calendar", False)) and message_id not in _cal_existing
                 need_urgent = (auto_urgent and message_id not in _urgent_existing
                                and not _folder.lower().startswith("sent")
                                and "sent" not in _folder.lower()
                                and not _is_alert_echo
                                and not _is_self_mail)
-                if not need_sum and not need_reply and not need_class and not need_cal and not need_urgent:
+                if not need_sum and not need_reply and not need_away_reply and not need_class and not need_cal and not need_urgent:
                     already_cached += 1
                     await _emit_progress(progress_cb, f"Checked {examined}/{len(uid_list)} · {already_cached} already cached")
                     continue
                 subject = _decode_header(msg.get("Subject", ""))
                 sender = _decode_header(msg.get("From", ""))
+                if need_away_reply:
+                    try:
+                        sent_away, away_detail = _send_away_reply(
+                            settings, account_owner, account_id, msg, message_id, sender, subject
+                        )
+                        if sent_away:
+                            _away_replies_sent += 1
+                            _uid_text = uid.decode() if isinstance(uid, bytes) else str(uid)
+                            _detail_lines.append(f"away reply · {_folder}#{_uid_text} · {subject or '(no subject)'} — {away_detail}")
+                        else:
+                            _away_replies_skipped += 1
+                            logger.info(f"Away reply skipped for uid={uid}: {away_detail}")
+                    except Exception as e:
+                        _away_replies_failed += 1
+                        _uid_text = uid.decode() if isinstance(uid, bytes) else str(uid)
+                        _detail_lines.append(f"away reply failed · {_folder}#{_uid_text} · {subject or '(no subject)'}")
+                        logger.warning(f"Away reply {uid} failed: {e}")
                 body = _extract_text(msg)
                 # Pull text out of any PDFs / text attachments and append to
                 # the body so summaries / replies can actually reason about
@@ -454,7 +771,7 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
                 elif need_reply:
                     if not body:
                         body = subject
-                elif (not body or len(body) < 100) and not att_text:
+                elif not need_away_reply and (not body or len(body) < 100) and not att_text:
                     too_short += 1
                     continue
                 # Augmented body sent to the LLM: original body + attachment text.
@@ -993,7 +1310,8 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
         # Build a clear status message
         ops = []
         if auto_sum: ops.append("summary")
-        if auto_reply: ops.append("reply")
+        if auto_reply_draft: ops.append("reply")
+        if auto_reply_away: ops.append("away")
         if auto_tag: ops.append("tag")
         if auto_spam: ops.append("spam")
         ops_label = "/".join(ops) or "none"
@@ -1002,10 +1320,14 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
             parts.append(f"processed {processed} new")
         if auto_sum:
             parts.append(f"summarized {_summaries_created}")
-        if auto_reply:
+        if auto_reply_draft:
             parts.append(f"drafted {_replies_drafted} repl" + ("y" if _replies_drafted == 1 else "ies"))
             if _reply_failed:
                 parts.append(f"{_reply_failed} reply failed")
+        if auto_reply_away:
+            parts.append(f"sent {_away_replies_sent} away repl" + ("y" if _away_replies_sent == 1 else "ies"))
+            if _away_replies_failed:
+                parts.append(f"{_away_replies_failed} away failed")
         if already_cached:
             parts.append(f"{already_cached} already cached")
         if too_short:
@@ -1032,12 +1354,13 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
 
 
 async def _auto_summarize_poller():
-    """Background loop kept for backward compatibility — calls _auto_summarize_pass every 60s.
+    """Background loop kept for backward compatibility — calls _auto_summarize_pass periodically.
     Newer setups should use scheduled tasks instead (summarize_emails, draft_email_replies)."""
     import asyncio as _asyncio
     while True:
         try:
-            await _asyncio.sleep(1800)
+            settings = _load_settings()
+            await _asyncio.sleep(60 if settings.get("email_auto_reply", False) else 1800)
             await _auto_summarize_pass()
         except Exception as e:
             logger.error(f"Auto-summarize poller crash: {e}")
@@ -1068,6 +1391,28 @@ def _scheduled_poll_once() -> dict:
         for r in rows:
             sid = r[0]
             try:
+                # Atomically claim this row before doing any work. Two
+                # pollers can race here (the in-process asyncio task and an
+                # externally cron-driven `odysseus-mail poll-scheduled`, or
+                # an admin running the CLI manually alongside the in-process
+                # one despite the ODYSSEUS_INPROCESS_POLLERS=0 guidance) -
+                # both can SELECT the same 'pending' row before either has
+                # updated its status. The UPDATE...WHERE status='pending' is
+                # the atomicity boundary: only the poller whose UPDATE
+                # actually changes a row (rowcount == 1) proceeds to send;
+                # a loser sees rowcount == 0 and skips it instead of sending
+                # a duplicate.
+                claim_conn = sqlite3.connect(SCHEDULED_DB)
+                claim_cur = claim_conn.execute(
+                    "UPDATE scheduled_emails SET status='sending' WHERE id=? AND status='pending'",
+                    (sid,),
+                )
+                claim_conn.commit()
+                claimed = claim_cur.rowcount == 1
+                claim_conn.close()
+                if not claimed:
+                    continue
+
                 attachments = json.loads(r[8] or "[]")
                 row_account_id = r[9] if len(r) > 9 else None
                 odysseus_kind = r[10] if len(r) > 10 else "scheduled"

routes/gallery/gallery_routes.py

@@ -1,7 +1,9 @@
 """Gallery routes — browsable library for photos and AI-generated images."""
 
 import os
+import base64
 import hashlib
+import io
 import logging
 import re
 import uuid
@@ -27,6 +29,165 @@ from routes.gallery.gallery_helpers import (
 
 logger = logging.getLogger(__name__)
 
+_SAM_STATE: Dict[str, Any] = {}
+_GROUNDING_STATE: Dict[str, Any] = {}
+
+
+def _b64_to_pil_image(image_b64: str, *, mode: str = "RGBA"):
+    if not image_b64:
+        raise HTTPException(400, "Missing image")
+    if "," in image_b64 and image_b64.split(",", 1)[0].startswith("data:"):
+        image_b64 = image_b64.split(",", 1)[1]
+    try:
+        from PIL import Image
+
+        raw = base64.b64decode(image_b64)
+        return Image.open(io.BytesIO(raw)).convert(mode)
+    except HTTPException:
+        raise
+    except Exception as exc:
+        raise HTTPException(400, "Invalid image") from exc
+
+
+def _pil_image_to_b64(img, *, fmt: str = "PNG") -> str:
+    buf = io.BytesIO()
+    img.save(buf, format=fmt)
+    return base64.b64encode(buf.getvalue()).decode("ascii")
+
+
+def _load_sam_backend():
+    model_id = os.getenv("ODYSSEUS_SAM_MODEL", "facebook/sam-vit-base")
+    cached = _SAM_STATE.get(model_id)
+    if cached:
+        return cached
+    try:
+        import torch
+        from transformers import SamModel, SamProcessor
+    except Exception as exc:
+        raise HTTPException(
+            501,
+            "SAM mask tools are not installed. Install Cookbook Dependencies -> SAM mask tools.",
+        ) from exc
+
+    device = "cpu"
+    try:
+        if torch.cuda.is_available():
+            device = "cuda"
+        elif getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
+            device = "mps"
+    except Exception:
+        device = "cpu"
+
+    try:
+        processor = SamProcessor.from_pretrained(model_id)
+        model = SamModel.from_pretrained(model_id)
+        model.to(device)
+        model.eval()
+    except Exception as exc:
+        raise HTTPException(500, f"Failed to load SAM model {model_id}: {exc}") from exc
+
+    cached = {"torch": torch, "processor": processor, "model": model, "device": device, "model_id": model_id}
+    _SAM_STATE[model_id] = cached
+    return cached
+
+
+def _load_grounding_backend():
+    model_id = os.getenv("ODYSSEUS_GROUNDING_MODEL", "google/owlvit-base-patch32")
+    cached = _GROUNDING_STATE.get(model_id)
+    if cached:
+        return cached
+    try:
+        import torch
+        from transformers import OwlViTForObjectDetection, OwlViTProcessor
+    except Exception as exc:
+        raise HTTPException(
+            501,
+            "Object mask tools are not installed. Install Cookbook Dependencies -> SAM mask tools.",
+        ) from exc
+
+    device = "cpu"
+    try:
+        if torch.cuda.is_available():
+            device = "cuda"
+        elif getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
+            device = "mps"
+    except Exception:
+        device = "cpu"
+
+    try:
+        processor = OwlViTProcessor.from_pretrained(model_id)
+        model = OwlViTForObjectDetection.from_pretrained(model_id)
+        model.to(device)
+        model.eval()
+    except Exception as exc:
+        raise HTTPException(500, f"Failed to load object mask model {model_id}: {exc}") from exc
+
+    cached = {"torch": torch, "processor": processor, "model": model, "device": device, "model_id": model_id}
+    _GROUNDING_STATE[model_id] = cached
+    return cached
+
+
+def _ground_text_to_box(image, text: str, *, threshold: float = 0.05):
+    query = (text or "").strip()
+    if not query:
+        raise HTTPException(400, "Missing object text")
+    backend = _load_grounding_backend()
+    torch = backend["torch"]
+    processor = backend["processor"]
+    model = backend["model"]
+    device = backend["device"]
+
+    labels = [query]
+    if not query.lower().startswith(("a ", "an ", "the ")):
+        labels.append(f"a photo of {query}")
+    try:
+        inputs = processor(text=[labels], images=image, return_tensors="pt")
+        model_inputs = {
+            k: (v.to(device) if hasattr(v, "to") else v)
+            for k, v in inputs.items()
+        }
+        with torch.no_grad():
+            outputs = model(**model_inputs)
+        target_sizes = torch.tensor([[image.height, image.width]])
+        if hasattr(processor, "post_process_object_detection"):
+            results = processor.post_process_object_detection(
+                outputs=outputs,
+                target_sizes=target_sizes,
+                threshold=float(threshold),
+            )
+        elif hasattr(processor, "post_process_grounded_object_detection"):
+            results = processor.post_process_grounded_object_detection(
+                outputs=outputs,
+                target_sizes=target_sizes,
+                threshold=float(threshold),
+                text_labels=[labels],
+            )
+        else:
+            raise HTTPException(500, "Installed Transformers does not expose OWL-ViT object detection post-processing")
+        boxes = results[0].get("boxes")
+        scores = results[0].get("scores")
+        labels_idx = results[0].get("labels")
+        text_labels = results[0].get("text_labels") or results[0].get("labels_text")
+        if boxes is None or scores is None or len(boxes) == 0:
+            raise HTTPException(404, f"No visible object matched '{query}'")
+        idx = int(torch.argmax(scores).item())
+        box = [float(v) for v in boxes[idx].detach().cpu().tolist()]
+        label_idx = int(labels_idx[idx].detach().cpu().item()) if labels_idx is not None and len(labels_idx) else 0
+        label = labels[min(label_idx, len(labels) - 1)]
+        if text_labels and len(text_labels) > idx:
+            label = str(text_labels[idx])
+        return {
+            "box": box,
+            "score": float(scores[idx].detach().cpu().item()),
+            "label": label,
+            "model": backend["model_id"],
+        }
+    except HTTPException:
+        raise
+    except Exception as exc:
+        logger.exception("ground_text_to_box failed")
+        raise HTTPException(500, f"Object mask failed: {exc}") from exc
+
 
 def _current_user_is_admin(request: Request, user: str | None) -> bool:
     if not user:
@@ -1240,20 +1401,89 @@ def setup_gallery_routes() -> APIRouter:
             except httpx.TimeoutException:
                 raise HTTPException(504, "OpenAI inpaint timed out (120s)")
 
-        # Self-hosted diffusion server path
+        # Self-hosted diffusion server path. Newer Odysseus image
+        # wrappers expose the OpenAI-compatible /v1/images/edits
+        # multipart route even when they are local/self-hosted. Older
+        # diffusion_server.py exposes /v1/images/inpaint as JSON. Try the
+        # OpenAI-compatible local route first, then fall back.
         try:
             # Forward chosen_model so the diffusion server can route if it ever
             # supports multiple models per process. Harmless if ignored.
             if chosen_model:
                 body["model"] = chosen_model
-            async with httpx.AsyncClient(timeout=120) as client:
+            async with httpx.AsyncClient(timeout=240) as client:
+                try:
+                    import base64, io
+                    from PIL import Image
+
+                    img_bytes = base64.b64decode(body["image"])
+                    mask_bytes = base64.b64decode(body["mask"])
+                    # Normalize both inputs to PNG bytes. Local MLX and
+                    # Diffusers wrappers expect white mask pixels to mean
+                    # "edit this region", which matches the editor's mask.
+                    source_png = Image.open(io.BytesIO(img_bytes)).convert("RGBA")
+                    mask_png = Image.open(io.BytesIO(mask_bytes)).convert("L")
+                    src_buf = io.BytesIO()
+                    source_png.save(src_buf, format="PNG")
+                    mask_buf = io.BytesIO()
+                    mask_png.save(mask_buf, format="PNG")
+                    files = {
+                        "image": ("source.png", src_buf.getvalue(), "image/png"),
+                        "mask": ("mask.png", mask_buf.getvalue(), "image/png"),
+                    }
+                    data = {
+                        "model": chosen_model or body.get("model") or "",
+                        "prompt": body.get("prompt", ""),
+                        "size": f"{int(body.get('width') or source_png.width)}x{int(body.get('height') or source_png.height)}",
+                        "n": "1",
+                    }
+                    r = await client.post(_join_checked_gallery_endpoint(base, "/images/edits"), data=data, files=files)
+                    if r.status_code == 200:
+                        result = r.json()
+                        if isinstance(result, dict) and result.get("data"):
+                            item = result["data"][0]
+                            if item.get("b64_json"):
+                                return {"image": item["b64_json"]}
+                            if item.get("url"):
+                                raw_b64 = await _fetch_result_image_b64(item["url"])
+                                if raw_b64:
+                                    return {"image": raw_b64}
+                        if isinstance(result, dict) and result.get("image"):
+                            return {"image": result["image"]}
+                        raise HTTPException(502, "Image edit endpoint returned no image")
+                    if r.status_code not in (404, 405):
+                        logger.warning("inpaint_proxy self-hosted edits: status %s", r.status_code)
+                        detail = "Image edit request failed"
+                        try:
+                            err = r.json()
+                            detail = err.get("detail") or err.get("error") or detail
+                        except Exception:
+                            pass
+                        # A plain SD/SDXL checkpoint often exposes
+                        # generation only at /images/edits.
+                        # That does not mean the endpoint cannot inpaint:
+                        # Odysseus diffusion_server.py has a dedicated
+                        # /images/inpaint route that can derive/fallback to
+                        # inpaint, img2img crop+composite, or txt2img
+                        # crop+composite. Fall through to that route instead
+                        # of surfacing "does not support image edits".
+                        if r.status_code == 400 and "does not support image edits" in str(detail).lower():
+                            logger.info("inpaint_proxy self-hosted edits unsupported; falling back to /images/inpaint")
+                        else:
+                            raise HTTPException(r.status_code, detail)
+                except HTTPException:
+                    raise
+                except Exception:
+                    logger.exception("inpaint_proxy: failed to prepare self-hosted edit request")
+                    raise HTTPException(400, "Failed to prepare inpaint request")
+
                 r = await client.post(_join_checked_gallery_endpoint(base, "/images/inpaint"), json=body)
                 if r.status_code != 200:
                     logger.error("inpaint_proxy diffusion: status %s", r.status_code)
                     raise HTTPException(r.status_code, "Inpaint request failed")
                 return r.json()
         except httpx.TimeoutException:
-            raise HTTPException(504, "Inpaint request timed out (120s)")
+            raise HTTPException(504, "Inpaint request timed out (240s)")
         except HTTPException:
             raise
         except Exception:
@@ -1588,6 +1818,135 @@ def setup_gallery_routes() -> APIRouter:
             return {"error": "AI upscale failed"}
 
     # ---- POST /api/image/remove-bg ----
+    @router.post("/api/image/mask")
+    async def smart_mask(request: Request):
+        """Create a neutral segmentation mask from user-provided points or a box.
+
+        This endpoint intentionally does not inspect edit prompts. It only
+        turns explicit visual selection hints into a binary mask that the
+        editor can reuse for wand/layer-mask/inpaint workflows.
+        """
+        require_privilege(request, "can_generate_images")
+        body = await request.json()
+        image = _b64_to_pil_image(body.get("image") or "", mode="RGB")
+        points = body.get("points") or []
+        box = body.get("box")
+        text = (body.get("text") or body.get("query") or "").strip()
+        grounded = None
+
+        if not points and not box and text:
+            grounded = _ground_text_to_box(image, text)
+            box = grounded["box"]
+
+        if not points and not box:
+            raise HTTPException(400, "Provide at least one point, box, or object text")
+
+        backend = _load_sam_backend()
+        torch = backend["torch"]
+        processor = backend["processor"]
+        model = backend["model"]
+        device = backend["device"]
+
+        kwargs: Dict[str, Any] = {"return_tensors": "pt"}
+        input_points = []
+        if points:
+            input_labels = []
+            for p in points:
+                try:
+                    input_points.append([float(p["x"]), float(p["y"])])
+                    input_labels.append(int(p.get("label", 1)))
+                except Exception as exc:
+                    raise HTTPException(400, "Invalid point format") from exc
+            kwargs["input_points"] = [input_points]
+            kwargs["input_labels"] = [input_labels]
+        if box:
+            if not isinstance(box, list) or len(box) != 4:
+                raise HTTPException(400, "Box must be [x1, y1, x2, y2]")
+            try:
+                kwargs["input_boxes"] = [[[float(v) for v in box]]]
+            except Exception as exc:
+                raise HTTPException(400, "Invalid box format") from exc
+
+        try:
+            inputs = processor(image, **kwargs)
+            model_inputs = {
+                k: (v.to(device) if hasattr(v, "to") else v)
+                for k, v in inputs.items()
+            }
+            with torch.no_grad():
+                outputs = model(**model_inputs)
+            masks = processor.image_processor.post_process_masks(
+                outputs.pred_masks.detach().cpu(),
+                inputs["original_sizes"].detach().cpu(),
+                inputs["reshaped_input_sizes"].detach().cpu(),
+            )
+            mask_tensor = masks[0]
+            while getattr(mask_tensor, "ndim", 0) > 3:
+                mask_tensor = mask_tensor[0]
+            if getattr(mask_tensor, "ndim", 0) == 3:
+                scores = outputs.iou_scores.detach().cpu()[0]
+                while getattr(scores, "ndim", 0) > 1:
+                    scores = scores[0]
+                # SAM commonly returns multiple candidates for a click. The
+                # highest-IoU candidate can be the entire image, which is
+                # useless as an editor selection. Prefer a candidate that
+                # contains the clicked point while keeping area reasonable.
+                point_xy = None
+                if input_points:
+                    try:
+                        point_xy = (
+                            int(round(float(input_points[0][0]))),
+                            int(round(float(input_points[0][1]))),
+                        )
+                    except Exception:
+                        point_xy = None
+                best_idx = 0
+                best_rank = None
+                total_px = max(1, int(mask_tensor.shape[-1]) * int(mask_tensor.shape[-2]))
+                for i in range(int(mask_tensor.shape[0])):
+                    candidate = mask_tensor[i]
+                    area_ratio = float(candidate.sum().item()) / float(total_px)
+                    if area_ratio >= 0.985:
+                        continue
+                    contains_click = True
+                    if point_xy:
+                        px = max(0, min(int(candidate.shape[-1]) - 1, point_xy[0]))
+                        py = max(0, min(int(candidate.shape[-2]) - 1, point_xy[1]))
+                        contains_click = bool(candidate[py, px].item())
+                    if not contains_click:
+                        continue
+                    score = float(scores[min(i, len(scores) - 1)].item()) if len(scores) else 0.0
+                    # Strongly penalize broad masks; a click-selection should
+                    # usually be local unless the user gives a box.
+                    rank = score - (area_ratio * 0.35)
+                    if best_rank is None or rank > best_rank:
+                        best_rank = rank
+                        best_idx = i
+                if best_rank is None and len(scores):
+                    best_idx = int(torch.argmax(scores).item())
+                mask_tensor = mask_tensor[min(best_idx, mask_tensor.shape[0] - 1)]
+            mask_array = (mask_tensor.numpy() > 0).astype("uint8") * 255
+            from PIL import Image
+
+            mask_img = Image.fromarray(mask_array, mode="L")
+            if mask_img.size != image.size:
+                mask_img = mask_img.resize(image.size, Image.NEAREST)
+            bbox = mask_img.getbbox()
+            result = {
+                "mask": _pil_image_to_b64(mask_img),
+                "bbox": list(bbox) if bbox else None,
+                "model": backend["model_id"],
+                "device": device,
+            }
+            if grounded:
+                result["grounding"] = grounded
+            return result
+        except HTTPException:
+            raise
+        except Exception as exc:
+            logger.exception("smart_mask failed")
+            raise HTTPException(500, f"SAM mask failed: {exc}") from exc
+
     @router.post("/api/image/remove-bg")
     async def remove_background(request: Request):
         """Remove background from an image. If the client passes a `hint_mask`

routes/history/history_routes.py

@@ -10,7 +10,9 @@ from fastapi import APIRouter, Request, HTTPException
 
 from core.models import ChatMessage
 from core.database import SessionLocal, ChatMessage as DbChatMessage, Session as DbSession
+from src.auth_helpers import effective_user
 from src.topic_analyzer import analyze_topics
+from src.upload_handler import reserve_message_upload_references
 from routes.session_routes import (
     _message_role,
     _message_text,
@@ -98,9 +100,29 @@ def _merge_continue_rows_to_delete(db_messages, db1, db2):
     return to_delete
 
 
-def setup_history_routes(session_manager) -> APIRouter:
+def setup_history_routes(session_manager, upload_handler=None) -> APIRouter:
     router = APIRouter(tags=["history"])
 
+    def _reserve_message_uploads(
+        request: Request,
+        content: Any,
+        metadata: Any = None,
+    ) -> None:
+        try:
+            missing_id = reserve_message_upload_references(
+                upload_handler,
+                effective_user(request),
+                content,
+                metadata,
+            )
+        except (TypeError, ValueError) as exc:
+            raise HTTPException(400, "Invalid message attachment metadata") from exc
+        if missing_id:
+            raise HTTPException(
+                409,
+                f"Referenced upload is no longer available: {missing_id}",
+            )
+
     def _db_history_entry(m: DbChatMessage) -> Dict[str, Any]:
         entry = {"role": m.role, "content": _history_display_content(m.content)}
         meta = {}
@@ -115,6 +137,44 @@ def setup_history_routes(session_manager) -> APIRouter:
             entry["metadata"] = meta
         return entry
 
+    def _db_message_metadata(m: DbChatMessage) -> Dict[str, Any]:
+        meta = {}
+        if m.meta_data:
+            try:
+                meta = json.loads(m.meta_data) or {}
+            except (json.JSONDecodeError, ValueError):
+                meta = {}
+        if m.timestamp and "timestamp" not in meta:
+            meta["timestamp"] = m.timestamp.isoformat() + "Z"
+        return meta
+
+    def _hydrate_session_history_from_db(session_id: str, rows: list[DbChatMessage]) -> None:
+        """Rebuild in-memory context from raw DB rows after a history load.
+
+        The browser history endpoint can return paged/display-trimmed messages,
+        but the next model call reads ``session.history``. After a restart or a
+        stale in-memory session, selecting an old chat through the paged endpoint
+        used to show the transcript while the model only saw fresh context.
+        """
+        if not rows:
+            return
+        try:
+            session = session_manager.get_session(session_id)
+        except KeyError:
+            return
+        session.history = [
+            ChatMessage(role=m.role, content=m.content, metadata=_db_message_metadata(m) or None)
+            for m in rows
+        ]
+        session.message_count = len(session.history)
+
+    def _session_needs_db_history_hydration(session_id: str, total: int) -> bool:
+        try:
+            session = session_manager.get_session(session_id)
+        except KeyError:
+            return False
+        return len(session.history or []) < int(total or 0)
+
     @router.get("/api/history/{session_id}")
     async def get_session_history(
         request: Request,
@@ -146,6 +206,14 @@ def setup_history_routes(session_manager) -> APIRouter:
                     .limit(page_limit)
                     .all()
                 )
+                if _session_needs_db_history_hydration(session_id, total):
+                    full_rows = (
+                        db.query(DbChatMessage)
+                        .filter(DbChatMessage.session_id == session_id)
+                        .order_by(DbChatMessage.timestamp)
+                        .all()
+                    )
+                    _hydrate_session_history_from_db(session_id, full_rows)
                 history_dict = [
                     entry for entry in (_db_history_entry(m) for m in rows)
                     if not (entry.get("metadata") or {}).get("hidden")
@@ -206,10 +274,7 @@ def setup_history_routes(session_manager) -> APIRouter:
                 if db_history:
                     # Rebuild in-memory history from the full set so hidden
                     # messages (e.g. compaction summaries) are kept for AI context.
-                    session.history = [
-                        ChatMessage(role=m["role"], content=m["content"], metadata=m.get("metadata"))
-                        for m in db_history
-                    ]
+                    _hydrate_session_history_from_db(session_id, db_messages)
                 # Response excludes hidden messages, matching the in-memory path.
                 history_dict = [
                     m for m in db_history
@@ -251,7 +316,9 @@ def setup_history_routes(session_manager) -> APIRouter:
             content = body.get("content", "")
             if not content:
                 raise HTTPException(400, "content is required")
-            msg = ChatMessage(role=role, content=content, metadata=body.get("metadata"))
+            metadata = body.get("metadata")
+            _reserve_message_uploads(request, content, metadata)
+            msg = ChatMessage(role=role, content=content, metadata=metadata)
             session_manager.add_message(session_id, msg)
             return {"status": "ok"}
         except KeyError:
@@ -331,6 +398,8 @@ def setup_history_routes(session_manager) -> APIRouter:
             if not msg_id or content is None:
                 raise HTTPException(400, "msg_id and content are required")
 
+            _reserve_message_uploads(request, content)
+
             session = session_manager.get_session(session_id)
             db = SessionLocal()
             try:
@@ -630,6 +699,55 @@ def setup_history_routes(session_manager) -> APIRouter:
         except Exception as e:
             raise HTTPException(500, f"Topic analysis failed: {e}")
 
+    @router.get("/api/session/{session_id}/context")
+    async def get_session_context_usage(request: Request, session_id: str) -> Dict[str, Any]:
+        """Return an estimated whole-chat context usage for the session's model.
+
+        Streaming footers report the prompt size for the last request. This
+        endpoint estimates the persisted session context so the header can show
+        when the whole chat is approaching compaction.
+        """
+        _verify_session_owner(request, session_id)
+        try:
+            session = session_manager.get_session(session_id)
+        except KeyError:
+            raise HTTPException(404, "Session not found")
+
+        try:
+            from src.model_context import estimate_tokens, get_context_length
+
+            messages = session.get_context_messages()
+            used = int(estimate_tokens(messages))
+            ctx_len = int(get_context_length(session.endpoint_url, session.model) or 0)
+            pct = round((used / ctx_len) * 100, 1) if ctx_len else 0.0
+            pct = max(0.0, min(100.0, pct))
+            visible_messages = sum(
+                1 for m in session.history
+                if not (getattr(m, "metadata", None) or {}).get("hidden")
+            )
+            compacted_messages = sum(
+                1 for m in session.history
+                if (getattr(m, "metadata", None) or {}).get("compacted")
+            )
+            can_compact = used > 0
+            return {
+                "session_id": session_id,
+                "model": session.model,
+                "endpoint_url": session.endpoint_url,
+                "used_tokens": used,
+                "context_length": ctx_len,
+                "context_percent": pct,
+                "messages": visible_messages,
+                "context_messages": len(messages),
+                "compacted_messages": compacted_messages,
+                "can_compact": can_compact,
+                "should_compact": pct >= 70,
+                "auto_compact_threshold": 85,
+            }
+        except Exception as e:
+            logger.error(f"Context usage error {session_id}: {e}")
+            raise HTTPException(500, str(e))
+
     @router.post("/api/session/{session_id}/compact")
     async def compact_session(request: Request, session_id: str):
         """Manually trigger context compaction for a session."""
@@ -674,7 +792,7 @@ def setup_history_routes(session_manager) -> APIRouter:
             compact_model = util_model or session.model
             compact_headers = util_headers if util_url else session.headers
 
-            from src.context_compactor import SELF_SUMMARY_SYSTEM_PROMPT
+            from src.context_compactor import SELF_SUMMARY_SYSTEM_PROMPT, normalize_compaction_summary
             compaction_count = sum(1 for m in session.history if isinstance(m, ChatMessage) and "[Conversation summary" in (m.content or ""))
             sys_prompt = SELF_SUMMARY_SYSTEM_PROMPT.replace("{count}", str(len(older))).replace("{n}", str(compaction_count + 1))
             summary = await llm_call_async(
@@ -686,6 +804,7 @@ def setup_history_routes(session_manager) -> APIRouter:
                 temperature=0.2, max_tokens=1024,
                 headers=compact_headers, timeout=30,
             )
+            summary = normalize_compaction_summary(summary)
 
             # Replace session history: summary as system message + recent messages
             # System message holds the full summary for AI context

routes/hwfit_routes.py

@@ -429,11 +429,27 @@ def setup_hwfit_routes():
             system["available_ram_gb"] = 0
             system["total_ram_gb"] = 0
         system = _apply_manual_hardware(system, manual_mode, manual_gpu_count, manual_vram_gb, manual_ram_gb, manual_backend)
-        # Image models use a single GPU — always use per-GPU VRAM
-        gpu_vrams = [float(g.get("vram_gb") or 0) for g in (system.get("gpus") or []) if isinstance(g, dict)]
-        single_vram = max(gpu_vrams) if gpu_vrams else ((system.get("gpu_vram_gb") or 0) / max(system.get("gpu_count") or 1, 1))
-        system["gpu_vram_gb"] = single_vram
-        system["gpu_count"] = 1 if single_vram > 0 else 0
+        try:
+            requested_gpu_count = int(gpu_count) if gpu_count != "" else None
+        except ValueError:
+            requested_gpu_count = None
+        if requested_gpu_count == 0:
+            # Respect the UI's RAM toggle. Before this route always rewrote the
+            # system to best-single-GPU VRAM, so image rows never changed when
+            # switching RAM/GPU.
+            system["has_gpu"] = False
+            system["gpu_vram_gb"] = 0
+            system["gpu_count"] = 0
+            system["gpu_only"] = False
+        else:
+            # Image diffusion backends generally use one device per pipeline,
+            # so rank GPU mode against the best single GPU rather than total
+            # multi-GPU VRAM.
+            gpu_vrams = [float(g.get("vram_gb") or 0) for g in (system.get("gpus") or []) if isinstance(g, dict)]
+            single_vram = max(gpu_vrams) if gpu_vrams else ((system.get("gpu_vram_gb") or 0) / max(system.get("gpu_count") or 1, 1))
+            system["gpu_vram_gb"] = single_vram
+            system["gpu_count"] = 1 if single_vram > 0 else 0
+            system["gpu_only"] = True if single_vram > 0 else False
         results = rank_image_models(system, search=search or None, sort=sort)
         return {"system": system, "models": results}
 

routes/model_routes.py

@@ -472,7 +472,11 @@ def _endpoint_kind(ep: Any) -> str:
 
 
 def _endpoint_refresh_mode(ep: Any, endpoint_kind: str | None = None) -> str:
-    return _normalize_refresh_mode(getattr(ep, "model_refresh_mode", None), endpoint_kind or _endpoint_kind(ep))
+    return _normalize_endpoint_refresh_mode(
+        getattr(ep, "model_refresh_mode", None),
+        endpoint_kind or _endpoint_kind(ep),
+        getattr(ep, "base_url", ""),
+    )
 
 
 def _endpoint_refresh_interval(ep: Any, category: str) -> float:
@@ -851,6 +855,99 @@ def _ollama_model_names(data: Any) -> List[str]:
     return out
 
 
+def _is_google_api_base(base_url: str) -> bool:
+    try:
+        return (urlparse(base_url).hostname or "").lower() == "generativelanguage.googleapis.com"
+    except Exception:
+        return False
+
+
+def _normalize_endpoint_refresh_mode(value: Any, endpoint_kind: str = "auto", base_url: str = "") -> str:
+    if not str(value or "").strip() and _is_google_api_base(base_url):
+        return "manual"
+    return _normalize_refresh_mode(value, endpoint_kind)
+
+
+def _google_native_root(base_url: str) -> str:
+    """Return the Gemini native API root for a Google endpoint.
+
+    Chat calls may be configured against Google's OpenAI-compatible
+    `/openai` path, but model catalog reads should use the native Models API
+    so we get Google's current Model resource shape.
+    """
+    try:
+        parsed = urlparse(base_url)
+    except Exception:
+        return "https://generativelanguage.googleapis.com/v1beta"
+    path = (parsed.path or "").rstrip("/")
+    if path.endswith("/openai"):
+        path = path[: -len("/openai")].rstrip("/")
+    if not path:
+        path = "/v1beta"
+    return urlunparse(parsed._replace(path=path, query="", fragment="")).rstrip("/")
+
+
+def _google_native_models_url(base_url: str) -> str:
+    return _google_native_root(base_url) + "/models"
+
+
+def _google_model_id_from_item(item: Any) -> str:
+    if not isinstance(item, dict):
+        return ""
+    value = item.get("baseModelId") or item.get("name") or item.get("model") or ""
+    return str(value or "").strip().removeprefix("models/")
+
+
+def _google_model_supports_chat(item: Any) -> bool:
+    """Return whether a native Google Model resource supports chat generation."""
+    if not isinstance(item, dict):
+        return False
+    methods = item.get("supportedGenerationMethods")
+    if not isinstance(methods, list):
+        return False
+    chat_methods = {"generateContent", "generateMessage", "generateText", "generateAnswer"}
+    return any(method in chat_methods for method in methods)
+
+
+def _probe_google_models(base_url: str, api_key: str = None, timeout: int = 5, page_size: int = 1000) -> List[str]:
+    """Read Google's native paginated Models API.
+
+    This intentionally returns only provider-reported model IDs. Capability
+    mapping is handled by the model capability reader and must not infer from
+    names here.
+    """
+    url = _google_native_models_url(base_url)
+    try:
+        page_size = min(max(int(page_size or 1000), 1), 1000)
+    except Exception:
+        page_size = 1000
+    headers = {"Accept": "application/json"}
+    if api_key:
+        headers["x-goog-api-key"] = api_key
+    params: Dict[str, Any] = {"pageSize": page_size}
+    models: List[str] = []
+    seen = set()
+    page_token = ""
+    for _ in range(20):
+        request_params = dict(params)
+        if page_token:
+            request_params["pageToken"] = page_token
+        r = httpx.get(url, headers=headers, params=request_params, timeout=timeout, verify=llm_verify())
+        r.raise_for_status()
+        data = r.json()
+        for item in data.get("models") or []:
+            if not _google_model_supports_chat(item):
+                continue
+            model_id = _google_model_id_from_item(item)
+            if model_id and model_id not in seen:
+                seen.add(model_id)
+                models.append(model_id)
+        page_token = str(data.get("nextPageToken") or "").strip()
+        if not page_token:
+            break
+    return models
+
+
 def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> List[str]:
     """Probe a base URL's /models endpoint and return list of model IDs.
     For Anthropic, queries their /v1/models API, falling back to hardcoded list."""
@@ -863,6 +960,17 @@ def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> Lis
         if api_key:
             return fetch_available_models(api_key, timeout=timeout)
         return []
+    if _is_google_api_base(base):
+        try:
+            models = _probe_google_models(base, api_key, timeout=timeout)
+            if models:
+                return models
+        except httpx.HTTPStatusError as e:
+            status = e.response.status_code if e.response is not None else "unknown"
+            logger.warning(f"Google native models probe failed: HTTP {status}")
+        except Exception as e:
+            logger.warning(f"Google native models probe failed: {e}")
+        return []
     if provider == "anthropic":
         # Try Anthropic's /v1/models endpoint first
         url = _safe_build_models_url(base)
@@ -1202,6 +1310,56 @@ def _visible_models(cached_models, hidden_models, pinned_models=None):
     return [m for m in merged if m not in hidden]
 
 
+def _picker_requires_pinning(base_url: str, kind: str) -> bool:
+    return _classify_endpoint(base_url, kind) == "api"
+
+
+def _has_explicit_pinned_models(ep) -> bool:
+    """Whether pinned_models was deliberately written for this endpoint.
+
+    API endpoints use pinned_models as an allow-list. An explicit empty JSON
+    list means "show no models"; it must not fall back to the old hidden-list
+    migration behavior.
+    """
+    raw = getattr(ep, "pinned_models", None)
+    return raw is not None and str(raw).strip() != ""
+
+
+def _legacy_visible_api_models(ep) -> List[str]:
+    """Return API models selected under the old hidden-list picker.
+
+    Before API endpoints switched to an explicit allow-list, selected models
+    were represented as cached_models minus hidden_models. Existing OpenRouter
+    rows can therefore have many checked models and an empty pinned_models
+    field. Treat that old state as the initial pinned list so settings and chat
+    agree after upgrade.
+    """
+    return _visible_models(
+        _cached_model_ids(ep),
+        getattr(ep, "hidden_models", None),
+        None,
+    )
+
+
+def _picker_models_for_endpoint(ep, base_url: str, kind: str):
+    """Return model IDs that should appear in the picker for an endpoint.
+
+    API providers expose remote inventory from /v1/models. Treat that cache as
+    inventory, not approval: only manually pinned API models should appear in
+    the picker. Local/self-hosted endpoints keep the older hide-list behavior.
+    """
+    pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
+    if _picker_requires_pinning(base_url, kind):
+        if not _has_explicit_pinned_models(ep):
+            pinned = _legacy_visible_api_models(ep) if _hidden_model_ids(ep) else []
+        return pinned, pinned
+    return _visible_models(
+        _cached_model_ids(ep),
+        getattr(ep, "hidden_models", None),
+        pinned,
+    ), pinned
+
+
 def _api_key_fingerprint(api_key: Optional[str]) -> str:
     """Stable, non-secret label for distinguishing same-URL credentials."""
     key = (api_key or "").strip()
@@ -1400,24 +1558,18 @@ def setup_model_routes(model_discovery):
         for ep in endpoints:
             base = _normalize_base(ep.base_url)
             provider = _safe_detect_provider(base)
-            # Merge cached + pinned models, then filter out hidden ones
             ep_model_type = getattr(ep, "model_type", None) or "llm"
-            model_ids = _visible_models(
-                _cached_model_ids(ep),
-                ep.hidden_models,
-                getattr(ep, "pinned_models", None),
-            )
             # Build correct URL based on provider
             chat_url = build_chat_url(base)
             kind = _effective_endpoint_kind(ep, base)
             category = _classify_endpoint(base, kind)
+            model_ids, pinned = _picker_models_for_endpoint(ep, base, kind)
 
             if model_ids:
                 curated_key = _match_provider_curated(base, None)
                 curated, extra = _curate_models(model_ids, curated_key)
                 # Pinned models are admin-selected — they always belong in the
                 # primary curated list, not buried in extras.
-                pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
                 for m in pinned:
                     if m not in curated:
                         curated.append(m)
@@ -1779,18 +1931,24 @@ def setup_model_routes(model_discovery):
                 _invalidate_models_cache()
             rows = db.query(ModelEndpoint).order_by(ModelEndpoint.created_at).all()
             results = []
+            upgraded_legacy_pins = False
             for r in rows:
                 all_models = _cached_model_ids(r)
                 hidden = _hidden_model_ids(r)
                 pinned = _normalize_model_ids(getattr(r, "pinned_models", None))
-                visible = _visible_models(all_models, r.hidden_models, pinned)
                 # Keep the list route cache-only. It feeds Settings →
                 # Added Models and must render immediately; explicit
                 # Refresh/Probe endpoints do the network work.
-                status = "online" if (all_models or pinned) else ("empty" if r.is_enabled else "offline")
                 ping = None
                 base = _normalize_base(r.base_url)
                 kind = _effective_endpoint_kind(r, base)
+                visible, pinned = _picker_models_for_endpoint(r, base, kind)
+                if _picker_requires_pinning(base, kind) and pinned and not _has_explicit_pinned_models(r):
+                    r.pinned_models = json.dumps(pinned)
+                    upgraded_legacy_pins = True
+                model_inventory_count = len(_merge_model_ids(all_models, pinned))
+                picker_requires_pinning = _picker_requires_pinning(base, kind)
+                status = "online" if (all_models or visible or pinned) else ("empty" if r.is_enabled else "offline")
                 results.append({
                     "id": r.id,
                     "name": r.name,
@@ -1799,6 +1957,8 @@ def setup_model_routes(model_discovery):
                     "api_key_fingerprint": _api_key_fingerprint(r.api_key),
                     "is_enabled": r.is_enabled,
                     "models": visible,
+                    "model_count": model_inventory_count,
+                    "picker_requires_pinning": picker_requires_pinning,
                     "pinned_models": pinned,
                     "hidden_count": len(hidden),
                     "online": status != "offline",
@@ -1812,6 +1972,9 @@ def setup_model_routes(model_discovery):
                     "model_refresh_interval": getattr(r, "model_refresh_interval", None),
                     "model_refresh_timeout": getattr(r, "model_refresh_timeout", None),
                 })
+            if upgraded_legacy_pins:
+                db.commit()
+                _invalidate_models_cache()
             return results
         finally:
             db.close()
@@ -1854,7 +2017,7 @@ def setup_model_routes(model_discovery):
             name = base_url.replace("http://", "").replace("https://", "").split("/")[0]
 
         requested_kind = _normalize_endpoint_kind(endpoint_kind)
-        refresh_mode = _normalize_refresh_mode(model_refresh_mode, requested_kind)
+        refresh_mode = _normalize_endpoint_refresh_mode(model_refresh_mode, requested_kind, base_url)
         refresh_interval = _parse_positive_int(model_refresh_interval, minimum=30, maximum=86400)
         refresh_timeout = _parse_positive_int(model_refresh_timeout, minimum=1, maximum=60)
         require_model_list = _truthy(require_models)
@@ -1915,6 +2078,10 @@ def setup_model_routes(model_discovery):
                 if refresh_timeout is not None:
                     existing.model_refresh_timeout = refresh_timeout
                     changed = True
+                incoming_model_type = (model_type or "").strip() or "llm"
+                if incoming_model_type and (getattr(existing, "model_type", None) or "llm") != incoming_model_type:
+                    existing.model_type = incoming_model_type
+                    changed = True
                 if api_key.strip() and not existing.api_key:
                     existing.api_key = api_key.strip()
                     changed = True
@@ -2147,9 +2314,10 @@ def setup_model_routes(model_discovery):
                 raise HTTPException(404, "Endpoint not found")
             hidden = _hidden_model_ids(ep)
             all_models = _cached_model_ids(ep)
+            base = _normalize_base(ep.base_url)
+            kind = _effective_endpoint_kind(ep, base)
+            picker_requires_pinning = _picker_requires_pinning(base, kind)
             if refresh:
-                base = _normalize_base(ep.base_url)
-                kind = _effective_endpoint_kind(ep, base)
                 category = _classify_endpoint(base, kind)
                 timeout = _manual_refresh_timeout(ep, category, refresh_timeout)
                 try:
@@ -2168,6 +2336,8 @@ def setup_model_routes(model_discovery):
                     response.headers["X-Model-Refresh-Status"] = "failed"
                     response.headers["X-Model-Refresh-Warning"] = "Model refresh failed or returned no models; kept cached models."
             pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
+            if picker_requires_pinning and not _has_explicit_pinned_models(ep):
+                pinned = _legacy_visible_api_models(ep)
             pinned_set = set(pinned)
             return [
                 {
@@ -2175,6 +2345,7 @@ def setup_model_routes(model_discovery):
                     "display": m.split("/")[-1],
                     "is_hidden": m in hidden,
                     "is_pinned": m in pinned_set,
+                    "picker_requires_pinning": picker_requires_pinning,
                 }
                 for m in _merge_model_ids(all_models, pinned)
             ]
@@ -2203,11 +2374,28 @@ def setup_model_routes(model_discovery):
                 hidden = body.get("hidden")
                 if not isinstance(hidden, list):
                     raise HTTPException(400, "hidden must be a list of model IDs")
-                ep.hidden_models = json.dumps(hidden) if hidden else None
+                base = _normalize_base(ep.base_url)
+                kind = _effective_endpoint_kind(ep, base)
+                if _picker_requires_pinning(base, kind):
+                    # Compatibility for older/admin UI paths that still submit
+                    # the previous hide-list shape. API pickers are allow-lists:
+                    # convert "unchecked models" into an explicit pinned list so
+                    # Settings summary, /api/models, and chat agree.
+                    selected = _visible_models(_cached_model_ids(ep), hidden, None)
+                    ep.pinned_models = json.dumps(selected)
+                    ep.hidden_models = None
+                else:
+                    ep.hidden_models = json.dumps(hidden) if hidden else None
             # Accept either "pinned" or "pinned_models" for the manual IDs list.
             if "pinned_models" in body or "pinned" in body:
                 pinned = _normalize_model_ids(body.get("pinned_models", body.get("pinned")))
-                ep.pinned_models = json.dumps(pinned) if pinned else None
+                base = _normalize_base(ep.base_url)
+                kind = _effective_endpoint_kind(ep, base)
+                if _picker_requires_pinning(base, kind):
+                    ep.pinned_models = json.dumps(pinned)
+                    ep.hidden_models = None
+                else:
+                    ep.pinned_models = json.dumps(pinned) if pinned else None
             db.commit()
             _invalidate_models_cache()
             hidden_count = len(json.loads(ep.hidden_models)) if ep.hidden_models else 0
@@ -2360,11 +2548,21 @@ def setup_model_routes(model_discovery):
                     ep.model_type = body["model_type"].strip() or ep.model_type
                 if "pinned_models" in body:
                     _pinned = _normalize_model_ids(body["pinned_models"])
-                    ep.pinned_models = json.dumps(_pinned) if _pinned else None
+                    _base_for_pins = _normalize_base(ep.base_url)
+                    _kind_for_pins = _effective_endpoint_kind(ep, _base_for_pins)
+                    if _picker_requires_pinning(_base_for_pins, _kind_for_pins):
+                        ep.pinned_models = json.dumps(_pinned)
+                        ep.hidden_models = None
+                    else:
+                        ep.pinned_models = json.dumps(_pinned) if _pinned else None
                 if "endpoint_kind" in body:
                     ep.endpoint_kind = _normalize_endpoint_kind(body.get("endpoint_kind"))
                 if "model_refresh_mode" in body:
-                    ep.model_refresh_mode = _normalize_refresh_mode(body.get("model_refresh_mode"), _endpoint_kind(ep))
+                    ep.model_refresh_mode = _normalize_endpoint_refresh_mode(
+                        body.get("model_refresh_mode"),
+                        _endpoint_kind(ep),
+                        ep.base_url,
+                    )
                 if "model_refresh_interval" in body:
                     interval = _parse_positive_int(body.get("model_refresh_interval"), minimum=30, maximum=86400)
                     ep.model_refresh_interval = interval

routes/note/__init__.py

@@ -0,0 +1,5 @@
+"""Note route domain package (slice 2f, #4082/#4071).
+
+Contains note_routes.py, migrated from the flat routes/ directory.
+Backward-compat shim at routes/note_routes.py re-exports from here.
+"""

routes/note/note_routes.py

@@ -0,0 +1,937 @@
+# routes/note_routes.py
+"""Google Keep-style notes / checklists API."""
+
+import json
+import uuid
+import logging
+from typing import Dict, Any, Optional
+
+from fastapi import APIRouter, HTTPException, Request
+from pydantic import BaseModel
+
+from core.database import SessionLocal, Note
+from core.middleware import INTERNAL_TOOL_USER
+from src.auth_helpers import require_user
+from src.constants import DATA_DIR
+from src.upload_handler import reserve_upload_references
+from sqlalchemy.orm.attributes import flag_modified
+
+logger = logging.getLogger(__name__)
+
+
+# ---------------------------------------------------------------------------
+# Request models
+# ---------------------------------------------------------------------------
+
+class NoteCreate(BaseModel):
+    title: str = ""
+    content: Optional[str] = None
+    items: Optional[list] = None
+    note_type: str = "note"
+    color: Optional[str] = None
+    label: Optional[str] = None
+    pinned: bool = False
+    due_date: Optional[str] = None
+    source: str = "user"
+    session_id: Optional[str] = None
+    image_url: Optional[str] = None
+    repeat: Optional[str] = "none"
+    sort_order: Optional[int] = None
+
+
+class NoteUpdate(BaseModel):
+    title: Optional[str] = None
+    content: Optional[str] = None
+    items: Optional[list] = None
+    note_type: Optional[str] = None
+    color: Optional[str] = None
+    label: Optional[str] = None
+    pinned: Optional[bool] = None
+    archived: Optional[bool] = None
+    due_date: Optional[str] = None
+    image_url: Optional[str] = None
+    repeat: Optional[str] = None
+    sort_order: Optional[int] = None
+    agent_session_id: Optional[str] = None
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _note_to_dict(note: Note) -> Dict[str, Any]:
+    items = None
+    if note.items:
+        try:
+            items = json.loads(note.items)
+        except (json.JSONDecodeError, TypeError):
+            items = None
+    ai_cls = None
+    raw_ai = getattr(note, "ai_classification", None)
+    if raw_ai:
+        try:
+            ai_cls = json.loads(raw_ai)
+        except (json.JSONDecodeError, TypeError):
+            ai_cls = None
+    return {
+        "id": note.id,
+        "owner": note.owner,
+        "title": note.title,
+        "content": note.content,
+        "items": items,
+        "note_type": note.note_type,
+        "color": note.color,
+        "label": note.label,
+        "pinned": note.pinned,
+        "archived": note.archived,
+        "due_date": note.due_date,
+        "source": note.source,
+        "session_id": note.session_id,
+        "sort_order": note.sort_order or 0,
+        "image_url": note.image_url,
+        "repeat": note.repeat or "none",
+        "ai_classification": ai_cls,
+        "ai_content_hash": getattr(note, "ai_content_hash", None),
+        "agent_session_id": getattr(note, "agent_session_id", None),
+        "created_at": note.created_at.isoformat() if note.created_at else None,
+        "updated_at": note.updated_at.isoformat() if note.updated_at else None,
+    }
+
+
+def _reminder_text_from_note(note: Note) -> tuple[str, str]:
+    """Return the reminder title/body from a stored note row."""
+    title = (note.title or "Note reminder").strip() or "Note reminder"
+    if note.items:
+        try:
+            items = json.loads(note.items)
+        except (json.JSONDecodeError, TypeError):
+            items = None
+        if isinstance(items, list):
+            pending: list[str] = []
+            for item in items:
+                if not isinstance(item, dict):
+                    continue
+                if item.get("done") or item.get("checked"):
+                    continue
+                text = str(item.get("text") or "").strip()
+                if text:
+                    pending.append(text)
+            if pending:
+                shown = "\n".join(f"- {text}" for text in pending[:8])
+                extra = f"\n...and {len(pending) - 8} more" if len(pending) > 8 else ""
+                return title, f"Pending ({len(pending)}):\n{shown}{extra}"
+            return title, f"{len(items)} item{'s' if len(items) != 1 else ''}"
+    return title, (note.content or "").strip()[:400]
+
+
+
+# ---------------------------------------------------------------------------
+# Reminder dispatch — module-level so background tasks (built-in actions)
+# can call it directly without an HTTP roundtrip + auth cookie. The route
+# version below is a thin wrapper that pulls `owner` from the request.
+# ---------------------------------------------------------------------------
+
+# Scheduler reference — set by setup_note_routes() so dispatch_reminder can
+# push a parallel in-app notification (frontend polls the scheduler's queue
+# and fires real browser Notification(...) popups). Optional; works without it.
+_scheduler_ref = None
+
+
+async def dispatch_reminder(
+    title: str,
+    note_body: str,
+    note_id: str,
+    owner: str = "",
+    queue_browser: bool = True,
+    settings_override: dict | None = None,
+) -> dict:
+    """Fire a reminder via the configured channel (browser/email/ntfy/webhook).
+
+    Args:
+        title: short headline shown to the user
+        note_body: longer body text
+        note_id: stable id (used as tag/dedupe in browser notifications)
+        owner: the user this reminder belongs to — scopes SMTP config to
+               their account so we don't cross-leak credentials
+
+    Returns: {synthesis, email_sent, ntfy_sent}. Browser channel is wired via
+    the in-memory notification queue picked up by the frontend poller, so
+    nothing is "sent" synchronously for it — the channel just routes there.
+    """
+    from src.settings import load_settings
+    settings = {**load_settings(), **(settings_override or {})}
+    channel = settings.get("reminder_channel", "browser")
+    llm_on = bool(settings.get("reminder_llm_synthesis", False))
+    title = (title or "").strip()
+    note_body = (note_body or "").strip()
+    cache_key = str(note_id) if note_id else ""
+    cache = {}
+    cache_path = None
+    if cache_key:
+        try:
+            import json as _json
+            from datetime import datetime as _dt, timezone as _tz, timedelta as _td
+            from pathlib import Path as _P
+            _slug = "".join(c if (c.isalnum() or c in "-_.@") else "_" for c in (owner or "default"))
+            cache_path = _P(DATA_DIR) / f"note_pings_{_slug}.json"
+            if cache_path.exists():
+                cache = _json.loads(cache_path.read_text(encoding="utf-8"))
+            last = cache.get(cache_key)
+            if last:
+                last_channel = None
+                if isinstance(last, dict):
+                    last_channel = last.get("channel")
+                    last = last.get("at")
+                last_dt = _dt.fromisoformat(str(last))
+                if last_dt.tzinfo is None:
+                    last_dt = last_dt.replace(tzinfo=_tz.utc)
+                # Legacy cache values were plain timestamps and could be
+                # written by the frontend even when the email/ntfy send failed.
+                # Treat those as browser-only dedupe so email reminders can be
+                # retried by the backend scanner after a failed frontend path.
+                should_skip = last_dt >= _dt.now(_tz.utc) - _td(minutes=25)
+                if should_skip and channel in ("email", "ntfy", "webhook"):
+                    should_skip = last_channel == channel
+                if should_skip:
+                    return {
+                        "synthesis": None,
+                        "email_sent": False,
+                        "ntfy_sent": False,
+                        "webhook_sent": False,
+                        "browser_sent": True,
+                        "skipped": True,
+                    }
+        except Exception as _e:
+            logger.debug(f"dispatch_reminder: cache read failed: {_e}")
+
+    synthesis = None
+    _SYNTH_FAILED_TAG = "[utility model unavailable — no summary generated]"
+    if llm_on:
+        try:
+            from src.endpoint_resolver import resolve_endpoint
+            from src.llm_core import llm_call_async
+            from src.reminder_personas import synthesis_system_prompt
+            url, model, headers = resolve_endpoint("utility", owner=owner or None)
+            if not url:
+                url, model, headers = resolve_endpoint("default", owner=owner or None)
+            if url and model:
+                persona_id = (settings.get("reminder_llm_persona") or "").strip()
+                sys_prompt = synthesis_system_prompt(persona_id)
+                raw = await llm_call_async(
+                    url=url, model=model,
+                    messages=[
+                        {"role": "system", "content": sys_prompt},
+                        {"role": "user", "content": f"Title: {title}\n\n{note_body}".strip()},
+                    ],
+                    temperature=0.7, max_tokens=200, headers=headers, timeout=30,
+                )
+                from src.text_helpers import strip_think as _strip_think
+                # prose=True strips untagged "The user wants me to…" chain-of-thought.
+                # prompt_echo=True strips Qwen-style "Thinking Process:" / leaked
+                # prompt prefixes. Both are safe here because this is a
+                # one-sentence LLM-only output, not user-pasted content.
+                synthesis = _strip_think(raw or "", prose=True, prompt_echo=True)
+                # Reminder synthesis is supposed to be ONE sentence. Strip-think's
+                # paragraph-based heuristic misses cases where the model puts
+                # reasoning + answer on consecutive lines inside one paragraph
+                # (e.g. "I should write... [\n] You have one task waiting...").
+                # Walk lines, drop reasoning/prompt-echo lines, then keep the
+                # last surviving line — that's the actual warm sentence.
+                if synthesis:
+                    import re as _re
+                    # Tightened: target ACTUAL self-talk (model narrating what
+                    # it'll do) rather than any first-person sentence. The old
+                    # pattern killed legit warm sentences like "I'll see you
+                    # tomorrow" or "I should be done by then". New rules:
+                    #  • "I (need|should|have|'ll|will) (write|draft|reply|…)"
+                    #    only matches when followed by a TASK verb taking an
+                    #    OBJECT (so first-person + intransitive verb passes).
+                    #  • Self-instructional patterns the model emits verbatim:
+                    #    "I should write something that reminds them…",
+                    #    "I need to draft…", "Let me think…".
+                    #  • Explicit instructions echoed back from the prompt:
+                    #    "Keep it under 25 words", "No greetings".
+                    _reasoning = _re.compile(
+                        r"^\s*(?:"
+                        # "I should write/draft/compose…" with a task-object follow
+                        r"i (?:need|should|have|'ll|will|am going|am)\s+to\s+"
+                        r"(?:write|draft|compose|craft|generate|produce|create|"
+                        r"summarize|answer|provide|note|address|remind|output)"
+                        r"\s+(?:a |an |the |something|this|that|here|them|him|her|"
+                        r"you|user|reply|response|sentence|message|line|warm)|"
+                        # The model literally narrating about the user
+                        r"the user (?:wants|is asking|asks|needs|wrote|said|requested) (?:me )?(?:to|for|that|about|something)|"
+                        # "Let me [think/write/draft/…] (about/for/the …)"
+                        r"let me (?:think|write|draft|consider|note|see|check)\b\s+(?:about|for|the|this|that|if|whether)|"
+                        # "Looking at the/this/that …"
+                        r"looking at (?:the|this|that)\b|"
+                        # "Based on the/this/what …"
+                        r"based on (?:the|this|what|context|that)\b|"
+                        # Prompt-echo of length / style instructions
+                        r"keep it under \d+ words\b|"
+                        r"(?:no greetings|no preamble|no hashtags|just output the)\b"
+                        r").*",
+                        _re.IGNORECASE,
+                    )
+                    # Echo of the prompt's "Pending:" / "<N> pending" tail.
+                    _echo = _re.compile(
+                        r"^\s*(?:pending\s*[:.]|(?:\d+|one|two|three|four|five)\s+pending\b)",
+                        _re.IGNORECASE,
+                    )
+                    lines = [ln for ln in synthesis.splitlines() if ln.strip()]
+                    cleaned = [ln for ln in lines if not _reasoning.match(ln) and not _echo.match(ln)]
+                    if cleaned:
+                        # The model's actual answer is normally the LAST surviving
+                        # line — reasoning leads, answer trails.
+                        synthesis = cleaned[-1].strip()
+            else:
+                synthesis = _SYNTH_FAILED_TAG
+        except Exception as e:
+            logger.warning(f"Reminder LLM synthesis failed: {e}")
+            synthesis = _SYNTH_FAILED_TAG
+        if synthesis:
+            _s = synthesis.strip(); _low = _s.lower()
+            if (not _s or _low.startswith("error:") or _low.startswith("[error")
+                    or "operation failed" in _low
+                    or ("upstream" in _low and "failed" in _low)) and synthesis != _SYNTH_FAILED_TAG:
+                logger.warning(f"Reminder synthesis looked like an error, replacing: {_s[:120]!r}")
+                synthesis = _SYNTH_FAILED_TAG
+
+    email_sent = False
+    email_error = ""
+    if channel == "email":
+        try:
+            from routes.email_routes import _get_email_config, _smtp_ready
+            from email.mime.text import MIMEText
+            from email.mime.multipart import MIMEMultipart
+            from datetime import datetime as _dt
+            # `reminder_email_account_id` lets the user pick WHICH email
+            # account to send reminders from (when they have several
+            # configured in Integrations). Falls back to the default
+            # account when no explicit choice is saved.
+            _acc_id = (settings.get("reminder_email_account_id") or "").strip() or None
+            cfg = _get_email_config(account_id=_acc_id, owner=owner or "")
+            if not _smtp_ready(cfg):
+                try:
+                    from core.database import SessionLocal as _SL, EmailAccount as _EA
+                    from sqlalchemy import and_, or_
+                    db = _SL()
+                    try:
+                        q = db.query(_EA).filter(_EA.enabled == True)  # noqa: E712
+                        if owner:
+                            unowned = or_(_EA.owner == None, _EA.owner == "")  # noqa: E711
+                            same_mailbox = or_(_EA.imap_user == owner, _EA.from_address == owner)
+                            q = q.filter(or_(_EA.owner == owner, and_(unowned, same_mailbox)))
+                        for row in q.order_by(_EA.is_default.desc(), _EA.created_at.asc()).all():
+                            trial = _get_email_config(account_id=row.id, owner=owner or "")
+                            if _smtp_ready(trial):
+                                cfg = trial
+                                break
+                    finally:
+                        db.close()
+                except Exception as _fallback_error:
+                    logger.debug(f"Reminder SMTP fallback lookup failed: {_fallback_error}")
+            from_addr = (cfg.get("from_address") or cfg.get("smtp_user") or "").strip()
+            recipient = (settings.get("reminder_email_to") or "").strip() or from_addr
+            # Loud diagnostic so we can see WHY a reminder didn't send (the
+            # previous "silently no-op when cfg has no smtp_host" was invisible).
+            logger.info(
+                "dispatch_reminder[email] note_id=%s owner=%r "
+                "has_smtp_host=%s has_smtp_user=%s has_from=%s has_recipient=%s",
+                note_id, owner,
+                bool(cfg.get("smtp_host")), bool(cfg.get("smtp_user")),
+                bool(from_addr), bool(recipient),
+            )
+            missing = []
+            if not cfg.get("smtp_host"):
+                missing.append("SMTP host")
+            if not cfg.get("smtp_user"):
+                missing.append("SMTP user")
+            if not (cfg.get("smtp_password") or cfg.get("oauth_provider")):
+                missing.append("SMTP credentials")
+            if not from_addr:
+                missing.append("from address")
+            if not recipient:
+                missing.append("recipient")
+            if missing:
+                email_error = "Missing " + ", ".join(missing)
+                logger.warning(
+                    "Reminder email not sent for note_id=%s account=%r: %s",
+                    note_id, cfg.get("account_name"), email_error,
+                )
+            else:
+                msg = MIMEMultipart("alternative")
+                msg["From"] = from_addr
+                msg["To"] = recipient
+                _t = title or 'Note'
+                _t = _t[len('Reminder:'):].strip() if _t.lower().startswith('reminder:') else _t
+                msg["Subject"] = f"Reminder (Odysseus): {_t}"
+                msg["Date"] = _dt.utcnow().strftime("%a, %d %b %Y %H:%M:%S +0000")
+                msg["X-Odysseus-Origin"] = "odysseus-ui"
+                msg["X-Odysseus-Kind"] = "reminder"
+                msg["X-Odysseus-Ref"] = str(note_id)
+                # Body shape: synthesis (warm sentence) → blank line → bold
+                # title header → note details. The title was previously only
+                # in the subject line, so the email read like a faceless
+                # to-do list with no anchor to which note triggered it.
+                _body_chunks = []
+                if synthesis:
+                    _body_chunks.append(synthesis)
+                if _t:
+                    _body_chunks.append(_t)
+                if note_body:
+                    _body_chunks.append(note_body)
+                plain = "\n\n".join(_body_chunks) if _body_chunks else title
+                msg.attach(MIMEText(plain, "plain", "utf-8"))
+
+                def _smtp_send():
+                    from routes.email_helpers import _send_smtp_message
+                    _send_smtp_message(cfg, from_addr, [recipient], msg.as_string())
+
+                import asyncio as _aio
+                await _aio.to_thread(_smtp_send)
+                email_sent = True
+        except Exception as e:
+            email_error = str(e) or e.__class__.__name__
+            logger.warning(f"Reminder email send failed: {e}")
+
+    webhook_sent = False
+    webhook_error = ""
+    if channel == "webhook":
+        try:
+            import httpx
+            import json as _wjson
+            from src.integrations import load_integrations
+            # Built-in payload defaults for known presets so users don't have
+            # to configure a template just to use a standard service.
+            _PRESET_TEMPLATE_DEFAULTS = {
+                "discord_webhook": '{"embeds": [{"title": "{{title}}", "description": "{{message}}", "color": 5793266}]}',
+            }
+            intg_id = settings.get("reminder_webhook_integration_id", "").strip()
+            template = settings.get("reminder_webhook_payload_template", "").strip()
+            if not intg_id:
+                webhook_error = "No webhook integration selected"
+            else:
+                intg = next(
+                    (i for i in load_integrations()
+                     if i.get("id") == intg_id and i.get("base_url")),
+                    None,
+                )
+                if not intg:
+                    webhook_error = f"Integration {intg_id!r} not found or missing base URL"
+                else:
+                    # Fall back to a built-in default for known presets so
+                    # users don't have to configure a template for standard
+                    # services like Discord.
+                    if not template:
+                        template = _PRESET_TEMPLATE_DEFAULTS.get(intg.get("preset", ""), "")
+                    if not template:
+                        webhook_error = "No payload template configured"
+                    else:
+                        # Render template: JSON-escape the values so the result
+                        # is always valid JSON regardless of special characters.
+                        # dumps() returns `"value"` — strip outer quotes.
+                        msg = (synthesis or note_body or title or "Reminder")[:4000]
+                        _t = _wjson.dumps(title or "Reminder")[1:-1]
+                        _m = _wjson.dumps(msg)[1:-1]
+                        rendered = template.replace("{{title}}", _t).replace("{{message}}", _m)
+                        hdrs = {"Content-Type": "application/json"}
+                        api_key = intg.get("api_key", "")
+                        auth_type = (intg.get("auth_type") or "none").lower()
+                        if api_key:
+                            if auth_type == "bearer":
+                                hdrs["Authorization"] = f"Bearer {api_key}"
+                            elif auth_type == "header":
+                                hdrs[intg.get("auth_header") or "Authorization"] = api_key
+                        url = intg["base_url"].rstrip("/")
+                        # SSRF guard — matches the pattern used by webhook_routes,
+                        # CalDAV, search, and embeddings. Blocks link-local / metadata
+                        # addresses (169.254.x.x) by default; set
+                        # REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS=true to also block
+                        # RFC-1918 ranges for locked-down deployments.
+                        import os as _os
+                        from src.url_safety import check_outbound_url as _chk
+                        _block = _os.getenv("REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS", "false").lower() == "true"
+                        _ok, _reason = _chk(url, block_private=_block)
+                        if not _ok:
+                            webhook_error = f"Webhook URL rejected: {_reason}"
+                        else:
+                            async with httpx.AsyncClient(timeout=10.0) as client:
+                                resp = await client.post(url, content=rendered.encode(), headers=hdrs)
+                                webhook_sent = resp.is_success
+                                if not webhook_sent:
+                                    webhook_error = f"Webhook returned HTTP {resp.status_code}"
+        except Exception as e:
+            webhook_error = str(e) or e.__class__.__name__
+            logger.warning(f"Reminder webhook send failed: {e}")
+
+    ntfy_sent = False
+    ntfy_error = ""
+    if channel == "ntfy":
+        try:
+            from src.integrations import load_integrations
+            import httpx
+            intg = next(
+                (i for i in load_integrations()
+                 if i.get("preset") == "ntfy" and i.get("enabled", True) and i.get("base_url")),
+                None,
+            )
+            if intg:
+                base = intg["base_url"].rstrip("/")
+                topic = settings.get("reminder_ntfy_topic") or "reminders"
+                ntfy_body = synthesis or note_body or title
+                # ntfy Title is an ASCII HTTP header; sanitize Unicode and cap its length.
+                _clean_title = (title or "Reminder").encode("ascii", "replace").decode("ascii")[:200]
+                hdrs = {"Title": _clean_title, "Priority": "high", "Tags": "bell"}
+                api_key = intg.get("api_key", "")
+                if api_key:
+                    hdrs["Authorization"] = f"Bearer {api_key}"
+                # SSRF guard — same check (and env knob) as the webhook branch
+                # above: link-local / metadata addresses are always rejected;
+                # REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS=true also blocks RFC-1918
+                # so a ntfy base_url can't be pointed at internal services.
+                import os as _os
+                from src.url_safety import check_outbound_url as _chk
+                _block = _os.getenv("REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS", "false").lower() == "true"
+                _ok, _reason = _chk(f"{base}/{topic}", block_private=_block)
+                if not _ok:
+                    ntfy_error = f"ntfy URL rejected: {_reason}"
+                else:
+                    async with httpx.AsyncClient(timeout=10.0) as client:
+                        resp = await client.post(f"{base}/{topic}", content=ntfy_body, headers=hdrs)
+                        ntfy_sent = resp.is_success
+                        if not ntfy_sent:
+                            ntfy_error = f"ntfy returned HTTP {resp.status_code}"
+            else:
+                ntfy_error = "No enabled ntfy integration"
+        except Exception as e:
+            ntfy_error = str(e) or e.__class__.__name__
+            logger.warning(f"Reminder ntfy send failed: {e}")
+
+    # In-app browser notification ALWAYS fires (regardless of channel). The
+    # frontend polls `/api/tasks/notifications` and turns any entry with a
+    # `body` into a real `Notification(...)` — same surface as task-success
+    # popups. Lets the user see reminders inside the app even when the
+    # primary channel is email/ntfy and the tab is open.
+    browser_sent = False
+    local_browser_sent = (not queue_browser and channel == "browser")
+    if queue_browser and _scheduler_ref is not None:
+        try:
+            _scheduler_ref.add_notification(
+                task_name=title or "Reminder",
+                status="success",
+                task_id=f"reminder-{note_id}",
+                owner=owner or None,
+                body=(synthesis or note_body or title or "").strip()[:500] or "Reminder",
+            )
+            browser_sent = True
+        except Exception as _e:
+            logger.debug(f"dispatch_reminder: in-app notif push failed: {_e}")
+
+    # Dedupe across paths: write to the same cache file `action_ping_notes`
+    # reads, so the background scanner's REPING_MIN window suppresses a
+    # second send for the same note within 25 min. Without this, a note
+    # whose due_date fires while the user has the app open got TWO emails
+    # (frontend-fired here + background-fired by ping_notes 0–5 min later).
+    if (email_sent or ntfy_sent or webhook_sent or browser_sent or local_browser_sent) and note_id:
+        try:
+            import json as _json
+            from datetime import datetime as _dt, timezone as _tz
+            from pathlib import Path as _P
+            # Per-owner cache so the scanner's prune step on user A's run
+            # doesn't drop user B's just-fired entry (review C4).
+            _STATE = cache_path
+            if _STATE is None:
+                _slug = "".join(c if (c.isalnum() or c in "-_.@") else "_" for c in (owner or "default"))
+                _STATE = _P(DATA_DIR) / f"note_pings_{_slug}.json"
+            _STATE.parent.mkdir(parents=True, exist_ok=True)
+            try:
+                _cache = cache or (_json.loads(_STATE.read_text(encoding="utf-8")) if _STATE.exists() else {})
+            except Exception:
+                _cache = {}
+            sent_channel = "email" if email_sent else "ntfy" if ntfy_sent else "webhook" if webhook_sent else "browser"
+            _cache[cache_key or str(note_id)] = {
+                "at": _dt.now(_tz.utc).isoformat(),
+                "channel": sent_channel,
+            }
+            _STATE.write_text(_json.dumps(_cache), encoding="utf-8")
+        except Exception as _e:
+            logger.debug(f"dispatch_reminder: cache write failed: {_e}")
+
+    return {
+        "channel": channel,
+        "synthesis": synthesis,
+        "email_sent": email_sent,
+        "email_error": email_error,
+        "ntfy_sent": ntfy_sent,
+        "ntfy_error": ntfy_error,
+        "webhook_sent": webhook_sent,
+        "webhook_error": webhook_error,
+        "browser_sent": browser_sent or local_browser_sent,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Router factory
+# ---------------------------------------------------------------------------
+
+def setup_note_routes(task_scheduler=None, upload_handler=None):
+    # Expose the scheduler to module-level `dispatch_reminder` so reminders
+    # can also push to the in-app notification queue (the polling system
+    # turns each entry into a real browser Notification + the existing
+    # tasks-tab badge / dot system).
+    global _scheduler_ref
+    _scheduler_ref = task_scheduler
+
+    router = APIRouter(prefix="/api/notes", tags=["notes"])
+
+    def _owner(request: Request) -> Optional[str]:
+        # require_user, not bare get_current_user: a request that reaches
+        # these owner-scoped routes with NO identity (auth-middleware
+        # regression, SSRF from a sibling service) must fail closed (401)
+        # when auth is configured — not be treated as the single-user mode
+        # and handed blanket access to every account's notes. The documented
+        # anonymous modes (AUTH_ENABLED=false, LOCALHOST_BYPASS on loopback,
+        # unconfigured first-run) still resolve to None, the single-user
+        # path. fire_reminder below already gated this way; the CRUD routes
+        # did not.
+        return require_user(request) or None
+
+    def _reserve_note_uploads(owner: Optional[str], *values) -> None:
+        missing_id = reserve_upload_references(upload_handler, owner, *values)
+        if missing_id:
+            raise HTTPException(409, f"Referenced upload is no longer available: {missing_id}")
+
+    def _is_admin_or_single_user(request: Request, user: str | None) -> bool:
+        if user == INTERNAL_TOOL_USER:
+            return True
+        if not user:
+            # require_user() already admitted this request, which only happens
+            # for auth-disabled, loopback-bypass, or unconfigured single-user
+            # modes. There is no separate non-admin account boundary there.
+            return True
+        try:
+            from core.auth import AuthManager
+            auth_mgr = getattr(request.app.state, "auth_manager", None) or AuthManager()
+            if not getattr(auth_mgr, "is_configured", True):
+                return True
+            return bool(auth_mgr.is_admin(user))
+        except Exception:
+            return False
+
+    # --- LIST ---
+    @router.get("")
+    def list_notes(
+        request: Request,
+        archived: Optional[bool] = None,
+        label: Optional[str] = None,
+    ):
+        user = _owner(request)
+        db = SessionLocal()
+        try:
+            q = db.query(Note)
+            if user is not None:
+                q = q.filter(Note.owner == user)
+            if archived is not None:
+                q = q.filter(Note.archived == archived)
+            else:
+                q = q.filter(Note.archived == False)
+            if label:
+                q = q.filter(Note.label == label)
+            # Archived view: most recently archived first. Active view: pin + manual order.
+            if archived is True:
+                notes = q.order_by(Note.updated_at.desc()).all()
+            else:
+                notes = q.order_by(Note.pinned.desc(), Note.sort_order.asc(), Note.updated_at.desc()).all()
+            return {"notes": [_note_to_dict(n) for n in notes]}
+        finally:
+            db.close()
+
+    # --- CREATE ---
+    @router.post("")
+    def create_note(request: Request, body: NoteCreate):
+        user = _owner(request)
+        _reserve_note_uploads(
+            user,
+            body.image_url,
+            body.color,
+            body.content,
+            json.dumps(body.items) if body.items is not None else None,
+        )
+        db = SessionLocal()
+        try:
+            note = Note(
+                id=str(uuid.uuid4()),
+                owner=user,
+                title=body.title,
+                content=body.content,
+                items=json.dumps(body.items) if body.items is not None else None,
+                note_type=body.note_type,
+                color=body.color,
+                label=body.label,
+                pinned=body.pinned,
+                due_date=body.due_date,
+                source=body.source,
+                session_id=body.session_id,
+                image_url=body.image_url,
+                repeat=body.repeat or "none",
+                sort_order=body.sort_order if body.sort_order is not None else 0,
+            )
+            db.add(note)
+            db.commit()
+            db.refresh(note)
+            return _note_to_dict(note)
+        finally:
+            db.close()
+
+    # --- GET ONE ---
+    @router.get("/{note_id}")
+    def get_note(request: Request, note_id: str):
+        user = _owner(request)
+        db = SessionLocal()
+        try:
+            note = db.query(Note).filter(Note.id == note_id).first()
+            if not note:
+                raise HTTPException(404, "Note not found")
+            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
+            # let any user touch a row whose owner field was null/empty.
+            if user is not None and note.owner != user:
+                raise HTTPException(404, "Note not found")
+            return _note_to_dict(note)
+        finally:
+            db.close()
+
+    # --- UPDATE ---
+    @router.put("/{note_id}")
+    def update_note(request: Request, note_id: str, body: NoteUpdate):
+        user = _owner(request)
+        db = SessionLocal()
+        try:
+            note = db.query(Note).filter(Note.id == note_id).first()
+            if not note:
+                raise HTTPException(404, "Note not found")
+            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
+            # let any user touch a row whose owner field was null/empty.
+            if user is not None and note.owner != user:
+                raise HTTPException(404, "Note not found")
+
+            _reserve_note_uploads(
+                user,
+                body.image_url,
+                body.color,
+                body.content,
+                json.dumps(body.items) if body.items is not None else None,
+            )
+            if body.title is not None:
+                note.title = body.title
+            if body.content is not None:
+                note.content = body.content
+            if body.items is not None:
+                note.items = json.dumps(body.items)
+                flag_modified(note, "items")
+            if body.note_type is not None:
+                note.note_type = body.note_type
+            if body.color is not None:
+                note.color = body.color
+            if body.label is not None:
+                note.label = body.label
+            if body.pinned is not None:
+                note.pinned = body.pinned
+            if body.archived is not None:
+                note.archived = body.archived
+            if body.due_date is not None:
+                note.due_date = body.due_date
+            if body.image_url is not None:
+                note.image_url = body.image_url
+            if body.repeat is not None:
+                note.repeat = body.repeat
+            if body.sort_order is not None:
+                note.sort_order = body.sort_order
+            if body.agent_session_id is not None:
+                note.agent_session_id = body.agent_session_id
+
+            db.commit()
+            db.refresh(note)
+            return _note_to_dict(note)
+        finally:
+            db.close()
+
+    # --- DELETE ---
+    @router.delete("/{note_id}")
+    def delete_note(request: Request, note_id: str):
+        user = _owner(request)
+        db = SessionLocal()
+        try:
+            note = db.query(Note).filter(Note.id == note_id).first()
+            if not note:
+                raise HTTPException(404, "Note not found")
+            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
+            # let any user touch a row whose owner field was null/empty.
+            if user is not None and note.owner != user:
+                raise HTTPException(404, "Note not found")
+            db.delete(note)
+            db.commit()
+            return {"ok": True}
+        finally:
+            db.close()
+
+    # --- TOGGLE PIN ---
+    @router.post("/{note_id}/pin")
+    def toggle_pin(request: Request, note_id: str):
+        user = _owner(request)
+        db = SessionLocal()
+        try:
+            note = db.query(Note).filter(Note.id == note_id).first()
+            if not note:
+                raise HTTPException(404, "Note not found")
+            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
+            # let any user touch a row whose owner field was null/empty.
+            if user is not None and note.owner != user:
+                raise HTTPException(404, "Note not found")
+            note.pinned = not note.pinned
+            db.commit()
+            return {"ok": True, "pinned": note.pinned}
+        finally:
+            db.close()
+
+    # --- TOGGLE ARCHIVE ---
+    @router.post("/{note_id}/archive")
+    def toggle_archive(request: Request, note_id: str):
+        user = _owner(request)
+        db = SessionLocal()
+        try:
+            note = db.query(Note).filter(Note.id == note_id).first()
+            if not note:
+                raise HTTPException(404, "Note not found")
+            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
+            # let any user touch a row whose owner field was null/empty.
+            if user is not None and note.owner != user:
+                raise HTTPException(404, "Note not found")
+            note.archived = not note.archived
+            db.commit()
+            return {"ok": True, "archived": note.archived}
+        finally:
+            db.close()
+
+    # --- TOGGLE CHECKLIST ITEM ---
+    @router.post("/{note_id}/items/{index}/toggle")
+    def toggle_item(request: Request, note_id: str, index: int):
+        user = _owner(request)
+        db = SessionLocal()
+        try:
+            note = db.query(Note).filter(Note.id == note_id).first()
+            if not note:
+                raise HTTPException(404, "Note not found")
+            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
+            # let any user touch a row whose owner field was null/empty.
+            if user is not None and note.owner != user:
+                raise HTTPException(404, "Note not found")
+            if not note.items:
+                raise HTTPException(400, "Note has no checklist items")
+            items = json.loads(note.items)
+            if index < 0 or index >= len(items):
+                raise HTTPException(400, f"Item index {index} out of range")
+            items[index]["done"] = not items[index].get("done", False)
+            note.items = json.dumps(items)
+            flag_modified(note, "items")
+            db.commit()
+            return {"ok": True, "items": items}
+        finally:
+            db.close()
+
+    # --- FIRE REMINDER ---
+    @router.post("/fire-reminder")
+    async def fire_reminder(request: Request):
+        """Dispatch a reminder according to user settings.
+
+        Called by the frontend when a reminder fires. Optionally generates an
+        LLM synthesis line and/or sends an email through configured SMTP.
+        Returns {synthesis, email_sent}.
+        """
+        # Gate against anonymous callers — LLM synthesis can burn tokens.
+        user = require_user(request)
+        body = await request.json()
+        note_id = str(body.get("note_id") or "").strip()
+        if not note_id:
+            raise HTTPException(400, "note_id required")
+
+        caller = _owner(request)
+        is_test = note_id.startswith("test-")
+        is_admin = _is_admin_or_single_user(request, user or caller)
+        _override: dict = {}
+        if is_test:
+            if not is_admin:
+                raise HTTPException(403, "Admin only")
+            title = (body.get("title") or "Test Reminder").strip() or "Test Reminder"
+            note_body = (body.get("body") or "").strip()
+            # Optional overrides let the admin settings test button pass the
+            # current UI values directly so it never races a pending save.
+            if body.get("channel"):
+                _override["reminder_channel"] = body["channel"]
+            if body.get("webhook_integration_id"):
+                _override["reminder_webhook_integration_id"] = body["webhook_integration_id"]
+            if body.get("webhook_payload_template"):
+                _override["reminder_webhook_payload_template"] = body["webhook_payload_template"]
+            # Mirror the in-UI AI Synthesis toggle + persona so the test
+            # actually exercises the synthesis path before/without a Save.
+            if "llm_synthesis" in body:
+                _override["reminder_llm_synthesis"] = bool(body["llm_synthesis"])
+            if "llm_persona" in body:
+                _override["reminder_llm_persona"] = str(body["llm_persona"] or "")
+        else:
+            db = SessionLocal()
+            try:
+                note = db.query(Note).filter(Note.id == note_id).first()
+                if not note:
+                    raise HTTPException(404, "Note not found")
+                if caller is not None and note.owner != caller:
+                    raise HTTPException(404, "Note not found")
+                title, note_body = _reminder_text_from_note(note)
+            finally:
+                db.close()
+
+        return await dispatch_reminder(
+            title=title, note_body=note_body, note_id=note_id,
+            owner=caller or "",
+            queue_browser=False,
+            settings_override=_override or None,
+        )
+
+    # --- REORDER NOTES ---
+    @router.post("/reorder")
+    async def reorder_notes(request: Request):
+        """Update sort_order for a list of note IDs in the order provided."""
+        user = _owner(request)
+        body = await request.json()
+        ids = body.get("ids", [])
+        if not isinstance(ids, list):
+            raise HTTPException(400, "ids must be a list")
+        # v2 review HIGH-12: drop the legacy `(owner == user) | (owner ==
+        # None)` OR which let an authenticated user silently reorder
+        # every legacy-null-owner note belonging to other accounts. In
+        # an unconfigured (single-user) auth deploy the OR is still safe
+        # because there's no second user to attack; we keep that branch
+        # explicit and gated on AuthManager.is_configured.
+        try:
+            from core.auth import AuthManager
+            _allow_null = not AuthManager().is_configured
+        except Exception:
+            _allow_null = False
+        db = SessionLocal()
+        try:
+            for i, nid in enumerate(ids):
+                q = db.query(Note).filter(Note.id == nid)
+                if user is not None:
+                    if _allow_null:
+                        q = q.filter((Note.owner == user) | (Note.owner == None))  # noqa: E711
+                    else:
+                        q = q.filter(Note.owner == user)
+                note = q.first()
+                if note:
+                    note.sort_order = i
+            db.commit()
+            return {"ok": True, "count": len(ids)}
+        finally:
+            db.close()
+
+    return router

routes/note_routes.py

@@ -1,915 +1,18 @@
-# routes/note_routes.py
-"""Google Keep-style notes / checklists API."""
+"""Backward-compat shim — canonical location is routes/note/note_routes.py.
 
-import json
-import uuid
-import logging
-from typing import Dict, Any, Optional
+This module is replaced in ``sys.modules`` by the canonical module object so
+that ``import routes.note_routes``, ``from routes.note_routes import X``,
+``importlib.import_module("routes.note_routes")``, and the
+``import ... as note_routes`` + ``monkeypatch.setattr(note_routes, "SessionLocal",
+...)`` pattern used by test_note_reminder_fire_scope.py /
+test_notes_fail_closed_auth.py all operate on the *same* object the
+application actually uses. Keeps existing import paths working after
+slice 2f (#4082/#4071). Source-introspection tests read the canonical file
+by path.
+"""
 
-from fastapi import APIRouter, HTTPException, Request
-from pydantic import BaseModel
+import sys as _sys
 
-from core.database import SessionLocal, Note
-from core.middleware import INTERNAL_TOOL_USER
-from src.auth_helpers import require_user
-from src.constants import DATA_DIR
-from sqlalchemy.orm.attributes import flag_modified
+from routes.note import note_routes as _canonical  # noqa: F401
 
-logger = logging.getLogger(__name__)
-
-
-# ---------------------------------------------------------------------------
-# Request models
-# ---------------------------------------------------------------------------
-
-class NoteCreate(BaseModel):
-    title: str = ""
-    content: Optional[str] = None
-    items: Optional[list] = None
-    note_type: str = "note"
-    color: Optional[str] = None
-    label: Optional[str] = None
-    pinned: bool = False
-    due_date: Optional[str] = None
-    source: str = "user"
-    session_id: Optional[str] = None
-    image_url: Optional[str] = None
-    repeat: Optional[str] = "none"
-    sort_order: Optional[int] = None
-
-
-class NoteUpdate(BaseModel):
-    title: Optional[str] = None
-    content: Optional[str] = None
-    items: Optional[list] = None
-    note_type: Optional[str] = None
-    color: Optional[str] = None
-    label: Optional[str] = None
-    pinned: Optional[bool] = None
-    archived: Optional[bool] = None
-    due_date: Optional[str] = None
-    image_url: Optional[str] = None
-    repeat: Optional[str] = None
-    sort_order: Optional[int] = None
-    agent_session_id: Optional[str] = None
-
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-def _note_to_dict(note: Note) -> Dict[str, Any]:
-    items = None
-    if note.items:
-        try:
-            items = json.loads(note.items)
-        except (json.JSONDecodeError, TypeError):
-            items = None
-    ai_cls = None
-    raw_ai = getattr(note, "ai_classification", None)
-    if raw_ai:
-        try:
-            ai_cls = json.loads(raw_ai)
-        except (json.JSONDecodeError, TypeError):
-            ai_cls = None
-    return {
-        "id": note.id,
-        "owner": note.owner,
-        "title": note.title,
-        "content": note.content,
-        "items": items,
-        "note_type": note.note_type,
-        "color": note.color,
-        "label": note.label,
-        "pinned": note.pinned,
-        "archived": note.archived,
-        "due_date": note.due_date,
-        "source": note.source,
-        "session_id": note.session_id,
-        "sort_order": note.sort_order or 0,
-        "image_url": note.image_url,
-        "repeat": note.repeat or "none",
-        "ai_classification": ai_cls,
-        "ai_content_hash": getattr(note, "ai_content_hash", None),
-        "agent_session_id": getattr(note, "agent_session_id", None),
-        "created_at": note.created_at.isoformat() if note.created_at else None,
-        "updated_at": note.updated_at.isoformat() if note.updated_at else None,
-    }
-
-
-def _reminder_text_from_note(note: Note) -> tuple[str, str]:
-    """Return the reminder title/body from a stored note row."""
-    title = (note.title or "Note reminder").strip() or "Note reminder"
-    if note.items:
-        try:
-            items = json.loads(note.items)
-        except (json.JSONDecodeError, TypeError):
-            items = None
-        if isinstance(items, list):
-            pending: list[str] = []
-            for item in items:
-                if not isinstance(item, dict):
-                    continue
-                if item.get("done") or item.get("checked"):
-                    continue
-                text = str(item.get("text") or "").strip()
-                if text:
-                    pending.append(text)
-            if pending:
-                shown = "\n".join(f"- {text}" for text in pending[:8])
-                extra = f"\n...and {len(pending) - 8} more" if len(pending) > 8 else ""
-                return title, f"Pending ({len(pending)}):\n{shown}{extra}"
-            return title, f"{len(items)} item{'s' if len(items) != 1 else ''}"
-    return title, (note.content or "").strip()[:400]
-
-
-
-# ---------------------------------------------------------------------------
-# Reminder dispatch — module-level so background tasks (built-in actions)
-# can call it directly without an HTTP roundtrip + auth cookie. The route
-# version below is a thin wrapper that pulls `owner` from the request.
-# ---------------------------------------------------------------------------
-
-# Scheduler reference — set by setup_note_routes() so dispatch_reminder can
-# push a parallel in-app notification (frontend polls the scheduler's queue
-# and fires real browser Notification(...) popups). Optional; works without it.
-_scheduler_ref = None
-
-
-async def dispatch_reminder(
-    title: str,
-    note_body: str,
-    note_id: str,
-    owner: str = "",
-    queue_browser: bool = True,
-    settings_override: dict | None = None,
-) -> dict:
-    """Fire a reminder via the configured channel (browser/email/ntfy/webhook).
-
-    Args:
-        title: short headline shown to the user
-        note_body: longer body text
-        note_id: stable id (used as tag/dedupe in browser notifications)
-        owner: the user this reminder belongs to — scopes SMTP config to
-               their account so we don't cross-leak credentials
-
-    Returns: {synthesis, email_sent, ntfy_sent}. Browser channel is wired via
-    the in-memory notification queue picked up by the frontend poller, so
-    nothing is "sent" synchronously for it — the channel just routes there.
-    """
-    from src.settings import load_settings
-    settings = {**load_settings(), **(settings_override or {})}
-    channel = settings.get("reminder_channel", "browser")
-    llm_on = bool(settings.get("reminder_llm_synthesis", False))
-    title = (title or "").strip()
-    note_body = (note_body or "").strip()
-    cache_key = str(note_id) if note_id else ""
-    cache = {}
-    cache_path = None
-    if cache_key:
-        try:
-            import json as _json
-            from datetime import datetime as _dt, timezone as _tz, timedelta as _td
-            from pathlib import Path as _P
-            _slug = "".join(c if (c.isalnum() or c in "-_.@") else "_" for c in (owner or "default"))
-            cache_path = _P(DATA_DIR) / f"note_pings_{_slug}.json"
-            if cache_path.exists():
-                cache = _json.loads(cache_path.read_text(encoding="utf-8"))
-            last = cache.get(cache_key)
-            if last:
-                last_channel = None
-                if isinstance(last, dict):
-                    last_channel = last.get("channel")
-                    last = last.get("at")
-                last_dt = _dt.fromisoformat(str(last))
-                if last_dt.tzinfo is None:
-                    last_dt = last_dt.replace(tzinfo=_tz.utc)
-                # Legacy cache values were plain timestamps and could be
-                # written by the frontend even when the email/ntfy send failed.
-                # Treat those as browser-only dedupe so email reminders can be
-                # retried by the backend scanner after a failed frontend path.
-                should_skip = last_dt >= _dt.now(_tz.utc) - _td(minutes=25)
-                if should_skip and channel in ("email", "ntfy", "webhook"):
-                    should_skip = last_channel == channel
-                if should_skip:
-                    return {
-                        "synthesis": None,
-                        "email_sent": False,
-                        "ntfy_sent": False,
-                        "webhook_sent": False,
-                        "browser_sent": True,
-                        "skipped": True,
-                    }
-        except Exception as _e:
-            logger.debug(f"dispatch_reminder: cache read failed: {_e}")
-
-    synthesis = None
-    _SYNTH_FAILED_TAG = "[utility model unavailable — no summary generated]"
-    if llm_on:
-        try:
-            from src.endpoint_resolver import resolve_endpoint
-            from src.llm_core import llm_call_async
-            from src.reminder_personas import synthesis_system_prompt
-            url, model, headers = resolve_endpoint("utility", owner=owner or None)
-            if not url:
-                url, model, headers = resolve_endpoint("default", owner=owner or None)
-            if url and model:
-                persona_id = (settings.get("reminder_llm_persona") or "").strip()
-                sys_prompt = synthesis_system_prompt(persona_id)
-                raw = await llm_call_async(
-                    url=url, model=model,
-                    messages=[
-                        {"role": "system", "content": sys_prompt},
-                        {"role": "user", "content": f"Title: {title}\n\n{note_body}".strip()},
-                    ],
-                    temperature=0.7, max_tokens=200, headers=headers, timeout=30,
-                )
-                from src.text_helpers import strip_think as _strip_think
-                # prose=True strips untagged "The user wants me to…" chain-of-thought.
-                # prompt_echo=True strips Qwen-style "Thinking Process:" / leaked
-                # prompt prefixes. Both are safe here because this is a
-                # one-sentence LLM-only output, not user-pasted content.
-                synthesis = _strip_think(raw or "", prose=True, prompt_echo=True)
-                # Reminder synthesis is supposed to be ONE sentence. Strip-think's
-                # paragraph-based heuristic misses cases where the model puts
-                # reasoning + answer on consecutive lines inside one paragraph
-                # (e.g. "I should write... [\n] You have one task waiting...").
-                # Walk lines, drop reasoning/prompt-echo lines, then keep the
-                # last surviving line — that's the actual warm sentence.
-                if synthesis:
-                    import re as _re
-                    # Tightened: target ACTUAL self-talk (model narrating what
-                    # it'll do) rather than any first-person sentence. The old
-                    # pattern killed legit warm sentences like "I'll see you
-                    # tomorrow" or "I should be done by then". New rules:
-                    #  • "I (need|should|have|'ll|will) (write|draft|reply|…)"
-                    #    only matches when followed by a TASK verb taking an
-                    #    OBJECT (so first-person + intransitive verb passes).
-                    #  • Self-instructional patterns the model emits verbatim:
-                    #    "I should write something that reminds them…",
-                    #    "I need to draft…", "Let me think…".
-                    #  • Explicit instructions echoed back from the prompt:
-                    #    "Keep it under 25 words", "No greetings".
-                    _reasoning = _re.compile(
-                        r"^\s*(?:"
-                        # "I should write/draft/compose…" with a task-object follow
-                        r"i (?:need|should|have|'ll|will|am going|am)\s+to\s+"
-                        r"(?:write|draft|compose|craft|generate|produce|create|"
-                        r"summarize|answer|provide|note|address|remind|output)"
-                        r"\s+(?:a |an |the |something|this|that|here|them|him|her|"
-                        r"you|user|reply|response|sentence|message|line|warm)|"
-                        # The model literally narrating about the user
-                        r"the user (?:wants|is asking|asks|needs|wrote|said|requested) (?:me )?(?:to|for|that|about|something)|"
-                        # "Let me [think/write/draft/…] (about/for/the …)"
-                        r"let me (?:think|write|draft|consider|note|see|check)\b\s+(?:about|for|the|this|that|if|whether)|"
-                        # "Looking at the/this/that …"
-                        r"looking at (?:the|this|that)\b|"
-                        # "Based on the/this/what …"
-                        r"based on (?:the|this|what|context|that)\b|"
-                        # Prompt-echo of length / style instructions
-                        r"keep it under \d+ words\b|"
-                        r"(?:no greetings|no preamble|no hashtags|just output the)\b"
-                        r").*",
-                        _re.IGNORECASE,
-                    )
-                    # Echo of the prompt's "Pending:" / "<N> pending" tail.
-                    _echo = _re.compile(
-                        r"^\s*(?:pending\s*[:.]|(?:\d+|one|two|three|four|five)\s+pending\b)",
-                        _re.IGNORECASE,
-                    )
-                    lines = [ln for ln in synthesis.splitlines() if ln.strip()]
-                    cleaned = [ln for ln in lines if not _reasoning.match(ln) and not _echo.match(ln)]
-                    if cleaned:
-                        # The model's actual answer is normally the LAST surviving
-                        # line — reasoning leads, answer trails.
-                        synthesis = cleaned[-1].strip()
-            else:
-                synthesis = _SYNTH_FAILED_TAG
-        except Exception as e:
-            logger.warning(f"Reminder LLM synthesis failed: {e}")
-            synthesis = _SYNTH_FAILED_TAG
-        if synthesis:
-            _s = synthesis.strip(); _low = _s.lower()
-            if (not _s or _low.startswith("error:") or _low.startswith("[error")
-                    or "operation failed" in _low
-                    or ("upstream" in _low and "failed" in _low)) and synthesis != _SYNTH_FAILED_TAG:
-                logger.warning(f"Reminder synthesis looked like an error, replacing: {_s[:120]!r}")
-                synthesis = _SYNTH_FAILED_TAG
-
-    email_sent = False
-    email_error = ""
-    if channel == "email":
-        try:
-            from routes.email_routes import _get_email_config
-            from email.mime.text import MIMEText
-            from email.mime.multipart import MIMEMultipart
-            from datetime import datetime as _dt
-            # `reminder_email_account_id` lets the user pick WHICH email
-            # account to send reminders from (when they have several
-            # configured in Integrations). Falls back to the default
-            # account when no explicit choice is saved.
-            _acc_id = (settings.get("reminder_email_account_id") or "").strip() or None
-            cfg = _get_email_config(account_id=_acc_id, owner=owner or "")
-            if not (cfg.get("smtp_host") and cfg.get("smtp_user") and cfg.get("smtp_password")):
-                try:
-                    from core.database import SessionLocal as _SL, EmailAccount as _EA
-                    from sqlalchemy import and_, or_
-                    db = _SL()
-                    try:
-                        q = db.query(_EA).filter(_EA.enabled == True)  # noqa: E712
-                        if owner:
-                            unowned = or_(_EA.owner == None, _EA.owner == "")  # noqa: E711
-                            same_mailbox = or_(_EA.imap_user == owner, _EA.from_address == owner)
-                            q = q.filter(or_(_EA.owner == owner, and_(unowned, same_mailbox)))
-                        for row in q.order_by(_EA.is_default.desc(), _EA.created_at.asc()).all():
-                            trial = _get_email_config(account_id=row.id, owner=owner or "")
-                            if trial.get("smtp_host") and trial.get("smtp_user") and trial.get("smtp_password"):
-                                cfg = trial
-                                break
-                    finally:
-                        db.close()
-                except Exception as _fallback_error:
-                    logger.debug(f"Reminder SMTP fallback lookup failed: {_fallback_error}")
-            from_addr = (cfg.get("from_address") or cfg.get("smtp_user") or "").strip()
-            recipient = (settings.get("reminder_email_to") or "").strip() or from_addr
-            # Loud diagnostic so we can see WHY a reminder didn't send (the
-            # previous "silently no-op when cfg has no smtp_host" was invisible).
-            logger.info(
-                "dispatch_reminder[email] note_id=%s owner=%r "
-                "has_smtp_host=%s has_smtp_user=%s has_from=%s has_recipient=%s",
-                note_id, owner,
-                bool(cfg.get("smtp_host")), bool(cfg.get("smtp_user")),
-                bool(from_addr), bool(recipient),
-            )
-            missing = []
-            if not cfg.get("smtp_host"):
-                missing.append("SMTP host")
-            if not cfg.get("smtp_user"):
-                missing.append("SMTP user")
-            if not cfg.get("smtp_password"):
-                missing.append("SMTP password")
-            if not from_addr:
-                missing.append("from address")
-            if not recipient:
-                missing.append("recipient")
-            if missing:
-                email_error = "Missing " + ", ".join(missing)
-                logger.warning(
-                    "Reminder email not sent for note_id=%s account=%r: %s",
-                    note_id, cfg.get("account_name"), email_error,
-                )
-            else:
-                msg = MIMEMultipart("alternative")
-                msg["From"] = from_addr
-                msg["To"] = recipient
-                _t = title or 'Note'
-                _t = _t[len('Reminder:'):].strip() if _t.lower().startswith('reminder:') else _t
-                msg["Subject"] = f"Reminder (Odysseus): {_t}"
-                msg["Date"] = _dt.utcnow().strftime("%a, %d %b %Y %H:%M:%S +0000")
-                msg["X-Odysseus-Origin"] = "odysseus-ui"
-                msg["X-Odysseus-Kind"] = "reminder"
-                msg["X-Odysseus-Ref"] = str(note_id)
-                # Body shape: synthesis (warm sentence) → blank line → bold
-                # title header → note details. The title was previously only
-                # in the subject line, so the email read like a faceless
-                # to-do list with no anchor to which note triggered it.
-                _body_chunks = []
-                if synthesis:
-                    _body_chunks.append(synthesis)
-                if _t:
-                    _body_chunks.append(_t)
-                if note_body:
-                    _body_chunks.append(note_body)
-                plain = "\n\n".join(_body_chunks) if _body_chunks else title
-                msg.attach(MIMEText(plain, "plain", "utf-8"))
-
-                def _smtp_send():
-                    from routes.email_helpers import _send_smtp_message
-                    _send_smtp_message(cfg, from_addr, [recipient], msg.as_string())
-
-                import asyncio as _aio
-                await _aio.to_thread(_smtp_send)
-                email_sent = True
-        except Exception as e:
-            email_error = str(e) or e.__class__.__name__
-            logger.warning(f"Reminder email send failed: {e}")
-
-    webhook_sent = False
-    webhook_error = ""
-    if channel == "webhook":
-        try:
-            import httpx
-            import json as _wjson
-            from src.integrations import load_integrations
-            # Built-in payload defaults for known presets so users don't have
-            # to configure a template just to use a standard service.
-            _PRESET_TEMPLATE_DEFAULTS = {
-                "discord_webhook": '{"embeds": [{"title": "{{title}}", "description": "{{message}}", "color": 5793266}]}',
-            }
-            intg_id = settings.get("reminder_webhook_integration_id", "").strip()
-            template = settings.get("reminder_webhook_payload_template", "").strip()
-            if not intg_id:
-                webhook_error = "No webhook integration selected"
-            else:
-                intg = next(
-                    (i for i in load_integrations()
-                     if i.get("id") == intg_id and i.get("base_url")),
-                    None,
-                )
-                if not intg:
-                    webhook_error = f"Integration {intg_id!r} not found or missing base URL"
-                else:
-                    # Fall back to a built-in default for known presets so
-                    # users don't have to configure a template for standard
-                    # services like Discord.
-                    if not template:
-                        template = _PRESET_TEMPLATE_DEFAULTS.get(intg.get("preset", ""), "")
-                    if not template:
-                        webhook_error = "No payload template configured"
-                    else:
-                        # Render template: JSON-escape the values so the result
-                        # is always valid JSON regardless of special characters.
-                        # dumps() returns `"value"` — strip outer quotes.
-                        msg = (synthesis or note_body or title or "Reminder")[:4000]
-                        _t = _wjson.dumps(title or "Reminder")[1:-1]
-                        _m = _wjson.dumps(msg)[1:-1]
-                        rendered = template.replace("{{title}}", _t).replace("{{message}}", _m)
-                        hdrs = {"Content-Type": "application/json"}
-                        api_key = intg.get("api_key", "")
-                        auth_type = (intg.get("auth_type") or "none").lower()
-                        if api_key:
-                            if auth_type == "bearer":
-                                hdrs["Authorization"] = f"Bearer {api_key}"
-                            elif auth_type == "header":
-                                hdrs[intg.get("auth_header") or "Authorization"] = api_key
-                        url = intg["base_url"].rstrip("/")
-                        # SSRF guard — matches the pattern used by webhook_routes,
-                        # CalDAV, search, and embeddings. Blocks link-local / metadata
-                        # addresses (169.254.x.x) by default; set
-                        # REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS=true to also block
-                        # RFC-1918 ranges for locked-down deployments.
-                        import os as _os
-                        from src.url_safety import check_outbound_url as _chk
-                        _block = _os.getenv("REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS", "false").lower() == "true"
-                        _ok, _reason = _chk(url, block_private=_block)
-                        if not _ok:
-                            webhook_error = f"Webhook URL rejected: {_reason}"
-                        else:
-                            async with httpx.AsyncClient(timeout=10.0) as client:
-                                resp = await client.post(url, content=rendered.encode(), headers=hdrs)
-                                webhook_sent = resp.is_success
-                                if not webhook_sent:
-                                    webhook_error = f"Webhook returned HTTP {resp.status_code}"
-        except Exception as e:
-            webhook_error = str(e) or e.__class__.__name__
-            logger.warning(f"Reminder webhook send failed: {e}")
-
-    ntfy_sent = False
-    ntfy_error = ""
-    if channel == "ntfy":
-        try:
-            from src.integrations import load_integrations
-            import httpx
-            intg = next(
-                (i for i in load_integrations()
-                 if i.get("preset") == "ntfy" and i.get("enabled", True) and i.get("base_url")),
-                None,
-            )
-            if intg:
-                base = intg["base_url"].rstrip("/")
-                topic = settings.get("reminder_ntfy_topic") or "reminders"
-                ntfy_body = synthesis or note_body or title
-                hdrs = {"Title": title or "Reminder", "Priority": "high", "Tags": "bell"}
-                api_key = intg.get("api_key", "")
-                if api_key:
-                    hdrs["Authorization"] = f"Bearer {api_key}"
-                # SSRF guard — same check (and env knob) as the webhook branch
-                # above: link-local / metadata addresses are always rejected;
-                # REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS=true also blocks RFC-1918
-                # so a ntfy base_url can't be pointed at internal services.
-                import os as _os
-                from src.url_safety import check_outbound_url as _chk
-                _block = _os.getenv("REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS", "false").lower() == "true"
-                _ok, _reason = _chk(f"{base}/{topic}", block_private=_block)
-                if not _ok:
-                    ntfy_error = f"ntfy URL rejected: {_reason}"
-                else:
-                    async with httpx.AsyncClient(timeout=10.0) as client:
-                        resp = await client.post(f"{base}/{topic}", content=ntfy_body, headers=hdrs)
-                        ntfy_sent = resp.is_success
-                        if not ntfy_sent:
-                            ntfy_error = f"ntfy returned HTTP {resp.status_code}"
-            else:
-                ntfy_error = "No enabled ntfy integration"
-        except Exception as e:
-            ntfy_error = str(e) or e.__class__.__name__
-            logger.warning(f"Reminder ntfy send failed: {e}")
-
-    # In-app browser notification ALWAYS fires (regardless of channel). The
-    # frontend polls `/api/tasks/notifications` and turns any entry with a
-    # `body` into a real `Notification(...)` — same surface as task-success
-    # popups. Lets the user see reminders inside the app even when the
-    # primary channel is email/ntfy and the tab is open.
-    browser_sent = False
-    local_browser_sent = (not queue_browser and channel == "browser")
-    if queue_browser and _scheduler_ref is not None:
-        try:
-            _scheduler_ref.add_notification(
-                task_name=title or "Reminder",
-                status="success",
-                task_id=f"reminder-{note_id}",
-                owner=owner or None,
-                body=(synthesis or note_body or title or "").strip()[:500] or "Reminder",
-            )
-            browser_sent = True
-        except Exception as _e:
-            logger.debug(f"dispatch_reminder: in-app notif push failed: {_e}")
-
-    # Dedupe across paths: write to the same cache file `action_ping_notes`
-    # reads, so the background scanner's REPING_MIN window suppresses a
-    # second send for the same note within 25 min. Without this, a note
-    # whose due_date fires while the user has the app open got TWO emails
-    # (frontend-fired here + background-fired by ping_notes 0–5 min later).
-    if (email_sent or ntfy_sent or webhook_sent or browser_sent or local_browser_sent) and note_id:
-        try:
-            import json as _json
-            from datetime import datetime as _dt, timezone as _tz
-            from pathlib import Path as _P
-            # Per-owner cache so the scanner's prune step on user A's run
-            # doesn't drop user B's just-fired entry (review C4).
-            _STATE = cache_path
-            if _STATE is None:
-                _slug = "".join(c if (c.isalnum() or c in "-_.@") else "_" for c in (owner or "default"))
-                _STATE = _P(DATA_DIR) / f"note_pings_{_slug}.json"
-            _STATE.parent.mkdir(parents=True, exist_ok=True)
-            try:
-                _cache = cache or (_json.loads(_STATE.read_text(encoding="utf-8")) if _STATE.exists() else {})
-            except Exception:
-                _cache = {}
-            sent_channel = "email" if email_sent else "ntfy" if ntfy_sent else "webhook" if webhook_sent else "browser"
-            _cache[cache_key or str(note_id)] = {
-                "at": _dt.now(_tz.utc).isoformat(),
-                "channel": sent_channel,
-            }
-            _STATE.write_text(_json.dumps(_cache), encoding="utf-8")
-        except Exception as _e:
-            logger.debug(f"dispatch_reminder: cache write failed: {_e}")
-
-    return {
-        "channel": channel,
-        "synthesis": synthesis,
-        "email_sent": email_sent,
-        "email_error": email_error,
-        "ntfy_sent": ntfy_sent,
-        "ntfy_error": ntfy_error,
-        "webhook_sent": webhook_sent,
-        "webhook_error": webhook_error,
-        "browser_sent": browser_sent or local_browser_sent,
-    }
-
-
-# ---------------------------------------------------------------------------
-# Router factory
-# ---------------------------------------------------------------------------
-
-def setup_note_routes(task_scheduler=None):
-    # Expose the scheduler to module-level `dispatch_reminder` so reminders
-    # can also push to the in-app notification queue (the polling system
-    # turns each entry into a real browser Notification + the existing
-    # tasks-tab badge / dot system).
-    global _scheduler_ref
-    _scheduler_ref = task_scheduler
-
-    router = APIRouter(prefix="/api/notes", tags=["notes"])
-
-    def _owner(request: Request) -> Optional[str]:
-        # require_user, not bare get_current_user: a request that reaches
-        # these owner-scoped routes with NO identity (auth-middleware
-        # regression, SSRF from a sibling service) must fail closed (401)
-        # when auth is configured — not be treated as the single-user mode
-        # and handed blanket access to every account's notes. The documented
-        # anonymous modes (AUTH_ENABLED=false, LOCALHOST_BYPASS on loopback,
-        # unconfigured first-run) still resolve to None, the single-user
-        # path. fire_reminder below already gated this way; the CRUD routes
-        # did not.
-        return require_user(request) or None
-
-    def _is_admin_or_single_user(request: Request, user: str | None) -> bool:
-        if user == INTERNAL_TOOL_USER:
-            return True
-        if not user:
-            # require_user() already admitted this request, which only happens
-            # for auth-disabled, loopback-bypass, or unconfigured single-user
-            # modes. There is no separate non-admin account boundary there.
-            return True
-        try:
-            from core.auth import AuthManager
-            auth_mgr = getattr(request.app.state, "auth_manager", None) or AuthManager()
-            if not getattr(auth_mgr, "is_configured", True):
-                return True
-            return bool(auth_mgr.is_admin(user))
-        except Exception:
-            return False
-
-    # --- LIST ---
-    @router.get("")
-    def list_notes(
-        request: Request,
-        archived: Optional[bool] = None,
-        label: Optional[str] = None,
-    ):
-        user = _owner(request)
-        db = SessionLocal()
-        try:
-            q = db.query(Note)
-            if user is not None:
-                q = q.filter(Note.owner == user)
-            if archived is not None:
-                q = q.filter(Note.archived == archived)
-            else:
-                q = q.filter(Note.archived == False)
-            if label:
-                q = q.filter(Note.label == label)
-            # Archived view: most recently archived first. Active view: pin + manual order.
-            if archived is True:
-                notes = q.order_by(Note.updated_at.desc()).all()
-            else:
-                notes = q.order_by(Note.pinned.desc(), Note.sort_order.asc(), Note.updated_at.desc()).all()
-            return {"notes": [_note_to_dict(n) for n in notes]}
-        finally:
-            db.close()
-
-    # --- CREATE ---
-    @router.post("")
-    def create_note(request: Request, body: NoteCreate):
-        user = _owner(request)
-        db = SessionLocal()
-        try:
-            note = Note(
-                id=str(uuid.uuid4()),
-                owner=user,
-                title=body.title,
-                content=body.content,
-                items=json.dumps(body.items) if body.items is not None else None,
-                note_type=body.note_type,
-                color=body.color,
-                label=body.label,
-                pinned=body.pinned,
-                due_date=body.due_date,
-                source=body.source,
-                session_id=body.session_id,
-                image_url=body.image_url,
-                repeat=body.repeat or "none",
-                sort_order=body.sort_order if body.sort_order is not None else 0,
-            )
-            db.add(note)
-            db.commit()
-            db.refresh(note)
-            return _note_to_dict(note)
-        finally:
-            db.close()
-
-    # --- GET ONE ---
-    @router.get("/{note_id}")
-    def get_note(request: Request, note_id: str):
-        user = _owner(request)
-        db = SessionLocal()
-        try:
-            note = db.query(Note).filter(Note.id == note_id).first()
-            if not note:
-                raise HTTPException(404, "Note not found")
-            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
-            # let any user touch a row whose owner field was null/empty.
-            if user is not None and note.owner != user:
-                raise HTTPException(404, "Note not found")
-            return _note_to_dict(note)
-        finally:
-            db.close()
-
-    # --- UPDATE ---
-    @router.put("/{note_id}")
-    def update_note(request: Request, note_id: str, body: NoteUpdate):
-        user = _owner(request)
-        db = SessionLocal()
-        try:
-            note = db.query(Note).filter(Note.id == note_id).first()
-            if not note:
-                raise HTTPException(404, "Note not found")
-            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
-            # let any user touch a row whose owner field was null/empty.
-            if user is not None and note.owner != user:
-                raise HTTPException(404, "Note not found")
-
-            if body.title is not None:
-                note.title = body.title
-            if body.content is not None:
-                note.content = body.content
-            if body.items is not None:
-                note.items = json.dumps(body.items)
-                flag_modified(note, "items")
-            if body.note_type is not None:
-                note.note_type = body.note_type
-            if body.color is not None:
-                note.color = body.color
-            if body.label is not None:
-                note.label = body.label
-            if body.pinned is not None:
-                note.pinned = body.pinned
-            if body.archived is not None:
-                note.archived = body.archived
-            if body.due_date is not None:
-                note.due_date = body.due_date
-            if body.image_url is not None:
-                note.image_url = body.image_url
-            if body.repeat is not None:
-                note.repeat = body.repeat
-            if body.sort_order is not None:
-                note.sort_order = body.sort_order
-            if body.agent_session_id is not None:
-                note.agent_session_id = body.agent_session_id
-
-            db.commit()
-            db.refresh(note)
-            return _note_to_dict(note)
-        finally:
-            db.close()
-
-    # --- DELETE ---
-    @router.delete("/{note_id}")
-    def delete_note(request: Request, note_id: str):
-        user = _owner(request)
-        db = SessionLocal()
-        try:
-            note = db.query(Note).filter(Note.id == note_id).first()
-            if not note:
-                raise HTTPException(404, "Note not found")
-            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
-            # let any user touch a row whose owner field was null/empty.
-            if user is not None and note.owner != user:
-                raise HTTPException(404, "Note not found")
-            db.delete(note)
-            db.commit()
-            return {"ok": True}
-        finally:
-            db.close()
-
-    # --- TOGGLE PIN ---
-    @router.post("/{note_id}/pin")
-    def toggle_pin(request: Request, note_id: str):
-        user = _owner(request)
-        db = SessionLocal()
-        try:
-            note = db.query(Note).filter(Note.id == note_id).first()
-            if not note:
-                raise HTTPException(404, "Note not found")
-            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
-            # let any user touch a row whose owner field was null/empty.
-            if user is not None and note.owner != user:
-                raise HTTPException(404, "Note not found")
-            note.pinned = not note.pinned
-            db.commit()
-            return {"ok": True, "pinned": note.pinned}
-        finally:
-            db.close()
-
-    # --- TOGGLE ARCHIVE ---
-    @router.post("/{note_id}/archive")
-    def toggle_archive(request: Request, note_id: str):
-        user = _owner(request)
-        db = SessionLocal()
-        try:
-            note = db.query(Note).filter(Note.id == note_id).first()
-            if not note:
-                raise HTTPException(404, "Note not found")
-            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
-            # let any user touch a row whose owner field was null/empty.
-            if user is not None and note.owner != user:
-                raise HTTPException(404, "Note not found")
-            note.archived = not note.archived
-            db.commit()
-            return {"ok": True, "archived": note.archived}
-        finally:
-            db.close()
-
-    # --- TOGGLE CHECKLIST ITEM ---
-    @router.post("/{note_id}/items/{index}/toggle")
-    def toggle_item(request: Request, note_id: str, index: int):
-        user = _owner(request)
-        db = SessionLocal()
-        try:
-            note = db.query(Note).filter(Note.id == note_id).first()
-            if not note:
-                raise HTTPException(404, "Note not found")
-            # SECURITY: strict ownership — previously `note.owner and note.owner != user`
-            # let any user touch a row whose owner field was null/empty.
-            if user is not None and note.owner != user:
-                raise HTTPException(404, "Note not found")
-            if not note.items:
-                raise HTTPException(400, "Note has no checklist items")
-            items = json.loads(note.items)
-            if index < 0 or index >= len(items):
-                raise HTTPException(400, f"Item index {index} out of range")
-            items[index]["done"] = not items[index].get("done", False)
-            note.items = json.dumps(items)
-            flag_modified(note, "items")
-            db.commit()
-            return {"ok": True, "items": items}
-        finally:
-            db.close()
-
-    # --- FIRE REMINDER ---
-    @router.post("/fire-reminder")
-    async def fire_reminder(request: Request):
-        """Dispatch a reminder according to user settings.
-
-        Called by the frontend when a reminder fires. Optionally generates an
-        LLM synthesis line and/or sends an email through configured SMTP.
-        Returns {synthesis, email_sent}.
-        """
-        # Gate against anonymous callers — LLM synthesis can burn tokens.
-        user = require_user(request)
-        body = await request.json()
-        note_id = str(body.get("note_id") or "").strip()
-        if not note_id:
-            raise HTTPException(400, "note_id required")
-
-        caller = _owner(request)
-        is_test = note_id.startswith("test-")
-        is_admin = _is_admin_or_single_user(request, user or caller)
-        _override: dict = {}
-        if is_test:
-            if not is_admin:
-                raise HTTPException(403, "Admin only")
-            title = (body.get("title") or "Test Reminder").strip() or "Test Reminder"
-            note_body = (body.get("body") or "").strip()
-            # Optional overrides let the admin settings test button pass the
-            # current UI values directly so it never races a pending save.
-            if body.get("channel"):
-                _override["reminder_channel"] = body["channel"]
-            if body.get("webhook_integration_id"):
-                _override["reminder_webhook_integration_id"] = body["webhook_integration_id"]
-            if body.get("webhook_payload_template"):
-                _override["reminder_webhook_payload_template"] = body["webhook_payload_template"]
-            # Mirror the in-UI AI Synthesis toggle + persona so the test
-            # actually exercises the synthesis path before/without a Save.
-            if "llm_synthesis" in body:
-                _override["reminder_llm_synthesis"] = bool(body["llm_synthesis"])
-            if "llm_persona" in body:
-                _override["reminder_llm_persona"] = str(body["llm_persona"] or "")
-        else:
-            db = SessionLocal()
-            try:
-                note = db.query(Note).filter(Note.id == note_id).first()
-                if not note:
-                    raise HTTPException(404, "Note not found")
-                if caller is not None and note.owner != caller:
-                    raise HTTPException(404, "Note not found")
-                title, note_body = _reminder_text_from_note(note)
-            finally:
-                db.close()
-
-        return await dispatch_reminder(
-            title=title, note_body=note_body, note_id=note_id,
-            owner=caller or "",
-            queue_browser=False,
-            settings_override=_override or None,
-        )
-
-    # --- REORDER NOTES ---
-    @router.post("/reorder")
-    async def reorder_notes(request: Request):
-        """Update sort_order for a list of note IDs in the order provided."""
-        user = _owner(request)
-        body = await request.json()
-        ids = body.get("ids", [])
-        if not isinstance(ids, list):
-            raise HTTPException(400, "ids must be a list")
-        # v2 review HIGH-12: drop the legacy `(owner == user) | (owner ==
-        # None)` OR which let an authenticated user silently reorder
-        # every legacy-null-owner note belonging to other accounts. In
-        # an unconfigured (single-user) auth deploy the OR is still safe
-        # because there's no second user to attack; we keep that branch
-        # explicit and gated on AuthManager.is_configured.
-        try:
-            from core.auth import AuthManager
-            _allow_null = not AuthManager().is_configured
-        except Exception:
-            _allow_null = False
-        db = SessionLocal()
-        try:
-            for i, nid in enumerate(ids):
-                q = db.query(Note).filter(Note.id == nid)
-                if user is not None:
-                    if _allow_null:
-                        q = q.filter((Note.owner == user) | (Note.owner == None))  # noqa: E711
-                    else:
-                        q = q.filter(Note.owner == user)
-                note = q.first()
-                if note:
-                    note.sort_order = i
-            db.commit()
-            return {"ok": True, "count": len(ids)}
-        finally:
-            db.close()
-
-    return router
+_sys.modules[__name__] = _canonical

routes/session_routes.py

@@ -12,7 +12,9 @@ from core.models import ChatMessage
 from src.request_models import SessionResponse
 from core.database import Session as DbSession, SessionLocal, Document, GalleryImage, utcnow_naive
 from src.auth_helpers import effective_user, _auth_disabled, owner_filter
+from src.session_image_cleanup import _generated_image_path_for_cleanup, session_image_refs
 from src.session_actions import is_session_recently_active
+from src.upload_handler import reserve_message_upload_references
 
 
 def _sanitize_export_filename(name: str) -> str:
@@ -203,7 +205,12 @@ def _pick_endpoint_for_sort(owner=None):
         return url, model, headers
     return None, None, None
 
-def setup_session_routes(session_manager: SessionManager, config: dict, webhook_manager=None):
+def setup_session_routes(
+    session_manager: SessionManager,
+    config: dict,
+    webhook_manager=None,
+    upload_handler=None,
+):
     """Setup session routes with the provided manager and config"""
 
     REQUEST_TIMEOUT = config.get("REQUEST_TIMEOUT", 20)
@@ -214,6 +221,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
     @router.get("/sessions")
     def list_sessions(request: Request):
         user = effective_user(request)
+        active_incognito_id = str(request.query_params.get("active_incognito_id") or "").strip()
         # Lazy purge: incognito sessions are ephemeral by design — wipe leftovers
         # from the DB and session_manager so they vanish on the next page refresh.
         # BUT: skip sessions that were created within the last 10 minutes.
@@ -234,6 +242,8 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
                     DbSession.created_at < _cutoff,
                 ).all()
                 for _g in _ghosts:
+                    if active_incognito_id and _g.id == active_incognito_id:
+                        continue
                     _purge_db.query(_DbMsg).filter(_DbMsg.session_id == _g.id).delete()
                     _purge_db.delete(_g)
                     if hasattr(session_manager, "delete_session"):
@@ -537,6 +547,22 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
         body = await request.json()
         messages = body.get("messages", [])
         from core.models import ChatMessage
+        owner = effective_user(request)
+        try:
+            for message in messages:
+                missing_id = reserve_message_upload_references(
+                    upload_handler,
+                    owner,
+                    message.get("content"),
+                    message.get("metadata"),
+                )
+                if missing_id:
+                    raise HTTPException(
+                        409,
+                        f"Referenced upload is no longer available: {missing_id}",
+                    )
+        except (AttributeError, TypeError, ValueError) as exc:
+            raise HTTPException(400, "Invalid message attachment metadata") from exc
         for m in messages:
             sess.add_message(ChatMessage(m["role"], m["content"], metadata=m.get("metadata")))
         session_manager.save_sessions()
@@ -619,13 +645,43 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
         db = SessionLocal()
         try:
             from core.database import ChatMessage as DbChatMessage
+            session_ids = [row[0] for row in db.query(DbSession.id).all()]
             count = db.query(DbSession).count()
+            image_ids: set[str] = set()
+            filenames: set[str] = set()
+            for sid in session_ids:
+                ids, names = session_image_refs(db, sid)
+                image_ids.update(ids)
+                filenames.update(names)
+            image_query = db.query(GalleryImage).filter(GalleryImage.session_id.in_(session_ids)) if session_ids else db.query(GalleryImage).filter(False)
+            if image_ids or filenames:
+                from sqlalchemy import or_
+                clauses = []
+                if session_ids:
+                    clauses.append(GalleryImage.session_id.in_(session_ids))
+                if image_ids:
+                    clauses.append(GalleryImage.id.in_(list(image_ids)))
+                if filenames:
+                    clauses.append(GalleryImage.filename.in_(list(filenames)))
+                image_query = db.query(GalleryImage).filter(or_(*clauses))
+            images = image_query.all()
+            removed_images = 0
+            for img in images:
+                img.is_active = False
+                if img.filename:
+                    path = _generated_image_path_for_cleanup(img.filename)
+                    if path and path.exists():
+                        try:
+                            path.unlink()
+                        except Exception as exc:
+                            logger.warning("Could not remove generated image %s during all-session delete: %s", img.filename, exc)
+                removed_images += 1
             db.query(DbChatMessage).delete()
             db.query(DbSession).delete()
             db.commit()
             session_manager.sessions.clear()
-            logger.info(f"Admin deleted all {count} sessions")
-            return {"status": "deleted", "count": count}
+            logger.info(f"Admin deleted all {count} sessions and {removed_images} linked images")
+            return {"status": "deleted", "count": count, "images_deleted": removed_images}
         except Exception as e:
             db.rollback()
             logger.error(f"Error deleting all sessions: {e}")

routes/shell_routes.py

@@ -157,6 +157,7 @@ def _package_installed_from_probe(name: str, probe: dict) -> bool:
     binaries = probe.get("binaries") if isinstance(probe.get("binaries"), dict) else {}
     dists = probe.get("dists") if isinstance(probe.get("dists"), dict) else {}
     modules = probe.get("modules") if isinstance(probe.get("modules"), dict) else {}
+    files = probe.get("files") if isinstance(probe.get("files"), dict) else {}
 
     if name == "vllm":
         return bool(binaries.get("vllm"))
@@ -166,11 +167,43 @@ def _package_installed_from_probe(name: str, probe: dict) -> bool:
         return bool(dists.get("sglang") or modules.get("sglang", {}).get("real_module"))
     if name == "mlx_lm":
         return bool(dists.get("mlx-lm") or modules.get("mlx_lm", {}).get("real_module"))
+    if name == "mflux":
+        return bool(
+            dists.get("mflux")
+            or modules.get("mflux", {}).get("real_module")
+            or binaries.get("mflux-generate-qwen")
+            or binaries.get("mflux-generate")
+        )
+    if name == "boogu_image_mlx":
+        return bool(
+            dists.get("boogu-image-mlx")
+            or modules.get("boogu_image_mlx", {}).get("real_module")
+        )
+    if name == "mlx_lama_swift":
+        return bool(
+            (binaries.get("odysseus-mlx-inpaint") or binaries.get("mlx-lama-serve"))
+            and (files.get("mlx.metallib") or files.get("default.metallib"))
+        )
+    if name == "mlx_ddcolor_swift":
+        return bool(
+            (binaries.get("odysseus-mlx-colorize") or binaries.get("mlx-ddcolor-serve"))
+            and (files.get("mlx.metallib") or files.get("default.metallib"))
+        )
     if name == "diffusers":
         return bool(
             (dists.get("diffusers") or modules.get("diffusers", {}).get("real_module"))
             and (dists.get("torch") or modules.get("torch", {}).get("real_module"))
         )
+    if name == "krea_diffusers":
+        return bool(
+            (dists.get("diffusers") or modules.get("diffusers", {}).get("real_module"))
+            and (dists.get("torch") or modules.get("torch", {}).get("real_module"))
+        )
+    if name == "sam_mask":
+        return bool(
+            (dists.get("transformers") or modules.get("transformers", {}).get("real_module"))
+            and (dists.get("torch") or modules.get("torch", {}).get("real_module"))
+        )
     if name == "hf_transfer":
         return bool(
             dists.get("hf-transfer")
@@ -183,6 +216,7 @@ def _package_status_note(name: str, probe: dict) -> str:
     binaries = probe.get("binaries") if isinstance(probe.get("binaries"), dict) else {}
     modules = probe.get("modules") if isinstance(probe.get("modules"), dict) else {}
     dists = probe.get("dists") if isinstance(probe.get("dists"), dict) else {}
+    files = probe.get("files") if isinstance(probe.get("files"), dict) else {}
     module = modules.get(name) if isinstance(modules.get(name), dict) else {}
     locations = module.get("locations") or []
     if name == "vllm":
@@ -212,10 +246,53 @@ def _package_status_note(name: str, probe: dict) -> str:
         if _package_installed_from_probe(name, probe):
             return f"diffusers {dists.get('diffusers', 'available')} with torch {dists.get('torch', 'available')}"
         return "Diffusers serving needs both diffusers and torch."
+    if name == "krea_diffusers":
+        if _package_installed_from_probe(name, probe):
+            return f"Latest Diffusers runtime: diffusers {dists.get('diffusers', 'available')} with torch {dists.get('torch', 'available')}. Use Update/Reinstall to pull latest Diffusers from Git."
+        return "Some newer image models need torch plus latest Diffusers from Git."
+    if name == "sam_mask":
+        if _package_installed_from_probe(name, probe):
+            return f"SAM object masks: transformers {dists.get('transformers', 'available')} with torch {dists.get('torch', 'available')}"
+        return "SAM click/object mask selection needs transformers and torch."
     if name == "mlx_lm":
         if _package_installed_from_probe(name, probe):
             return f"MLX LM {dists.get('mlx-lm', 'available')}"
         return "MLX serving needs mlx-lm on an Apple Silicon Mac."
+    if name == "mflux":
+        if _package_installed_from_probe(name, probe):
+            parts = []
+            if dists.get("mflux"):
+                parts.append(f"mflux {dists['mflux']}")
+            if binaries.get("mflux-generate-qwen"):
+                parts.append(f"Qwen CLI: {binaries['mflux-generate-qwen']}")
+            if binaries.get("mflux-generate"):
+                parts.append(f"Flux CLI: {binaries['mflux-generate']}")
+            return "; ".join(parts) if parts else "mflux available"
+        return "MLX image serving needs mflux on an Apple Silicon Mac."
+    if name == "boogu_image_mlx":
+        if _package_installed_from_probe(name, probe):
+            return f"Boogu MLX pipeline {dists.get('boogu-image-mlx', 'available')}"
+        return "Boogu image models need boogu-image-mlx on an Apple Silicon Mac."
+    if name == "mlx_lama_swift":
+        if _package_installed_from_probe(name, probe):
+            found = [
+                binaries.get("odysseus-mlx-inpaint"),
+                binaries.get("mlx-lama-serve"),
+            ]
+            return f"LaMa/MI-GAN Swift MLX runner: {next((p for p in found if p), 'available')}"
+        if binaries.get("odysseus-mlx-inpaint") or binaries.get("mlx-lama-serve"):
+            return "LaMa/MI-GAN Swift runner is installed, but mlx.metallib is missing next to the runner."
+        return "LaMa/MI-GAN inpainting models need an Odysseus-compatible mlx-lama-swift bridge on an Apple Silicon Mac."
+    if name == "mlx_ddcolor_swift":
+        if _package_installed_from_probe(name, probe):
+            found = [
+                binaries.get("odysseus-mlx-colorize"),
+                binaries.get("mlx-ddcolor-serve"),
+            ]
+            return f"DDColor Swift MLX runner: {next((p for p in found if p), 'available')}"
+        if binaries.get("odysseus-mlx-colorize") or binaries.get("mlx-ddcolor-serve"):
+            return "DDColor Swift runner is installed, but mlx.metallib is missing next to the runner."
+        return "DDColor colorization models need an Odysseus-compatible mlx-ddcolor-swift bridge on an Apple Silicon Mac."
     if name in dists:
         return f"{name} {dists[name]}"
     return ""
@@ -314,12 +391,22 @@ dist_names={{
     'llama_cpp':['llama-cpp-python'],
     'sglang':['sglang'],
     'mlx_lm':['mlx-lm'],
+    'mlx_vlm':['mlx-vlm'],
+    'mflux':['mflux'],
+    'boogu_image_mlx':['boogu-image-mlx'],
+    'mlx_lama_swift':[],
+    'mlx_ddcolor_swift':[],
     'diffusers':['diffusers','torch'],
+    'krea_diffusers':['diffusers','torch'],
+    'sam_mask':['transformers','torch'],
     'hf_transfer':['hf-transfer','hf_transfer'],
 }}
 bin_names={{
     'vllm':['vllm'],
     'llama_cpp':['llama-server'],
+    'mflux':['mflux-generate-qwen', 'mflux-generate'],
+    'mlx_lama_swift':['odysseus-mlx-inpaint', 'mlx-lama-serve'],
+    'mlx_ddcolor_swift':['odysseus-mlx-colorize', 'mlx-ddcolor-serve'],
     'tmux':['tmux'],
 }}
 
@@ -372,7 +459,19 @@ def probe(n):
         mods['torch'] = mod_status('torch')
     dists = dist_status(dist_names.get(n, [n]))
     bins = {{b: shutil.which(b) for b in bin_names.get(n, [])}}
-    return {{'modules': mods, 'dists': dists, 'binaries': bins}}
+    files = {{}}
+    if n in ('mlx_lama_swift', 'mlx_ddcolor_swift'):
+        for key in ('mlx.metallib', 'default.metallib'):
+            found = None
+            for b in bins.values():
+                if not b:
+                    continue
+                p = os.path.join(os.path.dirname(b), key)
+                if os.path.exists(p):
+                    found = p
+                    break
+            files[key] = found
+    return {{'modules': mods, 'dists': dists, 'binaries': bins, 'files': files}}
 
 print(json.dumps({{n: probe(n) for n in names}}))
 """
@@ -1088,6 +1187,8 @@ def setup_shell_routes() -> APIRouter:
         ssh_port: str | None = None,
         venv: str | None = None,
         backend: str | None = None,
+        platform: str | None = None,
+        model_hint: str | None = None,
     ):
         """Check which optional packages are installed.
 
@@ -1104,6 +1205,14 @@ def setup_shell_routes() -> APIRouter:
         import site
         import sys
 
+        platform_l = (platform or "").strip().lower()
+        model_hint_l = (model_hint or "").strip().lower()
+        has_krea_model = "krea" in model_hint_l
+        has_lama_mlx_model = any(
+            key in model_hint_l
+            for key in ("lama", "mi-gan", "migan", "inpainting-mlx")
+        )
+        has_ddcolor_mlx_model = "ddcolor" in model_hint_l
         _prepend_user_install_bins_to_path()
         importlib.invalidate_caches()
         try:
@@ -1158,7 +1267,7 @@ def setup_shell_routes() -> APIRouter:
                 "name": "hf_transfer",
                 "pip": "hf_transfer",
                 "desc": "Fast model downloads from HuggingFace",
-                "category": "LLM",
+                "category": "Tools",
                 "target": "remote",
             },
             {
@@ -1210,8 +1319,52 @@ def setup_shell_routes() -> APIRouter:
             # ── Image ── editor + diffusion model serving
             {
                 "name": "diffusers",
-                "pip": "diffusers[torch]",
-                "desc": "Image generation/editing pipelines (SD, Flux) with PyTorch",
+                "pip": "diffusers[torch] torchvision accelerate scipy python-multipart",
+                "desc": "Image generation/editing pipelines with PyTorch and Diffusers",
+                "category": "Image",
+                "target": "remote",
+            },
+            {
+                "name": "krea_diffusers",
+                "pip": "git+https://github.com/huggingface/diffusers.git torchvision accelerate scipy python-multipart",
+                "desc": "Latest Diffusers from Git for newly released image pipelines",
+                "category": "Image",
+                "target": "remote",
+            },
+            {
+                "name": "mflux",
+                "pip": "mflux",
+                "desc": "MLX image generation runtime for Apple Silicon models like Qwen Image",
+                "category": "Image",
+                "target": "remote",
+            },
+            {
+                "name": "boogu_image_mlx",
+                "pip": "git+https://github.com/xocialize/boogu-image-mlx.git",
+                "desc": "MLX image generation pipeline for Boogu Image models on Apple Silicon",
+                "category": "Image",
+                "target": "remote",
+            },
+            {
+                "name": "mlx_lama_swift",
+                "pip": "",
+                "desc": "Swift MLX runtime for LaMa / MI-GAN inpainting and object removal",
+                "category": "Image",
+                "target": "remote",
+                "install_hint": "Build an Odysseus-compatible mlx-lama-swift bridge on the selected Apple Silicon Mac and put odysseus-mlx-inpaint or mlx-lama-serve on PATH. Upstream currently ships Swift libraries plus smoke executables, not a stable image-edit CLI.",
+            },
+            {
+                "name": "mlx_ddcolor_swift",
+                "pip": "",
+                "desc": "Swift MLX runtime for DDColor automatic image colorization",
+                "category": "Image",
+                "target": "remote",
+                "install_hint": "Build an Odysseus-compatible mlx-ddcolor-swift bridge on the selected Apple Silicon Mac and put odysseus-mlx-colorize or mlx-ddcolor-serve on PATH. Upstream currently ships Swift libraries plus smoke executables, not a stable colorize CLI.",
+            },
+            {
+                "name": "mlx_vlm",
+                "pip": "mlx-vlm",
+                "desc": "MLX-VLM backbone used by HiDream image models on Apple Silicon",
                 "category": "Image",
                 "target": "remote",
             },
@@ -1222,6 +1375,13 @@ def setup_shell_routes() -> APIRouter:
                 "category": "Image",
                 "target": "remote",
             },
+            {
+                "name": "sam_mask",
+                "pip": "torch torchvision transformers accelerate pillow",
+                "desc": "Neutral click/box/object segmentation masks for the image editor",
+                "category": "Image",
+                "target": "local",
+            },
             {
                 "name": "rembg",
                 "pip": "rembg[gpu]",
@@ -1251,6 +1411,21 @@ def setup_shell_routes() -> APIRouter:
         for pkg in packages:
             pkg.setdefault("install_cmd", None)
             pkg.setdefault("update_cmd", None)
+        if not has_krea_model:
+            packages = [
+                p for p in packages
+                if p.get("name") not in {"krea_diffusers", "transformers"}
+            ]
+        if not has_lama_mlx_model:
+            packages = [
+                p for p in packages
+                if p.get("name") != "mlx_lama_swift"
+            ]
+        if not has_ddcolor_mlx_model:
+            packages = [
+                p for p in packages
+                if p.get("name") != "mlx_ddcolor_swift"
+            ]
         # Remote check: for remote-target packages, probe the selected server's
         # venv over SSH so a remote `pip install` actually reflects here.
         remote_status: dict = {}
@@ -1381,8 +1556,22 @@ def setup_shell_routes() -> APIRouter:
                 target_os_id = ""
             if sys.platform == "darwin":
                 target_os_id = "macos"
+        if not target_os_id and platform_l in {"darwin", "macos", "mac"}:
+            target_os_id = "macos"
 
         for pkg in packages:
+            if pkg.get("name") in {"mflux", "boogu_image_mlx", "mlx_vlm", "mlx_lama_swift", "mlx_ddcolor_swift"}:
+                is_apple_target = target_os_id == "macos" or (
+                    not host and IS_APPLE_SILICON
+                )
+                known_non_apple_target = bool(target_os_id and target_os_id != "macos") or (
+                    not host and not IS_APPLE_SILICON
+                )
+                pkg["applicable"] = is_apple_target
+                if known_non_apple_target:
+                    pkg["installed"] = None
+                    pkg["status_note"] = "Only relevant for Apple Silicon / MLX image serving."
+                    continue
             on_remote = bool(host and pkg.get("target") == "remote")
             probe = None
             if on_remote:
@@ -1588,6 +1777,10 @@ def setup_shell_routes() -> APIRouter:
             "sglang[all]",
             "diffusers",
             "diffusers[torch]",
+            "git+https://github.com/huggingface/diffusers.git",
+            "mflux",
+            "git+https://github.com/xocialize/boogu-image-mlx.git",
+            "mlx-vlm",
             "transformers",
             "TTS",
             "bark",

routes/upload_routes.py

@@ -10,16 +10,140 @@ from fastapi import APIRouter, Request, File, UploadFile, HTTPException, Form
 from typing import List, Optional
 import logging
 from core.middleware import require_admin
-from core.database import SessionLocal, GalleryImage, Session as DbSession
+from core.database import (
+    SessionLocal,
+    ChatMessage as DbChatMessage,
+    CalendarCal,
+    CalendarEvent,
+    Document,
+    DocumentVersion,
+    GalleryImage,
+    Note,
+    Session as DbSession,
+)
 from src.auth_helpers import effective_user
+from src.attachment_refs import attachment_refs_from_metadata
 from src.constants import GENERATED_IMAGES_DIR
-from src.upload_handler import count_recent_uploads
+from src.upload_handler import (
+    UploadCleanupSafetyError,
+    count_recent_uploads,
+    extract_upload_ids,
+)
 
 logger = logging.getLogger(__name__)
 
 router = APIRouter(prefix="/api/upload", tags=["upload"])
 UPLOAD_RESPONSE_HEADERS = {"X-Content-Type-Options": "nosniff"}
 
+def _upload_ids_from_persisted_text(value: object) -> set[str]:
+    """Return canonical upload IDs embedded in persisted text.
+
+    This covers attachment reference lines/URIs and the PDF source markers
+    stored by the document editor. False positives are intentionally
+    conservative: retaining an extra upload is safer than deleting referenced
+    bytes.
+    """
+    return extract_upload_ids(value)
+
+
+def _upload_ids_from_message_metadata(raw_metadata: object) -> set[str]:
+    """Extract attachment IDs from a persisted chat metadata JSON value.
+
+    Malformed metadata raises instead of being treated as an empty reference
+    set. The admin cleanup route catches that failure and aborts cleanup.
+    """
+    if raw_metadata in (None, ""):
+        return set()
+    if isinstance(raw_metadata, str):
+        metadata = json.loads(raw_metadata)
+    else:
+        metadata = raw_metadata
+    if not isinstance(metadata, dict):
+        raise ValueError("chat message metadata must be a JSON object")
+
+    attachments = metadata.get("attachments")
+    if attachments is not None:
+        if not isinstance(attachments, list) or any(
+            not isinstance(item, dict) for item in attachments
+        ):
+            raise ValueError("chat message attachments metadata is malformed")
+
+    ids = {
+        str(ref["attachment_id"])
+        for ref in attachment_refs_from_metadata(metadata)
+        if ref.get("attachment_id")
+    }
+    # Preserve canonical IDs even in older metadata shapes not normalized by
+    # attachment_refs_from_metadata().
+    ids.update(_upload_ids_from_persisted_text(json.dumps(metadata)))
+    return ids
+
+
+def _collect_persisted_upload_references() -> tuple[set[str], set[str]]:
+    """Collect upload IDs/hashes still referenced by durable application data.
+
+    The caller must treat any exception as an incomplete scan and fail closed.
+    There is no distinct artifact table in the current schema; artifact-like
+    attachment references persisted in chat/document text are covered by the
+    canonical-ID scan.
+    """
+    referenced_ids: set[str] = set()
+    referenced_hashes: set[str] = set()
+    db = SessionLocal()
+    try:
+        for content, raw_metadata in db.query(
+            DbChatMessage.content,
+            DbChatMessage.meta_data,
+        ).yield_per(500):
+            referenced_ids.update(_upload_ids_from_persisted_text(content))
+            referenced_ids.update(_upload_ids_from_message_metadata(raw_metadata))
+
+        for (content,) in db.query(Document.current_content).yield_per(500):
+            referenced_ids.update(_upload_ids_from_persisted_text(content))
+
+        for (content,) in db.query(DocumentVersion.content).yield_per(500):
+            referenced_ids.update(_upload_ids_from_persisted_text(content))
+
+        for filename, file_hash in db.query(
+            GalleryImage.filename,
+            GalleryImage.file_hash,
+        ).yield_per(500):
+            referenced_ids.update(_upload_ids_from_persisted_text(filename))
+            if file_hash:
+                referenced_hashes.add(str(file_hash))
+
+        for image_url, color, content, items in db.query(
+            Note.image_url,
+            Note.color,
+            Note.content,
+            Note.items,
+        ).yield_per(500):
+            for value in (image_url, color, content, items):
+                referenced_ids.update(_upload_ids_from_persisted_text(value))
+
+        for (color,) in db.query(CalendarCal.color).yield_per(500):
+            referenced_ids.update(_upload_ids_from_persisted_text(color))
+
+        for color, description, location in db.query(
+            CalendarEvent.color,
+            CalendarEvent.description,
+            CalendarEvent.location,
+        ).yield_per(500):
+            for value in (color, description, location):
+                referenced_ids.update(_upload_ids_from_persisted_text(value))
+
+        return referenced_ids, referenced_hashes
+    finally:
+        db.close()
+
+
+def _run_reference_safe_cleanup(upload_handler) -> int:
+    referenced_ids, referenced_hashes = _collect_persisted_upload_references()
+    return upload_handler.cleanup_old_uploads(
+        referenced_upload_ids=referenced_ids,
+        referenced_upload_hashes=referenced_hashes,
+    )
+
 def setup_upload_routes(upload_handler):
     """Setup upload routes with the provided handler"""
 
@@ -172,7 +296,9 @@ def setup_upload_routes(upload_handler):
                     "mime": meta["mime"],
                     "size": meta["size"],
                     "hash": meta["hash"],
+                    "checksum_sha256": meta.get("checksum_sha256") or meta["hash"],
                     "uploaded_at": meta["uploaded_at"],
+                    "created_at": meta.get("created_at") or meta["uploaded_at"],
                     "width": meta.get("width"),
                     "height": meta.get("height"),
                     "is_duplicate": meta.get("is_duplicate", False)
@@ -195,7 +321,23 @@ def setup_upload_routes(upload_handler):
     async def manual_cleanup(request: Request):
         """Manually trigger cleanup of old uploads."""
         require_admin(request)
-        cleaned_count = upload_handler.cleanup_old_uploads()
+        try:
+            cleaned_count = await asyncio.to_thread(
+                _run_reference_safe_cleanup,
+                upload_handler,
+            )
+        except UploadCleanupSafetyError:
+            logger.exception("Upload cleanup aborted because index safety checks failed")
+            raise HTTPException(
+                503,
+                "Upload cleanup aborted because upload index integrity could not be verified",
+            )
+        except Exception:
+            logger.exception("Upload cleanup skipped because reference discovery failed")
+            raise HTTPException(
+                503,
+                "Upload cleanup skipped because persisted references could not be verified",
+            )
         return {"status": "success", "files_cleaned": cleaned_count}
 
     @router.get("/stats")

scripts/check-docker-gpu.sh

@@ -214,6 +214,22 @@ _check_nvidia_smi() {
     echo
 }
 
+# WSL2 snap Docker cannot see /usr/lib/wsl/lib/libdxcore.so from its confined
+# namespace, so NVIDIA passthrough fails until the user switches to non-snap
+# Docker. DockerRootDir identifies snap installs more reliably than snap(8).
+_is_wsl() {
+    grep -qi microsoft /proc/version 2>/dev/null && return 0
+    [ -d /usr/lib/wsl ] && return 0
+    return 1
+}
+
+_is_docker_snap() {
+    case "$(docker info --format '{{.DockerRootDir}}' 2>/dev/null)" in
+        */snap/docker/*|*/snap.docker/*) return 0 ;;
+    esac
+    return 1
+}
+
 # Returns 1 if Docker is unavailable (callers should stop further GPU checks).
 _check_docker() {
     _info "Checking Docker..."
@@ -258,6 +274,26 @@ _check_gpu_passthrough() {
         echo
         _fail "GPU passthrough failed. Check these steps in order:"
         echo
+        if _is_wsl && _is_docker_snap; then
+            _warn "Detected: Docker installed via snap, running on WSL2."
+            _warn "This is a known incompatibility, not a toolkit/config problem:"
+            _warn "  snap confines Docker's mount namespace, so it cannot see the"
+            _warn "  WSL2-injected GPU library at /usr/lib/wsl/lib/libdxcore.so even"
+            _warn "  though the file exists on the host. Installing/reconfiguring"
+            _warn "  nvidia-container-toolkit will NOT fix this — the numbered"
+            _warn "  steps below will not help until Docker itself is replaced."
+            echo
+            _info "Fix: remove snap Docker and install the official apt-based Docker"
+            _info "Engine instead (unsandboxed, can see /usr/lib/wsl/lib):"
+            echo
+            echo "  sudo snap remove docker"
+            echo "  # then follow: https://docs.docker.com/engine/install/ubuntu/"
+            echo "  sudo nvidia-ctk runtime configure --runtime=docker"
+            echo "  sudo systemctl restart docker"
+            echo
+            _info "Re-run this script afterward to confirm passthrough works."
+            echo
+        fi
         echo "  1. Install NVIDIA Container Toolkit (if not already installed):"
         echo "     Arch:    sudo pacman -S nvidia-container-toolkit"
         echo "     Debian:  sudo apt install nvidia-container-toolkit"

scripts/diffusion_server.py

@@ -26,13 +26,14 @@ import io
 import json
 import logging
 import time
+import uuid
 from pathlib import Path
 
 from contextlib import asynccontextmanager
 
 import torch
 import uvicorn
-from fastapi import FastAPI
+from fastapi import FastAPI, File, Form, HTTPException, UploadFile
 from fastapi.middleware.cors import CORSMiddleware
 from starlette.middleware.trustedhost import TrustedHostMiddleware
 from pydantic import BaseModel
@@ -44,6 +45,7 @@ _pipe = None
 _model_id = ""
 DTYPE_MAP = {"bfloat16": torch.bfloat16, "float16": torch.float16, "float32": torch.float32}
 _args = None
+_PROGRESS = {}
 
 
 @asynccontextmanager
@@ -112,6 +114,77 @@ def _configure_security_middleware(application, allowed_hosts, allowed_origins):
 _configure_security_middleware(app, _DEFAULT_ALLOWED_HOSTS, _DEFAULT_CORS_ORIGINS)
 
 
+def _start_progress(request_id: str, total_steps: int, prompt: str, kind: str) -> str:
+    rid = (request_id or "").strip() or uuid.uuid4().hex
+    _PROGRESS[rid] = {
+        "id": rid,
+        "kind": kind,
+        "status": "running",
+        "step": 0,
+        "total": max(1, int(total_steps or 1)),
+        "percent": 0,
+        "prompt": (prompt or "")[:120],
+        "started_at": time.time(),
+        "updated_at": time.time(),
+    }
+    return rid
+
+
+def _update_progress(request_id: str, step: int, total_steps: int | None = None, status: str = "running"):
+    if not request_id:
+        return
+    item = _PROGRESS.get(request_id)
+    if not item:
+        return
+    total = max(1, int(total_steps or item.get("total") or 1))
+    current = max(0, min(int(step or 0), total))
+    item.update({
+        "status": status,
+        "step": current,
+        "total": total,
+        "percent": round((current / total) * 100, 1),
+        "updated_at": time.time(),
+    })
+
+
+def _finish_progress(request_id: str, status: str = "done", error: str = ""):
+    if not request_id:
+        return
+    item = _PROGRESS.get(request_id)
+    if not item:
+        return
+    total = max(1, int(item.get("total") or 1))
+    item.update({
+        "status": status,
+        "step": total if status == "done" else item.get("step", 0),
+        "total": total,
+        "percent": 100 if status == "done" else item.get("percent", 0),
+        "error": error,
+        "updated_at": time.time(),
+    })
+
+
+def _run_pipeline_with_progress(pipe, request_id: str, total_steps: int, **kwargs):
+    def step_end_callback(_pipe, step, timestep, callback_kwargs):
+        _update_progress(request_id, int(step) + 1, total_steps)
+        return callback_kwargs
+
+    def legacy_callback(step, timestep, latents):
+        _update_progress(request_id, int(step) + 1, total_steps)
+
+    try:
+        return pipe(callback_on_step_end=step_end_callback, **kwargs)
+    except TypeError as exc:
+        if "callback_on_step_end" not in str(exc):
+            raise
+    try:
+        return pipe(callback=legacy_callback, callback_steps=1, **kwargs)
+    except TypeError as exc:
+        if "callback" not in str(exc) and "callback_steps" not in str(exc):
+            raise
+    return pipe(**kwargs)
+
+
 class ImageRequest(BaseModel):
     model: str = ""
     prompt: str
@@ -119,10 +192,71 @@ class ImageRequest(BaseModel):
     size: str = "1024x1024"
     quality: str = "medium"
     response_format: str = "b64_json"
+    request_id: str = ""
+
+
+def _parse_size(size: str) -> tuple[int, int]:
+    try:
+        w, h = (size or "").split("x")
+        return int(w), int(h)
+    except Exception:
+        return _args.width, _args.height
+
+
+def _quality_steps(quality: str) -> int:
+    default_steps = _args.steps or 8
+    steps_map = {"low": 4, "medium": default_steps, "high": 20, "auto": 12}
+    return steps_map.get(quality, default_steps)
+
+
+def _guidance_scale() -> float:
+    return float(_args.guidance_scale)
+
+
+def _default_negative_prompt() -> str | None:
+    value = (_args.negative_prompt or "").strip()
+    return value or None
+
+
+def _pipeline_accepts_arg(name: str) -> bool:
+    try:
+        import inspect
+        sig = inspect.signature(_pipe.__call__)
+        return name in sig.parameters
+    except Exception:
+        return True
+
+
+def _pipeline_call_kwargs(**kwargs) -> dict:
+    """Filter kwargs to the active pipeline signature.
+
+    Diffusers/community pipelines are not consistent: ordinary img2img
+    usually accepts `image`, while instruction-edit models such as OmniGen2
+    accept `input_images` plus model-specific guidance fields. Filtering keeps
+    the server generic and avoids hardcoding repo IDs.
+    """
+    try:
+        import inspect
+        sig = inspect.signature(_pipe.__call__)
+        params = sig.parameters
+        if any(p.kind == inspect.Parameter.VAR_KEYWORD for p in params.values()):
+            return {k: v for k, v in kwargs.items() if v is not None}
+        return {k: v for k, v in kwargs.items() if v is not None and k in params}
+    except Exception:
+        return {k: v for k, v in kwargs.items() if v is not None}
+
+
+def _image_response(images) -> dict:
+    data = []
+    for img in images:
+        buf = io.BytesIO()
+        img.save(buf, format="PNG")
+        data.append({"b64_json": base64.b64encode(buf.getvalue()).decode()})
+    return {"created": int(time.time()), "data": data}
 
 
 def _fix_meta_tensors(pipe, dtype):
-    """Replace any meta tensors with real zero tensors on CPU so .to(cuda) works."""
+    """Replace any meta tensors with real zero tensors on CPU so .to(device) works."""
     for name, component in pipe.components.items():
         if not hasattr(component, 'parameters'):
             continue
@@ -142,6 +276,69 @@ def _fix_meta_tensors(pipe, dtype):
             logger.info(f"  Fixed {fixed} meta tensors in {name}")
 
 
+def _target_device() -> str:
+    """Best available torch device for Diffusers on this host."""
+    try:
+        if torch.cuda.is_available():
+            return "cuda"
+    except Exception:
+        pass
+    try:
+        if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
+            return "mps"
+    except Exception:
+        pass
+    return "cpu"
+
+
+def _can_cpu_offload(device: str) -> bool:
+    # Diffusers CPU offload helpers are CUDA/accelerate-oriented. On Apple
+    # Silicon they either no-op poorly or fail; MPS should use direct .to("mps").
+    return device == "cuda"
+
+
+def _load_omnigen2_pipeline(model_path: str, torch_dtype, target_device: str, use_offload: bool) -> bool:
+    """Load OmniGen2 from the official repo package.
+
+    OmniGen2 publishes a Diffusers-style model_index.json, but current
+    public diffusers builds do not expose OmniGen2Pipeline as a stock class.
+    The official examples import the pipeline from the cloned `omnigen2`
+    package, so support that path without hardcoding any private model.
+    """
+    global _pipe
+    try:
+        from omnigen2.pipelines.omnigen2.pipeline_omnigen2 import OmniGen2Pipeline
+        from omnigen2.models.transformers.transformer_omnigen2 import OmniGen2Transformer2DModel
+    except Exception as exc:
+        logger.warning("OmniGen2 package import failed: %s", exc)
+        return False
+
+    try:
+        logger.info("Loading OmniGen2 pipeline via official omnigen2 package")
+        pipe = OmniGen2Pipeline.from_pretrained(
+            model_path,
+            torch_dtype=torch_dtype,
+            trust_remote_code=True,
+        )
+        pipe.transformer = OmniGen2Transformer2DModel.from_pretrained(
+            model_path,
+            subfolder="transformer",
+            torch_dtype=torch_dtype,
+        )
+        if use_offload and _can_cpu_offload(target_device):
+            pipe.enable_model_cpu_offload()
+            logger.info("Loaded OmniGen2 with CPU offload")
+        else:
+            pipe = pipe.to(target_device)
+            logger.info("Loaded OmniGen2 on %s", target_device)
+        _pipe = pipe
+        return True
+    except Exception as exc:
+        logger.warning("Official OmniGen2 loader failed: %s", exc)
+        _pipe = None
+        return False
+
+
 def load_model():
     global _pipe, _model_id
     import diffusers
@@ -151,8 +348,12 @@ def load_model():
     dtype_map = {"bfloat16": torch.bfloat16, "float16": torch.float16, "float32": torch.float32}
     torch_dtype = dtype_map.get(_args.dtype, torch.bfloat16)
     use_offload = _args.cpu_offload
+    target_device = _target_device()
+    if target_device != "cuda" and use_offload:
+        logger.warning("CPU offload requested but %s is the active device; using direct device placement instead", target_device)
+        use_offload = False
 
-    logger.info(f"Loading model from {model_path} (dtype={_args.dtype}, offload={use_offload})...")
+    logger.info(f"Loading model from {model_path} (dtype={_args.dtype}, offload={use_offload}, device={target_device})...")
 
     # Ensure HF token is available for gated repos
     _hf_token = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
@@ -207,23 +408,45 @@ def load_model():
         except Exception as e:
             logger.debug(f"GPU cache clear failed: {e}")
 
+    loaded = False
+    if cls_name_from_index == "OmniGen2Pipeline" or "omnigen2" in str(model_path).lower():
+        loaded = _load_omnigen2_pipeline(model_path, torch_dtype, target_device, use_offload)
+
     def _load_pipe(cls, name):
         """Try loading pipeline, handling meta tensor issues."""
         global _pipe
 
         # First try normal load
         try:
-            _pipe = cls.from_pretrained(model_path, torch_dtype=torch_dtype)
+            kwargs = {"torch_dtype": torch_dtype}
+            if name == "DiffusionPipeline" and cls_name_from_index and not hasattr(diffusers, cls_name_from_index):
+                kwargs["trust_remote_code"] = True
+            _pipe = cls.from_pretrained(model_path, **kwargs)
         except Exception as e:
             logger.warning(f"{name} from_pretrained failed: {e}")
-            _pipe = None
-            _cleanup()
-            return False
+            if name == "DiffusionPipeline" and cls_name_from_index and not hasattr(diffusers, cls_name_from_index):
+                try:
+                    logger.info(f"Retrying {name} with custom_pipeline={model_path}")
+                    _pipe = cls.from_pretrained(
+                        model_path,
+                        torch_dtype=torch_dtype,
+                        custom_pipeline=model_path,
+                        trust_remote_code=True,
+                    )
+                except Exception as e2:
+                    logger.warning(f"{name} custom_pipeline retry failed: {e2}")
+                    _pipe = None
+                    _cleanup()
+                    return False
+            else:
+                _pipe = None
+                _cleanup()
+                return False
 
         # Materialize any meta tensors before moving to device
         _fix_meta_tensors(_pipe, torch_dtype)
 
-        if use_offload:
+        if use_offload and _can_cpu_offload(target_device):
             try:
                 _pipe.enable_model_cpu_offload()
                 logger.info(f"Loaded as {name} with CPU offload")
@@ -234,24 +457,27 @@ def load_model():
                 _cleanup()
                 return False
 
-        # Try full CUDA
+        # Try direct device placement
         try:
-            _pipe = _pipe.to("cuda")
-            logger.info(f"Loaded as {name} on CUDA")
+            _pipe = _pipe.to(target_device)
+            logger.info(f"Loaded as {name} on {target_device}")
             return True
         except Exception as e:
-            logger.warning(f"{name} + .to(cuda) failed: {e}")
+            logger.warning(f"{name} + .to({target_device}) failed: {e}")
             _pipe = None
             _cleanup()
 
         if not use_offload:
-            logger.error(f"{name} doesn't fit in VRAM. Use --cpu-offload to enable offloading.")
+            logger.error(f"{name} could not be placed on {target_device}. On CUDA, try --cpu-offload; on Apple Silicon try a smaller model or lower resolution.")
             return False
 
         # OOM — reload and try with CPU offload
         try:
             logger.info(f"Reloading {name} with CPU offload...")
-            _pipe = cls.from_pretrained(model_path, torch_dtype=torch_dtype)
+            kwargs = {"torch_dtype": torch_dtype}
+            if name == "DiffusionPipeline" and cls_name_from_index and not hasattr(diffusers, cls_name_from_index):
+                kwargs["trust_remote_code"] = True
+            _pipe = cls.from_pretrained(model_path, **kwargs)
             _fix_meta_tensors(_pipe, torch_dtype)
             _pipe.enable_model_cpu_offload()
             logger.info(f"Loaded as {name} with CPU offload")
@@ -264,7 +490,10 @@ def load_model():
         # Last resort — sequential offload
         try:
             logger.info(f"Reloading {name} with sequential CPU offload...")
-            _pipe = cls.from_pretrained(model_path, torch_dtype=torch_dtype)
+            kwargs = {"torch_dtype": torch_dtype}
+            if name == "DiffusionPipeline" and cls_name_from_index and not hasattr(diffusers, cls_name_from_index):
+                kwargs["trust_remote_code"] = True
+            _pipe = cls.from_pretrained(model_path, **kwargs)
             _fix_meta_tensors(_pipe, torch_dtype)
             _pipe.enable_sequential_cpu_offload()
             logger.info(f"Loaded as {name} with sequential CPU offload")
@@ -276,11 +505,11 @@ def load_model():
 
         return False
 
-    loaded = False
-    for cls, name in candidates:
-        if _load_pipe(cls, name):
-            loaded = True
-            break
+    if not loaded:
+        for cls, name in candidates:
+            if _load_pipe(cls, name):
+                loaded = True
+                break
 
     # Last resort: override unknown pipeline class
     if not loaded and cls_name_from_index and not hasattr(diffusers, cls_name_from_index):
@@ -322,36 +551,19 @@ def load_model():
 
         if single_file:
             logger.info(f"Trying from_single_file with: {single_file}")
-            # Detect model family from path/filename to prioritize the right pipeline + config
-            _path_lower = (model_path + "/" + (single_file or "")).lower()
-            _SD35_CONFIGS = ["stabilityai/stable-diffusion-3.5-large", "stabilityai/stable-diffusion-3.5-medium"]
-            _SD3_CONFIGS = ["stabilityai/stable-diffusion-3-medium-diffusers"]
-            _FLUX2_CONFIGS = ["black-forest-labs/FLUX.2-dev"]
-            _FLUX_CONFIGS = ["black-forest-labs/FLUX.1-schnell", "black-forest-labs/FLUX.1-dev"]
-            _SDXL_CONFIGS = ["stabilityai/stable-diffusion-xl-base-1.0"]
-
-            # Build ordered pipeline candidates based on model name hints
-            _pipeline_configs = []
-            if "sd3.5" in _path_lower or "stable-diffusion-3.5" in _path_lower:
-                _pipeline_configs.append(("StableDiffusion3Pipeline", _SD35_CONFIGS))
-            elif "sd3" in _path_lower or "stable-diffusion-3" in _path_lower:
-                _pipeline_configs.append(("StableDiffusion3Pipeline", _SD3_CONFIGS + _SD35_CONFIGS))
-            elif "flux.2" in _path_lower or "flux2" in _path_lower:
-                _pipeline_configs.append(("Flux2Pipeline", _FLUX2_CONFIGS))
-                _pipeline_configs.append(("FluxPipeline", _FLUX_CONFIGS))
-            elif "flux" in _path_lower:
-                _pipeline_configs.append(("FluxPipeline", _FLUX_CONFIGS))
-                _pipeline_configs.append(("Flux2Pipeline", _FLUX2_CONFIGS))
-            elif "sdxl" in _path_lower or "xl" in _path_lower:
-                _pipeline_configs.append(("StableDiffusionXLPipeline", _SDXL_CONFIGS))
-            # Always add all pipelines as fallbacks
-            _pipeline_configs.extend([
-                ("Flux2Pipeline", _FLUX2_CONFIGS),
-                ("StableDiffusion3Pipeline", _SD35_CONFIGS + _SD3_CONFIGS),
-                ("FluxPipeline", _FLUX_CONFIGS),
-                ("StableDiffusionXLPipeline", _SDXL_CONFIGS + [None]),
-                ("StableDiffusionPipeline", [None]),
-            ])
+            explicit_configs = [
+                c.strip()
+                for c in str(_args.single_file_config or "").replace("\n", ",").split(",")
+                if c.strip()
+            ]
+            config_candidates = explicit_configs or [None]
+            _pipeline_configs = [
+                ("Flux2Pipeline", config_candidates),
+                ("StableDiffusion3Pipeline", config_candidates),
+                ("FluxPipeline", config_candidates),
+                ("StableDiffusionXLPipeline", config_candidates),
+                ("StableDiffusionPipeline", config_candidates),
+            ]
             # Deduplicate while preserving order
             _seen = set()
             _deduped = []
@@ -409,12 +621,12 @@ def load_model():
                             logger.info(f"Trying {cls_name}.from_single_file with config={config}")
                         _pipe = cls.from_single_file(single_file, **kwargs)
                         _fix_meta_tensors(_pipe, torch_dtype)
-                        if use_offload:
+                        if use_offload and _can_cpu_offload(target_device):
                             _pipe.enable_model_cpu_offload()
                             logger.info(f"Loaded as {cls_name} (single file, config={config}) with CPU offload")
                         else:
-                            _pipe = _pipe.to("cuda")
-                            logger.info(f"Loaded as {cls_name} (single file, config={config}) on CUDA")
+                            _pipe = _pipe.to(target_device)
+                            logger.info(f"Loaded as {cls_name} (single file, config={config}) on {target_device}")
                         loaded = True
                         break
                     except Exception as e:
@@ -480,17 +692,9 @@ def generate_image(req: ImageRequest):
     if _pipe is None:
         return {"error": "Model not loaded"}
 
-    # Parse size
-    try:
-        w, h = req.size.split("x")
-        width, height = int(w), int(h)
-    except Exception:
-        width, height = _args.width, _args.height
-
-    # Map quality to num_inference_steps
-    default_steps = _args.steps or 8
-    steps_map = {"low": 4, "medium": default_steps, "high": 20, "auto": 12}
-    steps = steps_map.get(req.quality, default_steps)
+    width, height = _parse_size(req.size)
+    steps = _quality_steps(req.quality)
+    request_id = _start_progress(req.request_id, steps * max(1, int(req.n or 1)), req.prompt, "generation")
 
     logger.info(f"Generating: {req.prompt[:80]}... ({width}x{height}, {steps} steps)")
     start = time.time()
@@ -499,44 +703,172 @@ def generate_image(req: ImageRequest):
     _is_inpaint_pipe = 'inpaint' in type(_pipe).__name__.lower()
 
     images = []
-    for _ in range(req.n):
-        if _is_inpaint_pipe:
-            # Inpaint pipelines need an image + mask — create blank ones for txt2img
-            from PIL import Image as _PILGen
-            _blank = _PILGen.new('RGB', (width, height), (128, 128, 128))
-            _mask = _PILGen.new('L', (width, height), 255)  # full white = regenerate everything
-            result = _pipe(
-                prompt=req.prompt,
-                image=_blank,
-                mask_image=_mask,
-                width=width,
-                height=height,
-                num_inference_steps=steps,
-                guidance_scale=3.5,
+    try:
+        for image_index in range(req.n):
+            progress_offset = image_index * steps
+            negative_prompt = _default_negative_prompt() if _pipeline_accepts_arg("negative_prompt") else None
+            if _is_inpaint_pipe:
+                # Inpaint pipelines need an image + mask — create blank ones for txt2img
+                from PIL import Image as _PILGen
+                _blank = _PILGen.new('RGB', (width, height), (128, 128, 128))
+                _mask = _PILGen.new('L', (width, height), 255)  # full white = regenerate everything
+                kwargs = {
+                    "prompt": req.prompt,
+                    "image": _blank,
+                    "mask_image": _mask,
+                    "width": width,
+                    "height": height,
+                    "num_inference_steps": steps,
+                    "guidance_scale": _guidance_scale(),
+                }
+            else:
+                kwargs = {
+                    "prompt": req.prompt,
+                    "width": width,
+                    "height": height,
+                    "num_inference_steps": steps,
+                    "guidance_scale": _guidance_scale(),
+                }
+            if negative_prompt:
+                kwargs["negative_prompt"] = negative_prompt
+            result = _run_pipeline_with_progress(
+                _pipe,
+                request_id,
+                steps * max(1, int(req.n or 1)),
+                **kwargs,
             )
-        else:
-            result = _pipe(
-                prompt=req.prompt,
-                width=width,
-                height=height,
-                num_inference_steps=steps,
-                guidance_scale=3.5,
-            )
-        img = result.images[0]
-
-        # Convert to base64
-        buf = io.BytesIO()
-        img.save(buf, format="PNG")
-        b64 = base64.b64encode(buf.getvalue()).decode()
-        images.append({"b64_json": b64})
+            _update_progress(request_id, progress_offset + steps, steps * max(1, int(req.n or 1)))
+            img = result.images[0]
+            images.append(img)
+    except Exception as e:
+        _finish_progress(request_id, "error", str(e))
+        raise
 
     elapsed = time.time() - start
     logger.info(f"Generated {req.n} image(s) in {elapsed:.1f}s")
+    _finish_progress(request_id)
 
-    return {
-        "created": int(time.time()),
-        "data": images,
-    }
+    return _image_response(images)
+
+
+@app.get("/v1/images/progress/{request_id}")
+def image_progress(request_id: str):
+    item = _PROGRESS.get(request_id)
+    if not item:
+        return {"id": request_id, "status": "unknown", "step": 0, "total": 0, "percent": 0}
+    return item
+
+
+@app.post("/v1/images/edits")
+async def edit_image(
+    prompt: str = Form(...),
+    image: UploadFile = File(...),
+    model: str = Form(""),
+    n: int = Form(1),
+    size: str = Form("1024x1024"),
+    quality: str = Form("medium"),
+    response_format: str = Form("b64_json"),
+    request_id: str = Form(""),
+):
+    if _pipe is None:
+        return {"error": "Model not loaded"}
+    accepts_image = _pipeline_accepts_arg("image")
+    accepts_input_images = _pipeline_accepts_arg("input_images")
+    if not accepts_image and not accepts_input_images:
+        raise HTTPException(
+            status_code=400,
+            detail=f"{type(_pipe).__name__} does not support image edits. Use /v1/images/generations with this model.",
+        )
+
+    from PIL import Image as PILImage, ImageOps
+
+    width, height = _parse_size(size)
+    steps = _quality_steps(quality)
+    request_id = _start_progress(request_id, steps * max(1, min(int(n or 1), 4)), prompt, "edit")
+    raw = await image.read()
+    init_image = PILImage.open(io.BytesIO(raw)).convert("RGB")
+    if width > 0 and height > 0:
+        init_image = ImageOps.fit(init_image, (width, height), method=PILImage.LANCZOS, centering=(0.5, 0.5))
+
+    logger.info(f"Editing image: {prompt[:80]}... ({width}x{height}, {steps} steps)")
+    start = time.time()
+    images = []
+    total_images = max(1, min(int(n or 1), 4))
+    for image_index in range(total_images):
+        progress_offset = image_index * steps
+        try:
+            if accepts_input_images and not accepts_image:
+                negative_prompt = _default_negative_prompt()
+                kwargs = _pipeline_call_kwargs(
+                    prompt=prompt,
+                    input_images=[init_image],
+                    width=width,
+                    height=height,
+                    num_inference_steps=steps,
+                    max_sequence_length=1024,
+                    text_guidance_scale=_guidance_scale(),
+                    image_guidance_scale=2.0,
+                    cfg_range=(0.0, 1.0),
+                    negative_prompt=negative_prompt,
+                    num_images_per_prompt=1,
+                    output_type="pil",
+                    max_pixels=width * height if width > 0 and height > 0 else None,
+                    max_input_image_side_length=max(width, height) if width > 0 and height > 0 else None,
+                )
+            else:
+                kwargs = _pipeline_call_kwargs(
+                    image=init_image,
+                    prompt=prompt,
+                    width=width,
+                    height=height,
+                    num_inference_steps=steps,
+                    guidance_scale=3.5,
+                    true_cfg_scale=4.0,
+                    negative_prompt=_default_negative_prompt(),
+                    output_type="pil",
+                )
+            result = _run_pipeline_with_progress(
+                _pipe,
+                request_id,
+                steps * total_images,
+                **kwargs,
+            )
+        except TypeError:
+            if accepts_input_images and not accepts_image:
+                kwargs = _pipeline_call_kwargs(
+                    prompt=prompt,
+                    input_images=[init_image],
+                    num_inference_steps=steps,
+                    text_guidance_scale=_guidance_scale(),
+                    image_guidance_scale=2.0,
+                    negative_prompt=_default_negative_prompt(),
+                    output_type="pil",
+                )
+            else:
+                kwargs = _pipeline_call_kwargs(
+                    image=init_image,
+                    prompt=prompt,
+                    num_inference_steps=steps,
+                    guidance_scale=3.5,
+                    negative_prompt=_default_negative_prompt(),
+                    output_type="pil",
+                )
+            result = _run_pipeline_with_progress(
+                _pipe,
+                request_id,
+                steps * total_images,
+                **kwargs,
+            )
+        except Exception as e:
+            _finish_progress(request_id, "error", str(e))
+            raise
+        _update_progress(request_id, progress_offset + steps, steps * total_images)
+        images.append(result.images[0])
+
+    elapsed = time.time() - start
+    logger.info(f"Edited {len(images)} image(s) in {elapsed:.1f}s")
+    _finish_progress(request_id)
+    return _image_response(images)
 
 
 class InpaintRequest(BaseModel):
@@ -607,11 +939,12 @@ def _get_inpaint_pipe():
     ]
     torch_dtype = DTYPE_MAP.get(_args.dtype, torch.bfloat16)
     harmonize_gpu = _args.harmonize_gpu
+    target_device = _target_device()
     for name in img2img_names:
         cls = getattr(diffusers, name, None)
         if cls:
             try:
-                if harmonize_gpu is not None:
+                if harmonize_gpu is not None and target_device == "cuda":
                     # Load fresh on separate GPU
                     logger.info(f"Loading {name} on cuda:{harmonize_gpu}...")
                     _img2img_pipe = cls.from_pretrained(_args.model, torch_dtype=torch_dtype)
@@ -625,10 +958,10 @@ def _get_inpaint_pipe():
                 try:
                     # Some pipelines need from_pretrained instead of from_pipe
                     _img2img_pipe = cls.from_pretrained(_args.model, torch_dtype=torch_dtype)
-                    if _args.cpu_offload:
+                    if _args.cpu_offload and _can_cpu_offload(target_device):
                         _img2img_pipe.enable_model_cpu_offload()
                     else:
-                        _img2img_pipe = _img2img_pipe.to("cuda")
+                        _img2img_pipe = _img2img_pipe.to(target_device)
                     logger.info(f"Loaded img2img pipeline (from_pretrained): {name}")
                     return _img2img_pipe, 'img2img'
                 except Exception as e2:
@@ -1140,6 +1473,9 @@ if __name__ == "__main__":
     parser.add_argument("--dtype", default="bfloat16", choices=["bfloat16", "float16", "float32"])
     parser.add_argument("--device-map", default=None, help="Device map strategy (unused, kept for compat)")
     parser.add_argument("--steps", type=int, default=0, help="Default inference steps (0=auto)")
+    parser.add_argument("--guidance-scale", type=float, default=3.5, help="Default classifier-free guidance scale")
+    parser.add_argument("--negative-prompt", default="", help="Default negative prompt for pipelines that support it")
+    parser.add_argument("--single-file-config", default="", help="Base Diffusers repo/path for single-file checkpoints that need missing components. Comma-separated values are tried in order.")
     parser.add_argument("--width", type=int, default=1024, help="Default output width")
     parser.add_argument("--height", type=int, default=1024, help="Default output height")
     parser.add_argument("--cpu-offload", action="store_true", help="Enable model CPU offload")

scripts/mlx_image_server.py

@@ -0,0 +1,465 @@
+#!/usr/bin/env python3
+"""OpenAI-compatible image API wrapper for MLX image models.
+
+This is intentionally small: it exposes the same `/v1/images/generations`
+shape Odysseus already uses for local image endpoints, then delegates to the
+MLX image CLI for the actual generation. Text MLX models still use
+`mlx_lm.server`; image MLX models should use this wrapper.
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import logging
+from pathlib import Path
+
+import uvicorn
+from fastapi import FastAPI, File, Form, HTTPException, UploadFile
+from pydantic import BaseModel
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger("mlx_image_server")
+
+
+class ImageRequest(BaseModel):
+    model: str = ""
+    prompt: str
+    n: int = 1
+    size: str = "1024x1024"
+    quality: str = "medium"
+    response_format: str = "b64_json"
+
+
+class HarmonizeRequest(BaseModel):
+    image: str
+    prompt: str = ""
+    mask: str | None = None
+    body_mask: str | None = None
+    seam_mask: str | None = None
+    strength: float = 0.35
+
+
+app = FastAPI(title="Odysseus MLX Image Server")
+_args: argparse.Namespace
+
+
+def _steps(quality: str) -> int:
+    if _args.steps:
+        return int(_args.steps)
+    return {"low": 8, "medium": 20, "high": 32, "auto": 20}.get((quality or "medium").lower(), 20)
+
+
+def _size(size: str) -> tuple[int, int]:
+    try:
+        w, h = str(size or "").lower().split("x", 1)
+        return max(64, int(w)), max(64, int(h))
+    except Exception:
+        return int(_args.width), int(_args.height)
+
+
+def _cli_for_model(model: str) -> str:
+    lower = model.lower()
+    if "qwen" in lower:
+        return "mflux-generate-qwen"
+    if "flux" in lower:
+        return "mflux-generate"
+    return "mflux-generate"
+
+
+def _resolve_cli(name: str) -> str:
+    found = shutil.which(name)
+    if found:
+        return found
+    local = Path(sys.executable).resolve().parent / name
+    if local.exists():
+        return str(local)
+    prefix_local = Path(sys.prefix).resolve() / "bin" / name
+    if prefix_local.exists():
+        return str(prefix_local)
+    return ""
+
+
+def _valid_numbers(values: list[str]) -> list[str]:
+    out: list[str] = []
+    for value in values or []:
+        s = str(value).strip()
+        if not s:
+            continue
+        try:
+            float(s)
+        except Exception:
+            continue
+        out.append(s)
+    return out
+
+
+def _is_hidream(model: str) -> bool:
+    return "hidream" in (model or "").lower()
+
+
+def _is_boogu(model: str) -> bool:
+    return "boogu" in (model or "").lower()
+
+
+def _is_lama_inpaint(model: str) -> bool:
+    lower = (model or "").lower()
+    return "mi-gan" in lower or "migan" in lower or "lama" in lower
+
+
+def _is_ddcolor(model: str) -> bool:
+    return "ddcolor" in (model or "").lower()
+
+
+def _unsupported_swift_mlx_runtime(model: str) -> HTTPException:
+    if _is_ddcolor(model):
+        return HTTPException(
+            503,
+            "DDColor MLX models require an Odysseus-compatible mlx-ddcolor-swift bridge. "
+            "Build/install a bridge binary named odysseus-mlx-colorize or mlx-ddcolor-serve "
+            "on the Apple Silicon host PATH. Upstream currently ships Swift libraries and "
+            "smoke executables, not a stable colorize CLI.",
+        )
+    return HTTPException(
+        503,
+        "LaMa / MI-GAN MLX inpainting models require an Odysseus-compatible mlx-lama-swift bridge. "
+        "Build/install a bridge binary named odysseus-mlx-inpaint or mlx-lama-serve "
+        "on the Apple Silicon host PATH. Upstream currently ships Swift libraries and "
+        "smoke executables, not a stable image-edit CLI.",
+    )
+
+
+def _resolve_bridge(names: list[str]) -> str:
+    for name in names:
+        found = _resolve_cli(name)
+        if found:
+            return found
+    return ""
+
+
+def _snapshot_path(model: str) -> Path:
+    p = Path(model).expanduser()
+    if p.exists():
+        return p
+    try:
+        from huggingface_hub import snapshot_download
+    except Exception as e:
+        raise HTTPException(
+            503,
+            "huggingface_hub is required to download MLX image model snapshots. "
+            "Install the model requirements in the selected Python environment.",
+        ) from e
+    return Path(snapshot_download(model))
+
+
+def _weights_path(model: str) -> Path:
+    p = Path(model).expanduser()
+    if p.is_file():
+        return p
+    snap = _snapshot_path(model)
+    if snap.is_file():
+        return snap
+    candidates = sorted(snap.rglob("*.safetensors"))
+    if not candidates:
+        raise HTTPException(500, f"No safetensors weights found for {model} in {snap}")
+    return candidates[0]
+
+
+def _write_bridge_input_image(raw: bytes, out_path: Path) -> None:
+    try:
+        from PIL import Image
+        import io
+    except Exception as e:
+        raise HTTPException(503, "Pillow is required for MLX image edit bridge inputs.") from e
+    try:
+        img = Image.open(io.BytesIO(raw)).convert("RGBA")
+        img.save(out_path, format="PNG")
+    except Exception as e:
+        raise HTTPException(400, f"Invalid input image: {e}") from e
+
+
+def _write_bridge_mask(raw: bytes, out_path: Path) -> None:
+    try:
+        from PIL import Image
+        import io
+    except Exception as e:
+        raise HTTPException(503, "Pillow is required for MLX image edit bridge masks.") from e
+    try:
+        img = Image.open(io.BytesIO(raw))
+        if img.mode == "RGBA":
+            # OpenAI edits mask convention: transparent = regenerate.
+            alpha = img.getchannel("A")
+            mask = alpha.point(lambda p: 255 if p < 128 else 0)
+        else:
+            mask = img.convert("L")
+        mask.save(out_path, format="PNG")
+    except Exception as e:
+        raise HTTPException(400, f"Invalid mask image: {e}") from e
+
+
+def _run_bridge(cmd: list[str]) -> None:
+    env = os.environ.copy()
+    proc = subprocess.run(cmd, env=env, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    if proc.returncode != 0:
+        detail = (proc.stderr or proc.stdout or "MLX Swift bridge failed").strip()
+        logger.error("MLX Swift bridge failed (%s): %s\n%s", proc.returncode, " ".join(cmd), detail[-4000:])
+        raise HTTPException(500, detail[-4000:])
+
+
+def _run_ddcolor_bridge(model: str, image_raw: bytes, out_path: Path) -> None:
+    bridge = _resolve_bridge(["odysseus-mlx-colorize", "mlx-ddcolor-serve"])
+    if not bridge:
+        raise _unsupported_swift_mlx_runtime(model)
+    with tempfile.TemporaryDirectory(prefix="odysseus-ddcolor-") as td:
+        inp = Path(td) / "input.png"
+        _write_bridge_input_image(image_raw, inp)
+        weights = _weights_path(model)
+        tier = "tiny" if "tiny" in model.lower() else "large"
+        _run_bridge([
+            bridge,
+            "--model", str(weights),
+            "--image", str(inp),
+            "--output", str(out_path),
+            "--tier", tier,
+        ])
+
+
+def _run_inpaint_bridge(model: str, image_raw: bytes, mask_raw: bytes | None, out_path: Path) -> None:
+    if not mask_raw:
+        raise HTTPException(
+            422,
+            "LaMa / MI-GAN inpainting requires an image mask. Use the editor inpaint/object-removal tool so Odysseus can send the mask.",
+        )
+    bridge = _resolve_bridge(["odysseus-mlx-inpaint", "mlx-lama-serve"])
+    if not bridge:
+        raise _unsupported_swift_mlx_runtime(model)
+    with tempfile.TemporaryDirectory(prefix="odysseus-mlx-inpaint-") as td:
+        inp = Path(td) / "input.png"
+        mask = Path(td) / "mask.png"
+        _write_bridge_input_image(image_raw, inp)
+        _write_bridge_mask(mask_raw, mask)
+        weights = _weights_path(model)
+        mode = "fast" if ("mi-gan" in model.lower() or "migan" in model.lower()) else "best"
+        _run_bridge([
+            bridge,
+            "--model", str(weights),
+            "--image", str(inp),
+            "--mask", str(mask),
+            "--output", str(out_path),
+            "--mode", mode,
+        ])
+
+
+def _generate_hidream(model: str, prompt: str, out_path: Path, width: int, height: int, steps: int) -> None:
+    model_path = _snapshot_path(model)
+    script = model_path / "scripts" / "hidream_o1" / "generate_hidream_o1_mlx.py"
+    if not script.exists():
+        raise HTTPException(500, f"HiDream generator script not found in snapshot: {script}")
+    cmd = [
+        sys.executable,
+        str(script),
+        "--model-path",
+        str(model_path),
+        "--prompt",
+        prompt,
+        "--output",
+        str(out_path),
+        "--width",
+        str(width),
+        "--height",
+        str(height),
+        "--num-inference-steps",
+        str(steps),
+        "--no-snap-resolution",
+    ]
+    env = os.environ.copy()
+    proc = subprocess.run(cmd, env=env, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    if proc.returncode != 0:
+        detail = (proc.stderr or proc.stdout or "HiDream generator failed").strip()
+        raise HTTPException(500, detail[-4000:])
+
+
+def _generate_boogu(model: str, prompt: str, out_path: Path, width: int, height: int, steps: int) -> None:
+    try:
+        from boogu_image_mlx.pipeline_mlx import BooguImagePipeline
+        from PIL import Image
+    except Exception as e:
+        raise HTTPException(
+            503,
+            "Boogu MLX serving requires boogu-image-mlx in the launch Python. "
+            "Install with: python -m pip install -U git+https://github.com/xocialize/boogu-image-mlx.git",
+        ) from e
+
+    model_path = _snapshot_path(model)
+    vlm_model = (_args.vlm_model or os.environ.get("ODYSSEUS_MLX_IMAGE_VLM_MODEL") or "").strip()
+    if not vlm_model:
+        raise HTTPException(
+            422,
+            "This MLX image pipeline requires a companion vision-language model. "
+            "Relaunch with --vlm-model <repo_or_path> or set ODYSSEUS_MLX_IMAGE_VLM_MODEL.",
+        )
+    try:
+        pipe = BooguImagePipeline.from_pretrained(
+            str(model_path),
+            vlm_model,
+        )
+        img = pipe.generate(
+            prompt,
+            height=height,
+            width=width,
+            steps=steps,
+            guidance=3.5,
+        )
+        Image.fromarray(img).save(out_path)
+    except Exception as e:
+        raise HTTPException(500, f"Boogu MLX generation failed: {e}") from e
+
+
+@app.get("/v1/models")
+def list_models():
+    return {"data": [{"id": _args.model, "object": "model", "owned_by": "local"}]}
+
+
+@app.post("/v1/images/generations")
+def generate(req: ImageRequest):
+    model = req.model or _args.model
+    width, height = _size(req.size)
+    out_images = []
+    count = max(1, min(int(req.n or 1), 4))
+    for _ in range(count):
+        with tempfile.TemporaryDirectory(prefix="odysseus-mlx-image-") as td:
+            out_path = Path(td) / "image.png"
+            if _is_hidream(model):
+                _generate_hidream(model, req.prompt, out_path, width, height, _steps(req.quality))
+            elif _is_boogu(model):
+                _generate_boogu(model, req.prompt, out_path, width, height, _steps(req.quality))
+            elif _is_lama_inpaint(model) or _is_ddcolor(model):
+                raise _unsupported_swift_mlx_runtime(model)
+            else:
+                cli = _cli_for_model(model)
+                cli_path = _resolve_cli(cli)
+                if not cli_path:
+                    raise HTTPException(
+                        503,
+                        f"{cli} not found in PATH or next to {sys.executable}. Install the MLX image runtime with: python3 -m pip install -U mflux",
+                    )
+                cmd = [
+                    cli_path,
+                    "--model",
+                    model,
+                    "--prompt",
+                    req.prompt,
+                    "--steps",
+                    str(_steps(req.quality)),
+                    "--output",
+                    str(out_path),
+                ]
+                if _args.base_model:
+                    cmd += ["--base-model", _args.base_model]
+                if _args.lora_style:
+                    cmd += ["--lora-style", _args.lora_style]
+                if _args.lora_paths:
+                    cmd += ["--lora-paths", *_args.lora_paths]
+                lora_scales = _valid_numbers(_args.lora_scales)
+                if lora_scales:
+                    cmd += ["--lora-scales", *lora_scales]
+                if "qwen" not in model.lower():
+                    cmd += ["--width", str(width), "--height", str(height)]
+                env = os.environ.copy()
+                proc = subprocess.run(cmd, env=env, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                if proc.returncode != 0:
+                    detail = (proc.stderr or proc.stdout or f"{cli} failed").strip()
+                    logger.error("MLX image command failed (%s): %s\n%s", proc.returncode, " ".join(cmd), detail[-4000:])
+                    raise HTTPException(500, detail[-4000:])
+            if not out_path.exists():
+                raise HTTPException(500, f"MLX image generator completed but did not write {out_path}")
+            b64 = base64.b64encode(out_path.read_bytes()).decode("ascii")
+            out_images.append({"b64_json": b64})
+    return {"created": 0, "data": out_images}
+
+
+@app.post("/v1/images/edits")
+async def edit_image(
+    image: UploadFile = File(...),
+    mask: UploadFile | None = File(None),
+    prompt: str = Form(""),
+    model: str = Form(""),
+    n: int = Form(1),
+    size: str = Form("1024x1024"),
+    response_format: str = Form("b64_json"),
+):
+    active_model = model or _args.model
+    if _is_lama_inpaint(active_model) or _is_ddcolor(active_model):
+        image_raw = await image.read()
+        mask_raw = await mask.read() if mask is not None else None
+        out_images = []
+        count = max(1, min(int(n or 1), 4))
+        for _ in range(count):
+            with tempfile.TemporaryDirectory(prefix="odysseus-mlx-edit-") as td:
+                out_path = Path(td) / "image.png"
+                if _is_ddcolor(active_model):
+                    _run_ddcolor_bridge(active_model, image_raw, out_path)
+                else:
+                    _run_inpaint_bridge(active_model, image_raw, mask_raw, out_path)
+                if not out_path.exists():
+                    raise HTTPException(500, f"MLX Swift bridge completed but did not write {out_path}")
+                out_images.append({"b64_json": base64.b64encode(out_path.read_bytes()).decode("ascii")})
+        return {"created": 0, "data": out_images}
+    raise HTTPException(
+        422,
+        "This MLX image endpoint supports text-to-image generation only. "
+        "Use /v1/images/generations, or serve an edit/img2img-capable model.",
+    )
+
+
+@app.post("/v1/images/harmonize")
+def harmonize_image(req: HarmonizeRequest):
+    active_model = _args.model
+    if _is_lama_inpaint(active_model) or _is_ddcolor(active_model):
+        try:
+            image_raw = base64.b64decode(req.image.split(",", 1)[-1])
+            mask_b64 = req.body_mask or req.mask
+            mask_raw = base64.b64decode(mask_b64.split(",", 1)[-1]) if mask_b64 else None
+        except Exception as e:
+            raise HTTPException(400, f"Invalid base64 image payload: {e}") from e
+        with tempfile.TemporaryDirectory(prefix="odysseus-mlx-harmonize-") as td:
+            out_path = Path(td) / "image.png"
+            if _is_ddcolor(active_model):
+                _run_ddcolor_bridge(active_model, image_raw, out_path)
+            else:
+                _run_inpaint_bridge(active_model, image_raw, mask_raw, out_path)
+            if not out_path.exists():
+                raise HTTPException(500, f"MLX Swift bridge completed but did not write {out_path}")
+            return {"image": base64.b64encode(out_path.read_bytes()).decode("ascii")}
+    raise HTTPException(
+        422,
+        "This MLX image endpoint supports text-to-image generation only. "
+        "Use /v1/images/generations, or serve an edit/img2img-capable model.",
+    )
+
+
+def main() -> None:
+    global _args
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", required=True)
+    parser.add_argument("--host", default="127.0.0.1")
+    parser.add_argument("--port", type=int, default=8100)
+    parser.add_argument("--steps", type=int, default=0)
+    parser.add_argument("--width", type=int, default=1024)
+    parser.add_argument("--height", type=int, default=1024)
+    parser.add_argument("--base-model", default="")
+    parser.add_argument("--lora-style", default="")
+    parser.add_argument("--lora-paths", nargs="*", default=[])
+    parser.add_argument("--lora-scales", nargs="*", default=[])
+    parser.add_argument("--vlm-model", default="")
+    _args = parser.parse_args()
+    uvicorn.run(app, host=_args.host, port=_args.port)
+
+
+if __name__ == "__main__":
+    main()

scripts/odysseus-backup

@@ -133,18 +133,32 @@ def cmd_list(args):
         emit([], args)
         return
     entries = []
-    for p in sorted(_BACKUP_DIR.iterdir(), key=lambda x: x.stat().st_mtime, reverse=True):
-        if not p.is_file():
-            continue
-        entries.append({
-            "path": str(p),
-            "name": p.name,
-            "bytes": p.stat().st_size,
-            "modified": datetime.fromtimestamp(p.stat().st_mtime).isoformat(),
-        })
+    for p in _BACKUP_DIR.iterdir():
+        entry = _backup_entry(p)
+        if entry is not None:
+            entries.append(entry)
+    entries.sort(key=lambda entry: entry["_mtime"], reverse=True)
+    for entry in entries:
+        entry.pop("_mtime", None)
     emit(entries, args)
 
 
+def _backup_entry(p):
+    try:
+        if not p.is_file():
+            return None
+        st = p.stat()
+    except OSError:
+        return None
+    return {
+        "path": str(p),
+        "name": p.name,
+        "bytes": st.st_size,
+        "modified": datetime.fromtimestamp(st.st_mtime).isoformat(),
+        "_mtime": st.st_mtime,
+    }
+
+
 def cmd_verify(args):
     """Open the tarball read-only and walk its members — confirms
     integrity without extracting anything."""

scripts/odysseus-memory

@@ -90,7 +90,7 @@ def cmd_add(args):
     # add_entry doesn't save by default — the call in chat does it
     # after dedup checks. Persist here so a one-shot CLI add sticks.
     all_entries = _manager().load_all()
-    if not any(e.get("id") == entry.get("id") for e in all_entries):
+    if not any(isinstance(e, dict) and e.get("id") == entry.get("id") for e in all_entries):
         all_entries.append(entry)
         _manager().save(all_entries)
     emit(entry, args)

services/hwfit/image_models.py

@@ -1,278 +1,328 @@
 """Image generation model registry and VRAM fitting for Cookbook."""
 
-# Curated registry of image generation models supported by diffusers.
-# ONLY verified HuggingFace repo IDs.
-# VRAM estimates are for inference (single image generation).
-IMAGE_MODEL_REGISTRY = [
-    # ── Z-Image (Alibaba Tongyi) ──
-    {
-        "id": "Tongyi-MAI/Z-Image-Turbo",
-        "name": "Z-Image Turbo",
-        "provider": "Tongyi",
-        "params_b": 6.0,
-        "vram_bf16": 19.0,
-        "vram_fp8": 10.0,
-        "vram_q4": 6.0,
-        "default_quant": "BF16",
-        "quant_repos": {
-            "FP8": "drbaph/Z-Image-Turbo-FP8",
-        },
-        "capabilities": ["text-to-image"],
-        "description": "6B distilled, 8-step. Sub-second on H800. Apache 2.0.",
-        "quality": 92,
-        "speed": 95,
-        "released": "2025-12",
-    },
-    {
-        "id": "Tongyi-MAI/Z-Image",
-        "name": "Z-Image",
-        "provider": "Tongyi",
-        "params_b": 6.0,
-        "vram_bf16": 19.0,
-        "vram_fp8": 10.0,
-        "vram_q4": 6.0,
-        "default_quant": "BF16",
-        "quant_repos": {
-            "FP8": "drbaph/Z-Image-fp8",
-        },
-        "capabilities": ["text-to-image"],
-        "description": "Full undistilled model. Highest creative freedom. Apache 2.0.",
-        "quality": 93,
-        "speed": 70,
-        "released": "2025-12",
-    },
-    # ── Qwen Image ──
-    {
-        "id": "Qwen/Qwen-Image-2512",
-        "name": "Qwen Image 2512",
-        "provider": "Qwen",
-        "params_b": 20.0,
-        "vram_bf16": 42.0,
-        "vram_fp8": 22.0,
-        "vram_q4": 14.0,
-        "default_quant": "FP8",
-        "quant_repos": {},
-        "capabilities": ["text-to-image", "text-rendering"],
-        "description": "Dec 2025 update. Better humans, finer detail, strong text. Apache 2.0.",
-        "quality": 95,
-        "speed": 50,
-        "released": "2025-12",
-    },
-    {
-        "id": "Qwen/Qwen-Image",
-        "name": "Qwen Image",
-        "provider": "Qwen",
-        "params_b": 20.0,
-        "vram_bf16": 42.0,
-        "vram_fp8": 22.0,
-        "vram_q4": 14.0,
-        "default_quant": "FP8",
-        "quant_repos": {},
-        "capabilities": ["text-to-image", "text-rendering"],
-        "description": "20B foundation. Best text rendering in images. Apache 2.0.",
-        "quality": 94,
-        "speed": 50,
-        "released": "2025-08",
-    },
-    {
-        "id": "Qwen/Qwen-Image-Edit-2511",
-        "name": "Qwen Image Edit",
-        "provider": "Qwen",
-        "params_b": 20.0,
-        "vram_bf16": 42.0,
-        "vram_fp8": 22.0,
-        "vram_q4": 14.0,
-        "default_quant": "FP8",
-        "quant_repos": {},
-        "capabilities": ["image-editing", "inpainting"],
-        "description": "Dedicated editing. Style transfer, object removal. Apache 2.0.",
-        "quality": 92,
-        "speed": 50,
-        "released": "2025-11",
-    },
-    # ── Stable Diffusion (dedicated inpainting) ──
-    {
-        "id": "diffusers/stable-diffusion-xl-1.0-inpainting-0.1",
-        "name": "SDXL Inpainting",
-        "provider": "Stability AI",
-        "params_b": 3.5,
-        "vram_bf16": 12.0,
-        "vram_fp8": 8.0,
-        "vram_q4": 6.0,
-        "default_quant": "BF16",
-        "quant_repos": {},
-        "capabilities": ["inpainting", "image-editing"],
-        "description": "SDXL fine-tuned for inpainting (9-channel UNet). Best SD-family fill quality; fits a 24GB card comfortably.",
-        "quality": 86,
-        "speed": 68,
-        "released": "2023-11",
-    },
-    {
-        "id": "stable-diffusion-v1-5/stable-diffusion-inpainting",
-        "name": "SD 1.5 Inpainting",
-        "provider": "Stability AI",
-        "params_b": 1.1,
-        "vram_bf16": 4.0,
-        "vram_fp8": 3.0,
-        "vram_q4": 2.5,
-        "default_quant": "BF16",
-        "quant_repos": {},
-        "capabilities": ["inpainting"],
-        "description": "Classic SD 1.5 inpaint. Very light and fast; lower fidelity than SDXL.",
-        "quality": 70,
-        "speed": 92,
-        "released": "2022-10",
-    },
-    # ── FLUX ──
-    {
-        "id": "black-forest-labs/FLUX.1-dev",
-        "name": "FLUX.1 Dev",
-        "provider": "Black Forest Labs",
-        "params_b": 12.0,
-        "vram_bf16": 33.0,
-        "vram_fp8": 17.0,
-        "vram_q4": 10.0,
-        "default_quant": "FP8",
-        "quant_repos": {
-            "FP8": "diffusers/FLUX.1-dev-torchao-fp8",
-        },
-        "capabilities": ["text-to-image"],
-        "description": "High quality, detailed. Popular community model. Non-commercial.",
-        "quality": 92,
-        "speed": 55,
-        "released": "2024-08",
-    },
-    {
-        "id": "black-forest-labs/FLUX.1-schnell",
-        "name": "FLUX.1 Schnell",
-        "provider": "Black Forest Labs",
-        "params_b": 12.0,
-        "vram_bf16": 33.0,
-        "vram_fp8": 17.0,
-        "vram_q4": 10.0,
-        "default_quant": "FP8",
-        "quant_repos": {
-            "FP8": "Kijai/flux-fp8",
-        },
-        "capabilities": ["text-to-image"],
-        "description": "Fast 4-step variant. Apache 2.0 license.",
-        "quality": 85,
-        "speed": 90,
-        "released": "2024-08",
-    },
-    # ── Stable Diffusion ──
-    {
-        "id": "stabilityai/stable-diffusion-3.5-medium",
-        "name": "SD 3.5 Medium",
-        "provider": "Stability AI",
-        "params_b": 2.5,
-        "vram_bf16": 12.0,
-        "vram_fp8": 7.0,
-        "vram_q4": None,
-        "default_quant": "BF16",
-        "quant_repos": {
-            "FP8": "Comfy-Org/stable-diffusion-3.5-fp8",
-        },
-        "capabilities": ["text-to-image"],
-        "description": "2.5B lightweight, fast. Fits almost any GPU.",
-        "quality": 75,
-        "speed": 95,
-        "released": "2024-10",
-    },
-    {
-        "id": "stabilityai/stable-diffusion-3.5-large",
-        "name": "SD 3.5 Large",
-        "provider": "Stability AI",
-        "params_b": 8.1,
-        "vram_bf16": 22.0,
-        "vram_fp8": 12.0,
-        "vram_q4": None,
-        "default_quant": "BF16",
-        "quant_repos": {
-            "FP8": "Comfy-Org/stable-diffusion-3.5-fp8",
-        },
-        "capabilities": ["text-to-image"],
-        "description": "8B high quality. Good balance of speed and quality.",
-        "quality": 85,
-        "speed": 70,
-        "released": "2024-10",
-    },
-    {
-        "id": "stabilityai/stable-diffusion-3.5-large-turbo",
-        "name": "SD 3.5 Large Turbo",
-        "provider": "Stability AI",
-        "params_b": 8.1,
-        "vram_bf16": 22.0,
-        "vram_fp8": 12.0,
-        "vram_q4": None,
-        "default_quant": "BF16",
-        "quant_repos": {
-            "FP8": "Comfy-Org/stable-diffusion-3.5-fp8",
-        },
-        "capabilities": ["text-to-image"],
-        "description": "Distilled for few-step inference. Fastest large SD.",
-        "quality": 80,
-        "speed": 92,
-        "released": "2024-10",
-    },
-    {
-        "id": "stabilityai/stable-diffusion-xl-base-1.0",
-        "name": "SDXL",
-        "provider": "Stability AI",
-        "params_b": 3.5,
-        "vram_bf16": 10.0,
-        "vram_fp8": 6.0,
-        "vram_q4": None,
-        "default_quant": "BF16",
-        "quant_repos": {},
-        "capabilities": ["text-to-image"],
-        "description": "Classic workhorse. Huge LoRA ecosystem. Fits 8GB+.",
-        "quality": 72,
-        "speed": 90,
-        "released": "2023-07",
-    },
-    # ── Hunyuan ──
-    {
-        "id": "tencent/HunyuanImage-3.0",
-        "name": "HunyuanImage 3.0",
-        "provider": "Tencent",
-        "params_b": 13.0,
-        "vram_bf16": 30.0,
-        "vram_fp8": 16.0,
-        "vram_q4": 9.0,
-        "default_quant": "FP8",
-        "quant_repos": {
-            "Q4": "wikeeyang/Hunyuan-Image-30-Qint4",
-            "NF4": "EricRollei/HunyuanImage-3.0-Instruct-NF4",
-        },
-        "capabilities": ["text-to-image", "text-rendering"],
-        "description": "Strong text rendering. Bilingual Chinese/English. 13B activated per token.",
-        "quality": 88,
-        "speed": 60,
-        "released": "2025-09",
-    },
-    {
-        "id": "tencent/HunyuanImage-3.0-Instruct-Distil",
-        "name": "HunyuanImage 3.0 Distil",
-        "provider": "Tencent",
-        "params_b": 13.0,
-        "vram_bf16": 30.0,
-        "vram_fp8": 16.0,
-        "vram_q4": 9.0,
-        "default_quant": "FP8",
-        "quant_repos": {},
-        "capabilities": ["text-to-image", "text-rendering"],
-        "description": "Distilled variant, fewer steps. Faster with comparable quality.",
-        "quality": 85,
-        "speed": 80,
-        "released": "2026-01",
-    },
+from __future__ import annotations
+
+import json
+import re
+import time
+import urllib.parse
+import urllib.request
+from typing import Any
+
+# Image models are discovered from HuggingFace collections/search and local cache.
+# Keep this empty: source-coded repo IDs become hidden recommendations.
+IMAGE_MODEL_REGISTRY: list[dict[str, Any]] = []
+
+HF_IMAGE_COLLECTIONS = [
+    "stabilityai/image",
+    "stabilityai/stable-diffusion-35",
+    "black-forest-labs/flux2",
 ]
 
+HF_MLX_IMAGE_COLLECTIONS = [
+    "mlx-community/flux2-klein-mlx",
+    "mlx-community/inpainting-mlx",
+    "mlx-community/ddcolor-mlx",
+    "mlx-community/boogu-image-01-mlx",
+]
+
+HF_MLX_IMAGE_REPO_SEEDS: list[str] = []
+HF_IMAGE_REPO_SEEDS: list[str] = []
+
+_HF_COLLECTION_CACHE = {"ts": 0.0, "models": []}
+_HF_COLLECTION_TTL = 30 * 60
+_HF_VARIANT_CACHE: dict[str, dict[str, str]] = {}
+_HF_SEARCH_DISABLED_UNTIL = 0.0
+
+
+def _repo_display_name(repo_id: str) -> str:
+    name = str(repo_id or "").split("/")[-1]
+    return name.replace("-", " ").replace("_", " ").strip() or repo_id
+
+
+def _provider_from_repo(repo_id: str) -> str:
+    owner = str(repo_id or "").split("/", 1)[0].lower()
+    return {
+        "stabilityai": "Stability AI",
+        "black-forest-labs": "Black Forest Labs",
+        "tongyi-mai": "Tongyi",
+        "qwen": "Qwen",
+        "mlx-community": "mlx-community",
+    }.get(owner, owner.replace("-", " ").title() if owner else "HuggingFace")
+
+
+def _infer_capabilities(item: dict[str, Any], repo_id: str) -> list[str]:
+    tasks = set()
+    pipeline = str(item.get("pipeline_tag") or "").strip().lower()
+    if pipeline:
+        tasks.add(pipeline)
+    for provider in item.get("availableInferenceProviders") or []:
+        if isinstance(provider, dict) and provider.get("task"):
+            tasks.add(str(provider["task"]).strip().lower())
+    text = f"{repo_id} {' '.join(tasks)}".lower()
+    caps = []
+    if "image-to-image" in tasks or "edit" in text or "inpaint" in text:
+        caps.append("image-editing")
+    if "inpaint" in text:
+        caps.append("inpainting")
+    if "text-to-image" in tasks or not caps:
+        caps.append("text-to-image")
+    return caps
+
+
+def _estimate_image_model(repo_id: str) -> dict[str, Any]:
+    text = str(repo_id or "").lower()
+    params_b = 8.0
+    param_match = re.search(r"(?<![\d.])(\d+(?:\.\d+)?)\s*b(?:\b|[-_])", text)
+    if param_match:
+        params_b = max(0.01, float(param_match.group(1)))
+    if any(k in text for k in ("mi-gan", "big-lama", "lama-")):
+        return {"params_b": 0.01, "bf16": 1.0, "fp8": 0.7, "q4": 0.5, "quality": 65, "speed": 98, "quant": "BF16"}
+    quant = "BF16"
+    if any(k in text for k in ("4bit", "q4", "nf4")):
+        quant = "Q4"
+    elif "fp8" in text or "8bit" in text:
+        quant = "FP8"
+    bf16 = max(1.0, round(params_b * 2.6 + 3.0, 1))
+    fp8 = max(0.7, round(params_b * 1.35 + 2.0, 1))
+    q4 = max(0.5, round(params_b * 0.8 + 1.5, 1))
+    speed = max(35, min(95, int(98 - params_b * 3)))
+    quality = max(60, min(88, int(70 + min(params_b, 18) * 0.8)))
+    return {"params_b": params_b, "bf16": bf16, "fp8": fp8, "q4": q4, "quality": quality, "speed": speed, "quant": quant}
+
+
+def _params_b_from_item(item: dict[str, Any]) -> float | None:
+    raw = item.get("numParameters")
+    if isinstance(raw, (int, float)) and raw > 0:
+        return max(0.01, round(float(raw) / 1_000_000_000.0, 3))
+    return None
+
+
+def _mlx_quantize_estimate(repo_id: str, est: dict[str, Any]) -> dict[str, Any]:
+    text = str(repo_id or "").lower()
+    out = dict(est)
+    if "3bit" in text or "4bit" in text or "q4" in text:
+        out["quant"] = "Q4"
+        out["bf16"] = None
+        out["fp8"] = None
+    elif "8bit" in text:
+        out["quant"] = "FP8"
+        out["bf16"] = None
+    elif "6bit" in text or "5bit" in text:
+        out["quant"] = "Q4"
+        out["bf16"] = None
+        out["fp8"] = out.get("fp8") or out.get("q4")
+    elif "bf16" in text or "fp16" in text:
+        out["quant"] = "BF16"
+        out["fp8"] = None
+        out["q4"] = None
+    return out
+
+
+def _collection_item_to_model(item: dict[str, Any], collection_title: str = "", mlx_only: bool = False) -> dict[str, Any] | None:
+    repo_id = str(item.get("id") or "").strip()
+    if "/" not in repo_id:
+        return None
+    typ = str(item.get("type") or item.get("itemType") or "model").lower()
+    if typ not in {"", "model"}:
+        return None
+    est = _estimate_image_model(repo_id)
+    item_params_b = _params_b_from_item(item)
+    if item_params_b is not None:
+        est = {
+            **est,
+            "params_b": item_params_b,
+            "bf16": max(0.5, round(item_params_b * 2.4 + 0.8, 1)),
+            "fp8": max(0.5, round(item_params_b * 1.3 + 0.5, 1)),
+            "q4": max(0.4, round(item_params_b * 0.8 + 0.4, 1)),
+        }
+    if mlx_only:
+        est = _mlx_quantize_estimate(repo_id, est)
+    caps = _infer_capabilities(item, repo_id)
+    gated = item.get("gated")
+    desc_bits = []
+    if collection_title:
+        desc_bits.append(f"HF collection: {collection_title}.")
+    if gated:
+        desc_bits.append("Gated on HuggingFace.")
+    out = {
+        "id": repo_id,
+        "name": _repo_display_name(repo_id),
+        "provider": _provider_from_repo(repo_id),
+        "params_b": est["params_b"],
+        "vram_bf16": est["bf16"],
+        "vram_fp8": est["fp8"],
+        "vram_q4": est["q4"],
+        "default_quant": est["quant"],
+        "quant_repos": {},
+        "capabilities": caps,
+        "description": " ".join(desc_bits).strip() or "Imported from HuggingFace collection.",
+        "quality": est["quality"],
+        "speed": est["speed"],
+        "released": "",
+    }
+    if mlx_only:
+        out["mlx_only"] = True
+        out["description"] = (out["description"] + " Apple Silicon / MLX only.").strip()
+    return out
+
+
+def _fetch_hf_image_collection_models() -> list[dict[str, Any]]:
+    now = time.time()
+    if now - float(_HF_COLLECTION_CACHE.get("ts") or 0) < _HF_COLLECTION_TTL:
+        return list(_HF_COLLECTION_CACHE.get("models") or [])
+    models: list[dict[str, Any]] = []
+    for slug, mlx_only in [(slug, False) for slug in HF_IMAGE_COLLECTIONS] + [(slug, True) for slug in HF_MLX_IMAGE_COLLECTIONS]:
+        url = f"https://huggingface.co/api/collections/{slug}"
+        try:
+            req = urllib.request.Request(url, headers={"User-Agent": "Odysseus-Cookbook/1.0"})
+            with urllib.request.urlopen(req, timeout=2.5) as resp:
+                data = json.loads(resp.read().decode("utf-8", "replace"))
+        except Exception:
+            continue
+        title = str(data.get("title") or slug)
+        for item in data.get("items") or []:
+            if isinstance(item, dict):
+                model = _collection_item_to_model(item, title, mlx_only=mlx_only)
+                if model:
+                    models.append(model)
+    _HF_COLLECTION_CACHE["ts"] = now
+    _HF_COLLECTION_CACHE["models"] = models
+    return list(models)
+
+
+def _hf_model_search(query: str, limit: int = 10) -> list[dict[str, Any]]:
+    global _HF_SEARCH_DISABLED_UNTIL
+    now = time.time()
+    if now < _HF_SEARCH_DISABLED_UNTIL:
+        return []
+    url = "https://huggingface.co/api/models?" + urllib.parse.urlencode({
+        "search": query,
+        "limit": str(limit),
+    })
+    try:
+        req = urllib.request.Request(url, headers={"User-Agent": "Odysseus-Cookbook/1.0"})
+        with urllib.request.urlopen(req, timeout=2.5) as resp:
+            data = json.loads(resp.read().decode("utf-8", "replace"))
+        return data if isinstance(data, list) else []
+    except Exception:
+        _HF_SEARCH_DISABLED_UNTIL = now + 10 * 60
+        return []
+
+
+def _variant_score(candidate: dict[str, Any], base_repo: str, want: str) -> float:
+    rid = str(candidate.get("id") or candidate.get("modelId") or "")
+    text = " ".join([
+        rid,
+        str(candidate.get("library_name") or ""),
+        str(candidate.get("pipeline_tag") or ""),
+        " ".join(str(t) for t in candidate.get("tags") or []),
+    ]).lower()
+    base = base_repo.lower()
+    base_short = base_repo.rsplit("/", 1)[-1].lower()
+    if want == "gguf" and "gguf" not in text:
+        return -1
+    if want == "fp8" and not any(k in text for k in ("fp8", "nvfp4", "mxfp8", "mxfp4")):
+        return -1
+    score = float(candidate.get("downloads") or 0) / 1000.0 + float(candidate.get("likes") or 0)
+    if f"base_model:{base}" in text or f"base_model:quantized:{base}" in text:
+        score += 10000
+    elif base_short and base_short in rid.lower():
+        score += 1000
+    else:
+        score -= 200
+    if "diffusers" in text:
+        score += 50
+    if str(candidate.get("private")).lower() == "true":
+        score -= 10000
+    return score
+
+
+def _best_variant_repo(base_repo: str, want: str) -> str:
+    base_short = str(base_repo or "").rsplit("/", 1)[-1]
+    candidates = _hf_model_search(f"{base_short} {want}", limit=12)
+    scored = []
+    for item in candidates:
+        if not isinstance(item, dict):
+            continue
+        rid = str(item.get("id") or item.get("modelId") or "").strip()
+        if "/" not in rid or rid.lower() == base_repo.lower():
+            continue
+        score = _variant_score(item, base_repo, want)
+        if score >= 0:
+            scored.append((score, rid))
+    scored.sort(reverse=True)
+    return scored[0][1] if scored else ""
+
+
+def _should_discover_variants(repo_id: str) -> bool:
+    return False
+
+
+def _discover_quant_repos(repo_id: str, need_fp8: bool = True, need_gguf: bool = True) -> dict[str, str]:
+    key = str(repo_id or "").strip()
+    if not key:
+        return {}
+    cache_key = f"{key.lower()}|fp8={int(need_fp8)}|gguf={int(need_gguf)}"
+    if cache_key in _HF_VARIANT_CACHE:
+        return dict(_HF_VARIANT_CACHE[cache_key])
+    found: dict[str, str] = {}
+    if need_fp8:
+        fp8 = _best_variant_repo(key, "fp8")
+        if fp8:
+            found["FP8"] = fp8
+    if need_gguf:
+        gguf = _best_variant_repo(key, "gguf")
+        if gguf:
+            # The image-model fitter's smallest bucket is Q4; most HF image GGUF
+            # repos expose Q4/Q5/Q8 files under one repo, so use it as the low-VRAM
+            # download source while preserving the explicit GGUF label for callers.
+            found["Q4"] = gguf
+            found["GGUF"] = gguf
+    _HF_VARIANT_CACHE[cache_key] = found
+    return dict(found)
+
+
+def _merge_quant_repos(model: dict[str, Any]) -> dict[str, Any]:
+    out = dict(model)
+    existing = dict(out.get("quant_repos") or {})
+    repo_id = str(out.get("id") or "")
+    if _should_discover_variants(repo_id):
+        discovered = _discover_quant_repos(
+            repo_id,
+            need_fp8="FP8" not in existing,
+            need_gguf="Q4" not in existing and "GGUF" not in existing,
+        )
+        for k, v in discovered.items():
+            existing.setdefault(k, v)
+    out["quant_repos"] = existing
+    return out
+
 
 def get_image_models():
     """Return the image model registry."""
-    return IMAGE_MODEL_REGISTRY
+    merged = [_merge_quant_repos(m) for m in IMAGE_MODEL_REGISTRY]
+    seen = {str(m.get("id") or "").lower() for m in merged if isinstance(m, dict)}
+    for model in _fetch_hf_image_collection_models():
+        key = str(model.get("id") or "").lower()
+        if key and key not in seen:
+            merged.append(_merge_quant_repos(model))
+            seen.add(key)
+    return merged
+
+
+def _is_apple_image_system(system: dict[str, Any]) -> bool:
+    backend = str(system.get("backend") or "").lower()
+    gpu_name = str(system.get("gpu_name") or "").lower()
+    cpu_name = str(system.get("cpu_name") or "").lower()
+    platform = str(system.get("platform") or "").lower()
+    return (
+        bool(system.get("unified_memory"))
+        or backend in {"metal", "mps", "apple"}
+        or "apple" in gpu_name
+        or "apple" in cpu_name
+        or platform == "darwin"
+    )
 
 
 def rank_image_models(system, search=None, sort="fit"):
@@ -284,9 +334,17 @@ def rank_image_models(system, search=None, sort="fit"):
         system = {}
     gpu_vram = system.get("gpu_vram_gb", 0) or 0
     has_gpu = system.get("has_gpu", False)
+    ram_gb = system.get("available_ram_gb") or system.get("total_ram_gb") or 0
+    budget_gb = gpu_vram if has_gpu and gpu_vram > 0 else ram_gb
+    budget_kind = "gpu" if has_gpu and gpu_vram > 0 else "ram"
+    apple_system = _is_apple_image_system(system)
     results = []
 
-    for model in IMAGE_MODEL_REGISTRY:
+    for model in get_image_models():
+        if apple_system and not (model.get("mlx_only") or model.get("apple_ok")):
+            continue
+        if model.get("mlx_only") and not apple_system:
+            continue
         # Filter by search
         if isinstance(search, str) and search:
             s = search.lower()
@@ -299,11 +357,11 @@ def rank_image_models(system, search=None, sort="fit"):
         fits = False
         quant_repo = None
 
-        if has_gpu and gpu_vram > 0:
+        if budget_gb > 0:
             # Try BF16 first, then FP8, then Q4
             for q, vram_key in [("BF16", "vram_bf16"), ("FP8", "vram_fp8"), ("Q4", "vram_q4")]:
                 v = model.get(vram_key)
-                if v is not None and v <= gpu_vram * 0.90:  # 10% headroom
+                if v is not None and v <= budget_gb * 0.90:  # 10% headroom
                     quant = q
                     vram_needed = v
                     fits = True
@@ -315,15 +373,15 @@ def rank_image_models(system, search=None, sort="fit"):
                 vram_needed = model.get("vram_bf16", 0)
 
         # Fit label
-        if not has_gpu:
+        if budget_gb <= 0:
             fit = "no_gpu"
             fit_label = "No GPU"
         elif fits:
-            headroom = gpu_vram - vram_needed
-            if headroom > gpu_vram * 0.3:
+            headroom = budget_gb - vram_needed
+            if headroom > budget_gb * 0.3:
                 fit = "perfect"
                 fit_label = "Perfect"
-            elif headroom > gpu_vram * 0.1:
+            elif headroom > budget_gb * 0.1:
                 fit = "good"
                 fit_label = "Good"
             else:
@@ -355,6 +413,7 @@ def rank_image_models(system, search=None, sort="fit"):
             "fits": fits,
             "fit": fit,
             "fit_label": fit_label,
+            "fit_budget": budget_kind,
             "quality": model["quality"],
             "speed": model["speed"],
             "score": round(score, 1),

services/hwfit/models.py

@@ -145,7 +145,7 @@ def is_prequantized(model):
         or re.search(r"(^|[-_/])fp8($|[-_/\s])", text) is not None
         or (not (model.get("is_gguf") or model.get("gguf_sources")) and re.search(r"(^|[-_/])(?:int)?8bit($|[-_/\s])", text) is not None)
         or any(x in text for x in ("awq", "gptq", "mlx"))
-        or any(q.startswith(p) for p in PREQUANTIZED_PREFIXES)
+        or any(isinstance(q, str) and q.startswith(p) for p in PREQUANTIZED_PREFIXES)
     )
 
 
@@ -155,7 +155,7 @@ def params_b(model):
         return raw / 1_000_000_000.0
 
     pc = model.get("parameter_count", "")
-    if pc:
+    if isinstance(pc, str) and pc:
         pc = pc.strip().upper()
         m = re.match(r"^([\d.]+)\s*([BKMGT]?)$", pc)
         if m:

services/memory/skill_importer.py

@@ -6,7 +6,7 @@ import os
 import re
 from dataclasses import dataclass
 from typing import Dict, List, Optional, Tuple
-from urllib.parse import quote, urlparse
+from urllib.parse import quote, urljoin, urlparse
 
 import httpx
 
@@ -68,6 +68,52 @@ def _is_text_file(name: str) -> bool:
     return any(low.endswith(s) for s in ALLOWED_SUFFIXES)
 
 
+# Max redirect hops to follow manually while re-validating each one.
+_MAX_FETCH_REDIRECTS = 5
+
+
+def _check_fetch_url(url: str) -> None:
+    """SSRF guard for skill-import fetches (defense-in-depth).
+
+    Skill bundles only ever come from public GitHub, never an internal
+    address, so block private/loopback/link-local targets on every hop —
+    matching the hardened web-fetch path in
+    ``services/search/content.py:_get_public_url`` rather than the lenient
+    default used for admin-configured model endpoints.
+    """
+    ok, reason = check_outbound_url(url, block_private=True)
+    if not ok:
+        raise SkillImportError(reason)
+
+
+def _get_checked(
+    url: str,
+    *,
+    headers: Optional[dict] = None,
+    timeout: float = 30.0,
+) -> httpx.Response:
+    """GET that follows redirects manually, re-running the SSRF guard per hop.
+
+    ``httpx``'s ``follow_redirects=True`` validates only the initial URL, so a
+    ``3xx`` to an internal address (``169.254.169.254``, ``127.0.0.1``, …) would
+    still be connected to before any post-hoc host check. Following redirects by
+    hand lets us re-validate every hop, closing that blind-SSRF gap.
+    """
+    current = url
+    with httpx.Client(follow_redirects=False, timeout=timeout) as client:
+        for _ in range(_MAX_FETCH_REDIRECTS + 1):
+            _check_fetch_url(current)
+            r = client.get(current, headers=headers)
+            if r.status_code in (301, 302, 303, 307, 308):
+                location = r.headers.get("location")
+                if not location:
+                    return r
+                current = urljoin(str(r.url), location)
+                continue
+            return r
+    raise SkillImportError("too many redirects while fetching skill bundle")
+
+
 def parse_skill_source(url: str) -> ResolvedSource:
     """Normalize skills.sh / GitHub web URLs into owner/repo/ref/path."""
     raw = (url or "").strip()
@@ -76,22 +122,18 @@ def parse_skill_source(url: str) -> ResolvedSource:
 
     # skills.sh often links to GitHub; try to unwrap ?url= or redirect target later.
     if "skills.sh" in raw and "github.com" not in raw:
-        ok, reason = check_outbound_url(raw)
-        if not ok:
-            raise SkillImportError(reason)
-        with httpx.Client(follow_redirects=True, timeout=20.0) as client:
-            r = client.get(raw)
-            if r.status_code >= 400:
-                raise _github_response_error(r)
-            final = str(r.url)
-            _assert_github_url(final, context="redirect target")
-            # Page may embed a github link; prefer final URL if redirected.
-            if "github.com" in final:
-                raw = final
-            else:
-                m = re.search(r"https?://github\.com/[^\s\"')]+", r.text or "")
-                if m:
-                    raw = m.group(0).rstrip(".,)")
+        r = _get_checked(raw, timeout=20.0)
+        if r.status_code >= 400:
+            raise _github_response_error(r)
+        final = str(r.url)
+        _assert_github_url(final, context="redirect target")
+        # Page may embed a github link; prefer final URL if redirected.
+        if "github.com" in final:
+            raw = final
+        else:
+            m = re.search(r"https?://github\.com/[^\s\"')]+", r.text or "")
+            if m:
+                raw = m.group(0).rstrip(".,)")
 
     parsed = urlparse(raw)
     host = _github_host(raw)
@@ -165,17 +207,13 @@ def _github_response_error(response: httpx.Response) -> SkillImportError:
 
 
 def _fetch_bytes(url: str) -> bytes:
-    ok, reason = check_outbound_url(url)
-    if not ok:
-        raise SkillImportError(reason)
-    with httpx.Client(follow_redirects=True, timeout=30.0) as client:
-        r = client.get(url, headers={"Accept": "application/vnd.github+json"})
-        if r.status_code >= 400:
-            raise _github_response_error(r)
-        _assert_github_url(str(r.url), context="redirect target")
-        if len(r.content) > MAX_FILE_BYTES:
-            raise SkillImportError(f"file too large: {url}")
-        return r.content
+    r = _get_checked(url, headers={"Accept": "application/vnd.github+json"}, timeout=30.0)
+    if r.status_code >= 400:
+        raise _github_response_error(r)
+    _assert_github_url(str(r.url), context="redirect target")
+    if len(r.content) > MAX_FILE_BYTES:
+        raise SkillImportError(f"file too large: {url}")
+    return r.content
 
 
 def _fetch_text(url: str) -> str:
@@ -190,15 +228,11 @@ def _list_github_dir(src: ResolvedSource, rel_dir: str, out: Dict[str, str], *,
     if depth > 4 or len(out) >= MAX_FILES:
         return
     url = _api_contents_url(src, rel_dir)
-    ok, reason = check_outbound_url(url)
-    if not ok:
-        raise SkillImportError(reason)
-    with httpx.Client(follow_redirects=True, timeout=30.0) as client:
-        r = client.get(url, headers={"Accept": "application/vnd.github+json"})
-        if r.status_code >= 400:
-            raise _github_response_error(r)
-        _assert_github_url(str(r.url), context="redirect target")
-        entries = r.json()
+    r = _get_checked(url, headers={"Accept": "application/vnd.github+json"}, timeout=30.0)
+    if r.status_code >= 400:
+        raise _github_response_error(r)
+    _assert_github_url(str(r.url), context="redirect target")
+    entries = r.json()
     if not isinstance(entries, list):
         raise SkillImportError("expected a directory on GitHub")
     total = sum(len(v.encode("utf-8")) for v in out.values())

services/research/research_handler.py

@@ -186,6 +186,8 @@ class ResearchHandler:
         seen = set()
         sources = []
         for f in findings:
+            if not isinstance(f, dict):
+                continue
             url = f.get("url", "")
             title = f.get("title", "") or url
             summary = f.get("summary", "") or f.get("evidence", "")

services/tts/tts_service.py

@@ -68,7 +68,7 @@ class TTSService:
         if provider == "local":
             kokoro = self._get_kokoro()
             return kokoro is not None and kokoro.available
-        if provider.startswith("endpoint:"):
+        if isinstance(provider, str) and provider.startswith("endpoint:"):
             return True  # assume reachable; errors surface at synthesis time
         return False
 

specs/architecture-runtime-inventory.md

@@ -1,7 +1,7 @@
 # Architecture Runtime Inventory
 
 > **Purpose**: Phase 0 planning baseline for codebase readability improvements (#4071).
-> **Parent issue**: [#4082](https://github.com/pewdiepie-archdaemon/odysseus/issues/4082)
+> **Parent issue**: [#4082](https://github.com/odysseus-dev/odysseus/issues/4082)
 > **Last updated**: dev@b58af42 | 2026-06-16
 > **Status**: Draft — to be reviewed before follow-up slices open.
 > **Snapshot basis**: Importer / file / import-line counts are refreshed to `dev@b58af42` (2026-06-16) and are recomputable via the commands in §3.4. **Line counts** in §2.1 / §2.2 are a snapshot from an earlier baseline and drift as `dev` moves — recompute any of them with `wc -l <file>`. This inventory tracks structure and risk, not live metrics.

src/action_intents.py

@@ -110,6 +110,18 @@ _ROUTING_PATTERNS: tuple[tuple[str, str, Pattern[str]], ...] = tuple(
         ("research", "deep research imperative request", rf"{_PLEASE}(?:research|deep\s+dive|look\s+into|investigate)\s+.+"),
         ("research", "assistant deep research request", rf"{_ACTION_QUESTION}(?:research|do\s+research|deep\s+dive|look\s+into|investigate)\s+.+"),
 
+        # Workspace / coding-agent intent. These should promote to the agent
+        # workspace with shell/file tools available, not the "light" typed-tool
+        # path used for notes/calendar/email.
+        ("workspace", "repo implementation request", rf"{_PLEASE}(?:fix|debug|implement|change|update|refactor|patch|review|test)\b.{{0,160}}\b(?:repo|repository|codebase|project|app|server|api|frontend|backend|tests?|bug|issue|pr)\b"),
+        ("workspace", "assistant repo implementation request", rf"{_ACTION_QUESTION}(?:fix|debug|implement|change|update|refactor|patch|review|test)\b.{{0,160}}\b(?:repo|repository|codebase|project|app|server|api|frontend|backend|tests?|bug|issue|pr)\b"),
+        ("workspace", "test/build command request", rf"{_PLEASE}(?:run|execute|start|launch)\b.{{0,80}}\b(?:tests?|pytest|npm\s+test|pnpm\s+test|yarn\s+test|build|lint|typecheck|benchmark|eval|terminal[- ]bench|tbench)\b"),
+        ("workspace", "file/code inspection request", rf"{_PLEASE}(?:find|inspect|look\s+at|open|read|check)\b.{{0,120}}\b(?:file|folder|directory|repo|repository|code|source|logs?|trace|stack|diff)\b"),
+        ("workspace", "server/process debugging request", rf"{_PLEASE}(?:check|debug|fix|restart|start|stop|kill|tail|inspect)\b.{{0,120}}\b(?:server|service|process|port|docker|container|tmux|endpoint|logs?)\b"),
+        ("workspace", "local computer task request", r"\b(?:on|from|in|using|with)\s+(?:this|my|the)\s+(?:computer|machine|pc|laptop|device|system)\b|\b(?:local|host)\s+(?:computer|machine|files?|system)\b"),
+        ("workspace", "named computer task request", r"\b(?:on|from)\s+(?!this\b|my\b|the\b|a\b|an\b)(?:[a-z][a-z0-9_.-]{1,31})\b"),
+        ("workspace", "terminal workspace request", r"\b(?:terminal|shell|workspace|tmux|docker|container|git|branch|commit|diff|pytest|stacktrace|traceback|benchmark|terminal[- ]bench|tbench)\b"),
+
         # Shell / remote-host intent.
         ("shell", "ssh request", r"\bssh\s+(?:in)?to\b"),
         ("shell", "ssh target request", r"\bssh\s+\w+"),

src/agent_tools/__init__.py

@@ -21,7 +21,8 @@ logger = logging.getLogger(__name__)
 
 from .subprocess_tools import BashTool, PythonTool
 from .web_tools import WebSearchTool, WebFetchTool
-from .filesystem_tools import ReadFileTool, WriteFileTool, EditFileTool, LsTool, GlobTool, GrepTool, GetWorkspaceTool
+from .filesystem_tools import ReadFileTool, WriteFileTool, EditFileTool, ApplyPatchTool, LsTool, GlobTool, GrepTool, GetWorkspaceTool
+from .coding_tools import TodoWriteTool
 from .document_tools import CreateDocumentTool, UpdateDocumentTool, EditDocumentTool, SuggestDocumentTool, ManageDocumentTool
 from .interaction_tools import AskUserTool, UpdatePlanTool
 from .model_interaction_tools import ChatWithModelTool, AskTeacherTool, ListModelsTool
@@ -41,6 +42,8 @@ TOOL_HANDLERS = {
     "read_file": ReadFileTool().execute,
     "write_file": WriteFileTool().execute,
     "edit_file": EditFileTool().execute,
+    "apply_patch": ApplyPatchTool().execute,
+    "todowrite": TodoWriteTool().execute,
     "ls": LsTool().execute,
     "glob": GlobTool().execute,
     "grep": GrepTool().execute,
@@ -74,6 +77,7 @@ PYTHON_TIMEOUT = 30
 
 # Tool types that trigger execution
 TOOL_TAGS = {"bash", "python", "web_search", "web_fetch", "read_file", "write_file", "edit_file",
+             "apply_patch", "todowrite",
              "grep", "glob", "ls", "get_workspace", "manage_bg_jobs",
              "create_document", "update_document", "edit_document",
              "search_chats",

src/agent_tools/coding_tools.py

@@ -0,0 +1,67 @@
+import json
+import os
+import re
+from typing import Any, Dict, List
+
+from src.constants import DATA_DIR
+
+
+_TODO_DIR = os.path.join(DATA_DIR, "agent_todos")
+
+
+def _safe_session_id(value: str) -> str:
+    value = value or "current"
+    return re.sub(r"[^A-Za-z0-9_.-]+", "_", value)[:120] or "current"
+
+
+class TodoWriteTool:
+    async def execute(self, content: str, ctx: dict) -> dict:
+        try:
+            args = json.loads(content) if (content or "").strip().startswith("{") else {"todos": []}
+        except (json.JSONDecodeError, TypeError):
+            return {"error": "todowrite: JSON object required", "exit_code": 1}
+        todos = args.get("todos")
+        if not isinstance(todos, list):
+            return {"error": "todowrite: todos must be a list", "exit_code": 1}
+
+        normalized: List[Dict[str, Any]] = []
+        allowed_statuses = {"pending", "in_progress", "completed"}
+        allowed_priorities = {"low", "medium", "high"}
+        active_count = 0
+        for item in todos:
+            if not isinstance(item, dict):
+                return {"error": "todowrite: each todo must be an object", "exit_code": 1}
+            content_text = str(item.get("content") or item.get("text") or "").strip()
+            if not content_text:
+                return {"error": "todowrite: todo content required", "exit_code": 1}
+            status = str(item.get("status") or "pending").strip()
+            if status not in allowed_statuses:
+                return {"error": f"todowrite: invalid status {status!r}", "exit_code": 1}
+            if status == "in_progress":
+                active_count += 1
+            priority = str(item.get("priority") or "medium").strip()
+            if priority not in allowed_priorities:
+                priority = "medium"
+            normalized.append({
+                "content": content_text,
+                "status": status,
+                "priority": priority,
+            })
+        if active_count > 1:
+            return {"error": "todowrite: only one todo can be in_progress", "exit_code": 1}
+
+        session_id = _safe_session_id(str(ctx.get("session_id") or args.get("session_id") or "current"))
+        os.makedirs(_TODO_DIR, exist_ok=True)
+        path = os.path.join(_TODO_DIR, f"{session_id}.json")
+        with open(path, "w", encoding="utf-8") as f:
+            json.dump({"todos": normalized}, f, ensure_ascii=False, indent=2)
+
+        lines = []
+        for item in normalized:
+            marker = {"pending": " ", "in_progress": ">", "completed": "x"}[item["status"]]
+            lines.append(f"[{marker}] {item['content']} ({item['priority']})")
+        return {
+            "output": "Updated todo list:\n" + ("\n".join(lines) if lines else "(empty)"),
+            "exit_code": 0,
+            "todos": normalized,
+        }

src/agent_tools/document_tools.py

@@ -2,10 +2,16 @@ from typing import Any, Dict, List, Optional
 import logging
 import re
 from src.constants import MAX_READ_CHARS
-from src.tool_utils import _parse_tool_args
+from src.tool_utils import _parse_tool_args, get_upload_handler
+from src.upload_handler import reserve_upload_references
 
 logger = logging.getLogger(__name__)
 
+
+def _missing_document_upload(owner: Optional[str], content: Any) -> Optional[str]:
+    """Reserve explicit upload URLs before an agent persists document text."""
+    return reserve_upload_references(get_upload_handler(), owner, content)
+
 # ---------------------------------------------------------------------------
 # Active document state
 # ---------------------------------------------------------------------------
@@ -384,6 +390,13 @@ class CreateDocumentTool:
                 return {"error": "Cannot create document in another user's session"}
             _owner = _sess.owner if _sess else None
 
+            missing_id = _missing_document_upload(_owner, content)
+            if missing_id:
+                return {
+                    "error": f"Referenced upload is no longer available: {missing_id}",
+                    "exit_code": 1,
+                }
+
             doc = Document(
                 id=doc_id,
                 session_id=session_id,
@@ -455,6 +468,13 @@ class UpdateDocumentTool:
             if is_email_doc:
                 doc.language = "email"
 
+            missing_id = _missing_document_upload(owner, new_content)
+            if missing_id:
+                return {
+                    "error": f"Referenced upload is no longer available: {missing_id}",
+                    "exit_code": 1,
+                }
+
             if not is_email_doc and _pdf_source_upload_id(doc.current_content or ""):
                 return _create_pdf_text_derivative(
                     db,
@@ -532,6 +552,12 @@ class EditDocumentTool:
                     applied = 1
                     skipped = max(0, len(edits) - 1)
                     doc.language = "email"
+                    missing_id = _missing_document_upload(owner, updated_content)
+                    if missing_id:
+                        return {
+                            "error": f"Referenced upload is no longer available: {missing_id}",
+                            "exit_code": 1,
+                        }
                     new_ver = doc.version_count + 1
                     ver = DocumentVersion(
                         id=str(uuid.uuid4()),
@@ -584,6 +610,13 @@ class EditDocumentTool:
             if applied == 0:
                 return {"error": f"No edits applied — none of the FIND blocks matched the document content (skipped {skipped})"}
 
+            missing_id = _missing_document_upload(owner, updated_content)
+            if missing_id:
+                return {
+                    "error": f"Referenced upload is no longer available: {missing_id}",
+                    "exit_code": 1,
+                }
+
             if _pdf_source_upload_id(doc.current_content or ""):
                 return _create_pdf_text_derivative(
                     db,

src/agent_tools/filesystem_tools.py

@@ -5,7 +5,7 @@ import re
 import difflib
 import fnmatch
 import shutil
-from typing import Optional, Dict, Any, Tuple
+from typing import Optional, Dict, Any, Tuple, List
 
 from src.constants import MAX_READ_CHARS, MAX_DIFF_LINES, MAX_OUTPUT_CHARS
 
@@ -230,6 +230,181 @@ class WriteFileTool:
             result["diff"] = diff
         return result
 
+class ApplyPatchTool:
+    async def execute(self, content: str, ctx: dict) -> dict:
+        """Apply a small Codex-style patch using exact context matching.
+
+        This is deliberately stricter than git-apply: if an update hunk's old
+        text is not found exactly once, the whole patch is rejected before any
+        file is changed. That keeps agent edits reviewable and avoids fuzzy
+        corruption when the model patches stale context.
+        """
+        from src.tool_execution import _resolve_tool_path
+
+        patch_text = content or ""
+        stripped = patch_text.strip()
+        if stripped.startswith("{"):
+            try:
+                args = json.loads(stripped)
+                if isinstance(args, dict):
+                    patch_text = str(args.get("patch_text") or args.get("patchText") or args.get("patch") or "")
+            except (json.JSONDecodeError, TypeError):
+                pass
+        if not patch_text.strip():
+            return {"error": "apply_patch: patch_text required", "exit_code": 1}
+
+        try:
+            ops = _parse_agent_patch(patch_text)
+            if not ops:
+                return {"error": "apply_patch: no file operations found", "exit_code": 1}
+            prepared = []
+            for op in ops:
+                path = _resolve_tool_path(op["path"])
+                kind = op["kind"]
+                if kind == "add":
+                    if os.path.exists(path):
+                        return {"error": f"apply_patch: {op['path']}: already exists", "exit_code": 1}
+                    old = ""
+                    new = op["content"]
+                elif kind == "delete":
+                    if not os.path.isfile(path):
+                        return {"error": f"apply_patch: {op['path']}: not found", "exit_code": 1}
+                    with open(path, "r", encoding="utf-8") as f:
+                        old = f.read()
+                    new = ""
+                else:
+                    if not os.path.isfile(path):
+                        return {"error": f"apply_patch: {op['path']}: not found", "exit_code": 1}
+                    with open(path, "r", encoding="utf-8") as f:
+                        old = f.read()
+                    new = _apply_patch_hunks(old, op["hunks"], op["path"])
+                prepared.append((kind, path, old, new))
+
+            diffs = []
+            for kind, path, old, new in prepared:
+                if kind == "delete":
+                    os.remove(path)
+                else:
+                    directory = os.path.dirname(path)
+                    if directory:
+                        os.makedirs(directory, exist_ok=True)
+                    with open(path, "w", encoding="utf-8") as f:
+                        f.write(new)
+                diff = _unified_diff(old, new, path)
+                if diff:
+                    diffs.append(diff)
+        except (ValueError, UnicodeDecodeError, PermissionError, OSError) as e:
+            return {"error": f"apply_patch: {e}", "exit_code": 1}
+
+        added = sum(int(d.get("added") or 0) for d in diffs)
+        removed = sum(int(d.get("removed") or 0) for d in diffs)
+        text_parts = [d.get("text", "") for d in diffs if d.get("text")]
+        diff_text = "\n".join(text_parts)
+        if len(diff_text.splitlines()) > MAX_DIFF_LINES:
+            diff_text = "\n".join(diff_text.splitlines()[:MAX_DIFF_LINES]) + f"\n... diff truncated at {MAX_DIFF_LINES} lines"
+        result = {
+            "output": f"Applied patch ({len(prepared)} file{'s' if len(prepared) != 1 else ''}, +{added}/-{removed})",
+            "exit_code": 0,
+        }
+        if diffs:
+            result["diff"] = {
+                "text": diff_text,
+                "added": added,
+                "removed": removed,
+                "new_file": any(d.get("new_file") for d in diffs),
+                "file": "patch",
+            }
+        return result
+
+def _parse_agent_patch(patch_text: str) -> List[Dict[str, Any]]:
+    lines = patch_text.replace("\r\n", "\n").replace("\r", "\n").split("\n")
+    while lines and not lines[0].strip():
+        lines.pop(0)
+    while lines and not lines[-1].strip():
+        lines.pop()
+    if not lines or lines[0].strip() != "*** Begin Patch":
+        raise ValueError("patch must start with *** Begin Patch")
+    if lines[-1].strip() != "*** End Patch":
+        raise ValueError("patch must end with *** End Patch")
+
+    ops: List[Dict[str, Any]] = []
+    i = 1
+    while i < len(lines) - 1:
+        line = lines[i]
+        if not line:
+            i += 1
+            continue
+        if line.startswith("*** Add File: "):
+            path = line[len("*** Add File: "):].strip()
+            body = []
+            i += 1
+            while i < len(lines) - 1 and not lines[i].startswith("*** "):
+                if not lines[i].startswith("+"):
+                    raise ValueError(f"add file {path}: every content line must start with +")
+                body.append(lines[i][1:])
+                i += 1
+            ops.append({"kind": "add", "path": path, "content": "\n".join(body) + ("\n" if body else "")})
+            continue
+        if line.startswith("*** Delete File: "):
+            path = line[len("*** Delete File: "):].strip()
+            ops.append({"kind": "delete", "path": path})
+            i += 1
+            continue
+        if line.startswith("*** Update File: "):
+            path = line[len("*** Update File: "):].strip()
+            hunks = []
+            current = []
+            i += 1
+            if i < len(lines) - 1 and lines[i].startswith("*** Move to: "):
+                raise ValueError("move operations are not supported")
+            while i < len(lines) - 1 and not lines[i].startswith("*** "):
+                if lines[i].startswith("@@"):
+                    if current:
+                        hunks.append(current)
+                        current = []
+                elif lines[i].startswith((" ", "-", "+")):
+                    current.append(lines[i])
+                elif lines[i] == "":
+                    current.append(" ")
+                else:
+                    raise ValueError(f"update file {path}: invalid patch line {lines[i]!r}")
+                i += 1
+            if current:
+                hunks.append(current)
+            if not hunks:
+                raise ValueError(f"update file {path}: no hunks")
+            ops.append({"kind": "update", "path": path, "hunks": hunks})
+            continue
+        raise ValueError(f"unexpected patch line: {line!r}")
+    return ops
+
+def _apply_patch_hunks(original: str, hunks: List[List[str]], label: str) -> str:
+    updated = original
+    for idx, hunk in enumerate(hunks, 1):
+        old_lines = []
+        new_lines = []
+        for line in hunk:
+            prefix, body = line[:1], line[1:]
+            if prefix in (" ", "-"):
+                old_lines.append(body)
+            if prefix in (" ", "+"):
+                new_lines.append(body)
+        old_text = "\n".join(old_lines)
+        new_text = "\n".join(new_lines)
+        if old_text and old_text in updated:
+            occurrences = updated.count(old_text)
+            if occurrences != 1:
+                raise ValueError(f"{label}: hunk {idx} context matched {occurrences} times")
+            updated = updated.replace(old_text, new_text, 1)
+        elif old_text + "\n" in updated:
+            occurrences = updated.count(old_text + "\n")
+            if occurrences != 1:
+                raise ValueError(f"{label}: hunk {idx} context matched {occurrences} times")
+            updated = updated.replace(old_text + "\n", new_text + "\n", 1)
+        else:
+            raise ValueError(f"{label}: hunk {idx} context not found")
+    return updated
+
 class LsTool:
     async def execute(self, content: str, ctx: dict) -> dict:
         from src.tool_execution import _resolve_tool_path, _resolve_search_root, _truncate

src/agent_tools/subprocess_tools.py

@@ -1,4 +1,7 @@
 import asyncio
+import os
+import re
+import shutil
 import sys
 import time
 import collections
@@ -10,6 +13,175 @@ DEFAULT_PYTHON_TIMEOUT = 60 * 60
 
 PROGRESS_INTERVAL_S = 2.0
 PROGRESS_TAIL_LINES = 12
+TMUX_CAPTURE_LINES = 2000
+
+
+def _tmux_session_name(session_id: Optional[str]) -> str:
+    raw = re.sub(r"[^A-Za-z0-9_.-]+", "-", str(session_id or "default")).strip("-")
+    return f"ody-agent-{raw[:80] or 'default'}"
+
+
+async def _run_exec(*args: str, timeout: float = 10) -> Tuple[str, str, int]:
+    proc = await asyncio.create_subprocess_exec(
+        *args,
+        stdout=asyncio.subprocess.PIPE,
+        stderr=asyncio.subprocess.PIPE,
+    )
+    try:
+        out_b, err_b = await asyncio.wait_for(proc.communicate(), timeout=timeout)
+    except asyncio.TimeoutError:
+        try:
+            proc.kill()
+        except Exception:
+            pass
+        return "", "timeout", 124
+    return (
+        out_b.decode("utf-8", errors="replace"),
+        err_b.decode("utf-8", errors="replace"),
+        proc.returncode or 0,
+    )
+
+
+async def _tmux_has_session(name: str) -> bool:
+    _, _, rc = await _run_exec("tmux", "has-session", "-t", name, timeout=3)
+    return rc == 0
+
+
+async def _tmux_capture(name: str) -> str:
+    out, _, _ = await _run_exec(
+        "tmux", "capture-pane", "-p", "-J", "-S", f"-{TMUX_CAPTURE_LINES}", "-t", name,
+        timeout=5,
+    )
+    return out
+
+
+async def _tmux_send_line(name: str, line: str) -> None:
+    if line:
+        await _run_exec("tmux", "send-keys", "-t", name, "-l", line, timeout=5)
+    await _run_exec("tmux", "send-keys", "-t", name, "C-m", timeout=5)
+
+
+async def _ensure_tmux_session(name: str, cwd: str, env: Optional[dict]) -> None:
+    if await _tmux_has_session(name):
+        await _run_exec("tmux", "send-keys", "-t", name, "stty -echo", "C-m", timeout=5)
+        return
+    await _run_exec(
+        "tmux", "new-session", "-d", "-s", name, "-c", cwd,
+        "env",
+        f"TERM={env.get('TERM', 'xterm-256color') if env else 'xterm-256color'}",
+        f"COLUMNS={env.get('COLUMNS', '120') if env else '120'}",
+        f"LINES={env.get('LINES', '40') if env else '40'}",
+        "/bin/bash",
+        "--noprofile",
+        "--norc",
+        timeout=10,
+    )
+    if not await _tmux_has_session(name):
+        raise RuntimeError(f"failed to create tmux session {name}")
+    await _run_exec("tmux", "send-keys", "-t", name, "stty -echo", "C-m", timeout=5)
+
+
+def _output_after_marker(capture: str, start_marker: str, end_marker: str) -> Tuple[str, bool]:
+    lines = capture.splitlines()
+    start_idx = -1
+    for idx, line in enumerate(lines):
+        if line.strip() == start_marker:
+            start_idx = idx
+    if start_idx < 0:
+        return capture, False
+    end_idx = -1
+    for idx in range(start_idx + 1, len(lines)):
+        if lines[idx].strip().startswith(end_marker):
+            end_idx = idx
+    if end_idx < 0:
+        return "\n".join(lines[start_idx + 1:]), False
+    return "\n".join(lines[start_idx + 1:end_idx]), True
+
+
+def _extract_marker_rc(capture: str, end_marker: str) -> int:
+    for line in reversed(capture.splitlines()):
+        stripped = line.strip()
+        if stripped.startswith(end_marker):
+            suffix = stripped[len(end_marker):].strip()
+            if suffix.isdigit():
+                return int(suffix)
+    return 0
+
+
+async def _run_tmux_bash(
+    content: str,
+    *,
+    session_id: str,
+    cwd: str,
+    env: Optional[dict],
+    timeout: float,
+    progress_cb: Optional[Callable[[Dict], Awaitable[None]]] = None,
+) -> Tuple[str, str, Optional[int], bool]:
+    name = _tmux_session_name(session_id)
+    await _ensure_tmux_session(name, cwd, env)
+
+    stamp = f"{int(time.time() * 1000)}-{abs(hash(content)) % 1000000}"
+    start_marker = f"__ODYSSEUS_CMD_START_{stamp}__"
+    end_prefix = f"__ODYSSEUS_CMD_END_{stamp}__:"
+    wrapped = (
+        f"printf '\\n{start_marker}\\n'\n"
+        f"{content}\n"
+        f"__ody_rc=$?\n"
+        f"printf '\\n{end_prefix}%s\\n' \"$__ody_rc\"\n"
+    )
+    for line in wrapped.splitlines():
+        await _tmux_send_line(name, line)
+
+    started = time.time()
+    last_tail = ""
+    while True:
+        capture = await _tmux_capture(name)
+        body, done = _output_after_marker(capture, start_marker, end_prefix)
+        tail = "\n".join(body.splitlines()[-PROGRESS_TAIL_LINES:])
+        if progress_cb and tail != last_tail:
+            last_tail = tail
+            try:
+                await progress_cb({
+                    "elapsed_s": round(time.time() - started, 1),
+                    "tail": tail,
+                    "tmux_session": name,
+                })
+            except Exception:
+                pass
+        if done:
+            rc = _extract_marker_rc(capture, end_prefix)
+            cleaned = _clean_tmux_command_output(body, wrapped)
+            return cleaned, "", rc, False
+        if time.time() - started > timeout:
+            try:
+                await _run_exec("tmux", "send-keys", "-t", name, "C-c", timeout=3)
+            except Exception:
+                pass
+            cleaned = _clean_tmux_command_output(body, wrapped)
+            return cleaned, "", 124, True
+        await asyncio.sleep(0.5)
+
+
+def _clean_tmux_command_output(text: str, wrapped_command: str) -> str:
+    lines = text.splitlines()
+    wrapped_lines = {ln.rstrip() for ln in wrapped_command.splitlines() if ln.strip()}
+    cleaned = []
+    for line in lines:
+        raw = line.rstrip()
+        stripped = raw.strip()
+        if not stripped:
+            cleaned.append(raw)
+            continue
+        if stripped in wrapped_lines:
+            continue
+        if stripped.startswith("__ody_rc=") or stripped.startswith("printf "):
+            continue
+        if re.fullmatch(r"(?:bash|sh)-[\d.]+\$ ?", stripped):
+            continue
+        if re.fullmatch(r"[\w.@:/~+-]+[#$] ?", stripped):
+            continue
+        cleaned.append(raw)
+    return "\n".join(cleaned).strip()
 
 async def _run_subprocess_streaming(
     proc: asyncio.subprocess.Process,
@@ -103,8 +275,38 @@ async def _run_subprocess_streaming(
 class BashTool:
     async def execute(self, content: str, ctx: dict) -> dict:
         from src.tool_execution import agent_cwd, _truncate
+        if isinstance(content, dict):
+            content = str(content.get("command") or content.get("cmd") or content.get("code") or "")
         progress_cb = ctx.get("progress_cb")
         _subproc_env = ctx.get("subproc_env")
+        session_id = ctx.get("session_id")
+        if session_id and shutil.which("tmux"):
+            stdout, stderr, rc, timed_out = await _run_tmux_bash(
+                content,
+                session_id=str(session_id),
+                cwd=agent_cwd(),
+                env=_subproc_env,
+                timeout=DEFAULT_BASH_TIMEOUT,
+                progress_cb=progress_cb,
+            )
+            if timed_out:
+                return {
+                    "error": f"bash: timed out after {DEFAULT_BASH_TIMEOUT}s — sent Ctrl-C to tmux session",
+                    "exit_code": 124,
+                    "stdout": _truncate(stdout, MAX_OUTPUT_CHARS),
+                    "stderr": _truncate(stderr, MAX_OUTPUT_CHARS),
+                    "tmux_session": _tmux_session_name(str(session_id)),
+                }
+            output = stdout.rstrip()
+            err = stderr.rstrip()
+            if err:
+                output = (output + "\nSTDERR: " + err).strip() if output else "STDERR: " + err
+            return {
+                "output": _truncate(output, MAX_OUTPUT_CHARS) or "(no output)",
+                "exit_code": rc or 0,
+                "tmux_session": _tmux_session_name(str(session_id)),
+            }
+
         proc = await asyncio.create_subprocess_shell(
             content,
             stdout=asyncio.subprocess.PIPE,

src/ai_interaction.py

@@ -19,7 +19,7 @@ import json
 import logging
 import uuid
 import time
-from typing import Dict, Optional, Tuple
+from typing import Any, Awaitable, Callable, Dict, Optional, Tuple
 
 from src.constants import GENERATED_IMAGES_DIR
 
@@ -71,9 +71,10 @@ def set_rag_manager(rag_mgr, personal_docs_mgr=None):
 # ---------------------------------------------------------------------------
 
 from src.endpoint_resolver import build_chat_url, build_headers, build_models_url, resolve_endpoint_runtime
+from src.image_model_ids import looks_like_image_generation_model, model_id_leaf
 
 
-def _resolve_model(spec: str, owner: Optional[str] = None) -> Tuple[str, str, Dict]:
+def _resolve_model(spec: str, owner: Optional[str] = None, model_type: Optional[str] = None) -> Tuple[str, str, Dict]:
     """Resolve a model specifier to (endpoint_url, model_id, headers).
 
     Accepts:
@@ -97,9 +98,29 @@ def _resolve_model(spec: str, owner: Optional[str] = None) -> Tuple[str, str, Di
     else:
         model_name = spec
 
+    def _json_list(value) -> list[str]:
+        try:
+            data = json.loads(value or "[]")
+        except Exception:
+            return []
+        if not isinstance(data, list):
+            return []
+        return [str(x) for x in data if isinstance(x, (str, int, float)) and str(x)]
+
+    def _image_like(name: str) -> bool:
+        n = (name or "").lower()
+        if looks_like_image_generation_model(n):
+            return True
+        return any(k in n for k in (
+            "qwen-image", "qwen/image", "z-image", "flux", "stable-diffusion",
+            "sdxl", "hidream", "boogu", "krea-2", "image-edit",
+        ))
+
     db = SessionLocal()
     try:
         query = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True)
+        if model_type:
+            query = query.filter(ModelEndpoint.model_type == model_type)
         if target_endpoint_name:
             query = query.filter(ModelEndpoint.name.ilike(f"%{target_endpoint_name}%"))
         if owner:
@@ -129,11 +150,13 @@ def _resolve_model(spec: str, owner: Optional[str] = None) -> Tuple[str, str, Di
                     return build_chat_url(base), matched, headers
             else:
                 # OpenAI-compatible and native Ollama: probe the provider's model list.
+                endpoint_reachable = False
                 try:
                     models_url = build_models_url(base)
                     if models_url:
                         r = httpx.get(models_url, headers=headers, timeout=5)
                         r.raise_for_status()
+                        endpoint_reachable = True
                         data = r.json()
                         items = data if isinstance(data, list) else (data.get("data") or [])
                         model_ids = [m.get("id") for m in items if isinstance(m, dict) and m.get("id")]
@@ -144,10 +167,21 @@ def _resolve_model(spec: str, owner: Optional[str] = None) -> Tuple[str, str, Di
                                 if m.get("name") or m.get("model")
                             ]
                     else:
+                        endpoint_reachable = True
                         model_ids = json.loads(ep.cached_models or "[]")
                 except Exception:
                     model_ids = []
 
+                # Manual/local image endpoints are often registered with pinned
+                # model ids, while /models may return a runtime alias or only the
+                # served internal id. Include pinned/cached ids in the match set
+                # so chat sessions using the HF repo id still resolve. Do not use
+                # stale cached aliases when the endpoint itself is unreachable.
+                if model_type == "image" and endpoint_reachable:
+                    for extra in _json_list(getattr(ep, "pinned_models", None)) + _json_list(getattr(ep, "cached_models", None)):
+                        if extra not in model_ids:
+                            model_ids.append(extra)
+
                 # Exact match first
                 for mid in model_ids:
                     if mid.lower() == model_name.lower():
@@ -158,6 +192,13 @@ def _resolve_model(spec: str, owner: Optional[str] = None) -> Tuple[str, str, Di
                     if model_name.lower() in mid.lower() or mid.lower() in model_name.lower():
                         return build_chat_url(base), mid, headers
 
+                # Last resort for local image endpoints: if the requested model
+                # name is clearly an image model, use the endpoint's first known
+                # image model id. This prevents a harmless alias mismatch from
+                # blocking image generation.
+                if model_type == "image" and _image_like(model_name) and model_ids:
+                    return build_chat_url(base), model_ids[0], headers
+
         raise ValueError(f"Model '{spec}' not found on any configured endpoint")
     finally:
         db.close()
@@ -967,16 +1008,36 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
         if not model_spec:
             return {"error": "No image model found. Configure one in Admin → Image Generation."}
 
+    async def _resolve_image_model(model_name: str):
+        def _call():
+            try:
+                return _resolve_model(model_name, owner=owner, model_type="image")
+            except TypeError as exc:
+                if "model_type" not in str(exc):
+                    raise
+                return _resolve_model(model_name, owner=owner)
+        return await asyncio.to_thread(_call)
+
     # Resolve the model to find the right endpoint
     try:
-        url, model_id, headers = await asyncio.to_thread(_resolve_model, model_spec, owner=owner)
+        try:
+            url, model_id, headers = await _resolve_image_model(model_spec)
+        except ValueError:
+            _lower_model_spec = model_spec.lower()
+            if not (
+                any(_name in _lower_model_spec for _name in ("gpt-image", "dall-e"))
+                or looks_like_image_generation_model(_lower_model_spec)
+            ):
+                raise
+            url, model_id, headers = await asyncio.to_thread(_resolve_model, model_spec, owner=owner)
     except ValueError:
         return {"error": f"No endpoint found with image model '{model_spec}'. "
                 "Configure an OpenAI-compatible endpoint with image generation support."}
 
     # Detect if this is a GPT image model vs DALL-E vs local diffusion
-    is_gpt_image = "gpt-image" in model_id.lower()
-    is_dalle = "dall-e" in model_id.lower()
+    _model_leaf = model_id_leaf(model_id)
+    is_gpt_image = _model_leaf.startswith("gpt-image") or (_model_leaf.startswith("gpt-") and "-image" in _model_leaf)
+    is_dalle = _model_leaf.startswith("dall-e")
     is_local_diffusion = not is_gpt_image and not is_dalle
 
     # Build the images endpoint URL from the chat completions URL
@@ -1106,6 +1167,267 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
         return {"error": f"Image generation error: {str(e)}"}
 
 
+async def do_edit_image(
+    prompt: str,
+    image_path: str,
+    model_spec: str = "",
+    session_id: Optional[str] = None,
+    owner: Optional[str] = None,
+    size: str = "1024x1024",
+    quality: str = "medium",
+    progress_callback: Optional[Callable[[Dict[str, Any]], Awaitable[None]]] = None,
+) -> Dict:
+    """Edit an uploaded image using the configured image endpoint."""
+    import base64
+    import httpx
+    import mimetypes
+    import os
+    from pathlib import Path
+    from src.url_safety import check_outbound_url
+
+    prompt = (prompt or "").strip()
+    if not prompt:
+        return {"error": "Image edit prompt is required"}
+    path = Path(image_path)
+    if not path.exists() or not path.is_file():
+        return {"error": "Attached image file was not found"}
+
+    try:
+        from src.settings import load_settings
+        _settings = load_settings()
+    except Exception:
+        _settings = {}
+
+    if not model_spec:
+        model_spec = _settings.get("image_model", "")
+    if quality == "medium" and _settings.get("image_quality"):
+        quality = _settings["image_quality"]
+    if not model_spec:
+        return {"error": "No image model selected for image editing"}
+
+    try:
+        try:
+            def _call():
+                try:
+                    return _resolve_model(model_spec, owner=owner, model_type="image")
+                except TypeError as exc:
+                    if "model_type" not in str(exc):
+                        raise
+                    return _resolve_model(model_spec, owner=owner)
+            url, model_id, headers = await asyncio.to_thread(_call)
+        except ValueError:
+            url, model_id, headers = await asyncio.to_thread(_resolve_model, model_spec, owner=owner)
+    except ValueError:
+        return {"error": f"No endpoint found with image model '{model_spec}'."}
+
+    base_url = url.replace("/chat/completions", "").replace("/v1/messages", "").rstrip("/")
+    edits_url = base_url + "/images/edits"
+    mime = mimetypes.guess_type(str(path))[0] or "image/png"
+    payload = {
+        "model": model_id,
+        "prompt": prompt,
+        "n": "1",
+        "size": size,
+        "quality": quality if quality in ("low", "medium", "high", "auto") else "medium",
+        "response_format": "b64_json",
+    }
+    request_id = uuid.uuid4().hex
+    payload["request_id"] = request_id
+
+    logger.info("Image edit: model=%s, size=%s, quality=%s, image=%s, prompt=%s", model_id, size, quality, path.name, prompt[:80])
+
+    def _save_edited_image_to_gallery(filename: str) -> str:
+        try:
+            from src.database import SessionLocal as _GallerySL, GalleryImage
+            new_id = str(uuid.uuid4())
+            _gdb = _GallerySL()
+            _gdb.add(GalleryImage(
+                id=new_id,
+                filename=filename,
+                prompt=prompt,
+                model=model_id,
+                size=size,
+                quality=payload.get("quality", "medium"),
+                session_id=session_id,
+                owner=owner,
+            ))
+            _gdb.commit()
+            _gdb.close()
+            return new_id
+        except Exception as _ge:
+            logger.warning("Failed to save edited image gallery record: %s", _ge)
+            return ""
+
+    def _save_image_bytes(image_bytes: bytes, suffix: str = ".png") -> tuple[str, str]:
+        img_dir = Path(GENERATED_IMAGES_DIR)
+        img_dir.mkdir(parents=True, exist_ok=True)
+        filename = f"{uuid.uuid4().hex[:12]}{suffix}"
+        (img_dir / filename).write_bytes(image_bytes)
+        return f"/api/generated-image/{filename}", _save_edited_image_to_gallery(filename)
+
+    async def _try_local_img2img_fallback(client: httpx.AsyncClient) -> Optional[Dict[str, Any]]:
+        """Try Odysseus' local diffusion img2img endpoint.
+
+        Some self-hosted SD/SDXL endpoints expose text-to-image plus
+        `/images/harmonize`/img2img, but not OpenAI's multipart
+        `/images/edits`. For chat uploads ("image + prompt"), this gives the
+        expected instruction-edit behavior instead of stopping at a 400.
+        """
+        harmonize_url = base_url + "/images/harmonize"
+        try:
+            image_bytes = path.read_bytes()
+            image_b64 = base64.b64encode(image_bytes).decode()
+            fallback_payload = {
+                "image": image_b64,
+                "prompt": prompt,
+                "strength": 0.35,
+                "steps": 0,
+                "max_side": 1024,
+            }
+            if progress_callback:
+                await progress_callback({
+                    "status": "running",
+                    "message": "Trying image-to-image fallback",
+                    "step": 0,
+                    "total": 0,
+                })
+            fallback_resp = await client.post(harmonize_url, json=fallback_payload, headers=headers)
+            if fallback_resp.status_code == 404:
+                return None
+            if fallback_resp.status_code != 200:
+                error_text = fallback_resp.text[:500]
+                try:
+                    err_json = fallback_resp.json()
+                    error_text = err_json.get("detail") or err_json.get("error") or error_text
+                except Exception:
+                    pass
+                return {"error": f"Image edit fallback failed ({fallback_resp.status_code}): {error_text}"}
+            fallback_data = fallback_resp.json()
+            image_b64 = fallback_data.get("image")
+            if not image_b64:
+                return {"error": "Image edit fallback returned no image"}
+            image_url, image_id = _save_image_bytes(base64.b64decode(image_b64))
+            return {
+                "results": f"Edited image for: {prompt[:100]}",
+                "image_url": image_url,
+                "image_id": image_id,
+                "image_prompt": prompt,
+                "image_model": model_id,
+                "image_size": size,
+                "image_quality": payload.get("quality", "medium"),
+                "edit_route": "img2img",
+            }
+        except httpx.TimeoutException:
+            return {"error": "Image edit fallback timed out. The model may still be loading or overloaded."}
+        except Exception as fallback_error:
+            logger.warning("Image edit fallback failed: %s", fallback_error)
+            return {"error": f"Image edit fallback error: {fallback_error}"}
+
+    try:
+        async with httpx.AsyncClient(timeout=httpx.Timeout(connect=30.0, read=600.0, write=60.0, pool=30.0)) as client:
+            progress_task = None
+            if progress_callback:
+                progress_url = base_url + f"/images/progress/{request_id}"
+
+                async def _poll_progress():
+                    last_sig = None
+                    while True:
+                        try:
+                            pr = await client.get(progress_url, headers=headers, timeout=5.0)
+                            if pr.status_code == 404:
+                                return
+                            if pr.status_code == 200:
+                                data = pr.json()
+                                sig = (data.get("status"), data.get("step"), data.get("total"), data.get("percent"))
+                                if sig != last_sig:
+                                    last_sig = sig
+                                    await progress_callback(data)
+                                if data.get("status") in {"done", "error"}:
+                                    return
+                        except Exception:
+                            return
+                        await asyncio.sleep(1)
+
+                progress_task = asyncio.create_task(_poll_progress())
+            try:
+                with path.open("rb") as f:
+                    files = {"image": (path.name, f, mime)}
+                    resp = await client.post(edits_url, data=payload, files=files, headers=headers)
+            finally:
+                if progress_task:
+                    progress_task.cancel()
+                    try:
+                        await progress_task
+                    except asyncio.CancelledError:
+                        pass
+
+            if resp.status_code != 200:
+                error_text = resp.text[:500]
+                try:
+                    err_json = resp.json()
+                    err = err_json.get("error")
+                    error_text = (
+                        err.get("message", error_text)
+                        if isinstance(err, dict)
+                        else str(err or err_json.get("detail") or error_text)
+                    )
+                except Exception:
+                    pass
+                if resp.status_code in (400, 404, 405, 422):
+                    fallback = await _try_local_img2img_fallback(client)
+                    if fallback:
+                        return fallback
+                    if resp.status_code == 404:
+                        return {
+                            "error": (
+                                f"Image model '{model_id}' is reachable, but this endpoint does not expose image editing. "
+                                "Use it without an attached image for text-to-image generation, or serve an edit/img2img "
+                                "model for attached-image prompts."
+                            )
+                        }
+                return {"error": f"Image edit failed ({resp.status_code}): {error_text}"}
+
+            data = resp.json()
+            images = data.get("data", [])
+            if not images:
+                return {"error": "No image returned from edit API"}
+
+            img = images[0]
+            image_url = None
+            image_id = None
+
+            if img.get("b64_json"):
+                image_url, image_id = _save_image_bytes(base64.b64decode(img.get("b64_json")))
+            elif img.get("url"):
+                result_url = img["url"]
+                ok, reason = check_outbound_url(
+                    result_url,
+                    block_private=os.getenv("IMAGE_BLOCK_PRIVATE_IPS", "false").lower() == "true",
+                )
+                if not ok:
+                    return {"error": f"Image edit API returned unsafe image URL: {reason}"}
+                dl_resp = httpx.get(result_url, timeout=60)
+                if dl_resp.status_code != 200:
+                    return {"error": f"Could not download edited image ({dl_resp.status_code})"}
+                image_url, image_id = _save_image_bytes(dl_resp.content)
+            else:
+                return {"error": "Image edit API returned unexpected format (no b64_json or url)"}
+
+            return {
+                "results": f"Edited image for: {prompt[:100]}",
+                "image_url": image_url,
+                "image_id": image_id,
+                "image_prompt": prompt,
+                "image_model": model_id,
+                "image_size": size,
+                "image_quality": payload.get("quality", "medium"),
+            }
+    except httpx.TimeoutException:
+        return {"error": "Image edit timed out. The model may still be loading or overloaded."}
+    except Exception as e:
+        return {"error": f"Image edit error: {str(e)}"}
+
+
 # ---------------------------------------------------------------------------
 # Dispatcher (called from agent_tools.execute_tool_block)
 # ---------------------------------------------------------------------------

src/app_initializer.py

@@ -21,6 +21,7 @@ from src.model_discovery import ModelDiscovery
 from src.chat_handler import ChatHandler
 from src.research_handler import ResearchHandler
 from src.upload_handler import UploadHandler
+from src.tool_utils import set_upload_handler
 from src.search import update_search_config
 
 logger = logging.getLogger(__name__)
@@ -49,6 +50,8 @@ def initialize_managers(base_dir: str, rag_manager=None) -> Dict[str, Any]:
     session_manager = SessionManager(SESSIONS_FILE)
     set_session_manager(session_manager)  # Enable Session.add_message() persistence
     upload_handler = UploadHandler(base_dir, UPLOAD_DIR)
+    session_manager.upload_handler = upload_handler
+    set_upload_handler(upload_handler)
     personal_docs_manager = PersonalDocsManager(PERSONAL_DIR, rag_manager)
     api_key_manager = APIKeyManager(DATA_DIR)
     preset_manager = PresetManager(DATA_DIR)

src/attachment_refs.py

@@ -0,0 +1,164 @@
+"""Attachment reference helpers for chat storage and tool manifests.
+
+Live model calls may need provider-specific data URLs for the current turn.
+Persisted history and search indexes should keep stable upload references and
+human-readable text instead of duplicating raw media bytes.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from typing import Any, Iterable
+
+
+DATA_URL_RE = re.compile(
+    r"data:[^;,\s\"']+;base64,[A-Za-z0-9+/=]+",
+    re.IGNORECASE,
+)
+
+MEDIA_BLOCK_TYPES = {
+    "image",
+    "image_url",
+    "input_image",
+    "audio",
+    "input_audio",
+    "file",
+}
+
+
+def strip_inline_data_urls(text: str) -> str:
+    """Replace inline data URLs with a compact marker."""
+    if not isinstance(text, str) or ";base64," not in text:
+        return text
+    return DATA_URL_RE.sub("[inline media omitted from persisted history]", text)
+
+
+def attachment_ref(info: dict[str, Any]) -> dict[str, Any]:
+    """Return the stable attachment reference shape used outside raw uploads."""
+    upload_id = str(info.get("id") or info.get("attachment_id") or "").strip()
+    try:
+        size = int(info.get("size") or 0)
+    except (TypeError, ValueError):
+        size = 0
+    ref = {
+        "type": "attachment_ref",
+        "attachment_id": upload_id,
+        "name": info.get("name") or info.get("original_name") or upload_id,
+        "mime": info.get("mime") or "application/octet-stream",
+        "size": size,
+    }
+    checksum = info.get("checksum_sha256") or info.get("sha256") or info.get("hash")
+    if checksum:
+        ref["checksum_sha256"] = checksum
+    created_at = info.get("created_at") or info.get("uploaded_at")
+    if created_at:
+        ref["created_at"] = created_at
+    for key in ("width", "height", "vision", "vision_model", "gallery_id"):
+        value = info.get(key)
+        if value is not None:
+            ref[key] = value
+    return ref
+
+
+def attachment_refs_from_metadata(metadata: dict[str, Any] | None) -> list[dict[str, Any]]:
+    """Extract attachment refs from message metadata."""
+    attachments = (metadata or {}).get("attachments") or []
+    if not isinstance(attachments, list):
+        return []
+    refs: list[dict[str, Any]] = []
+    for item in attachments:
+        if isinstance(item, dict):
+            ref = attachment_ref(item)
+            if ref.get("attachment_id"):
+                refs.append(ref)
+    return refs
+
+
+def _ref_line(ref: dict[str, Any]) -> str:
+    parts = [f"Attachment: {ref.get('name') or ref.get('attachment_id') or 'upload'}"]
+    if ref.get("attachment_id"):
+        parts.append(f"id={ref['attachment_id']}")
+    if ref.get("mime"):
+        parts.append(f"mime={ref['mime']}")
+    if ref.get("size"):
+        parts.append(f"size={ref['size']} bytes")
+    if ref.get("checksum_sha256"):
+        parts.append(f"sha256={ref['checksum_sha256']}")
+    line = "[" + " | ".join(parts) + "]"
+    if ref.get("vision"):
+        line += f"\n[Attachment description: {str(ref['vision']).strip()}]"
+    return line
+
+
+def _text_from_blocks(blocks: Iterable[Any]) -> str:
+    lines: list[str] = []
+    omitted_media = 0
+    for block in blocks:
+        if isinstance(block, str):
+            lines.append(strip_inline_data_urls(block))
+            continue
+        if not isinstance(block, dict):
+            continue
+        block_type = block.get("type")
+        if block_type == "text":
+            text = block.get("text")
+            if isinstance(text, str) and text:
+                lines.append(strip_inline_data_urls(text))
+        elif block_type == "attachment_ref":
+            lines.append(_ref_line(block))
+        elif block_type in MEDIA_BLOCK_TYPES:
+            omitted_media += 1
+        else:
+            try:
+                encoded = json.dumps(block, ensure_ascii=True, sort_keys=True)
+            except TypeError:
+                encoded = str(block)
+            lines.append(strip_inline_data_urls(encoded))
+    if omitted_media:
+        plural = "s" if omitted_media != 1 else ""
+        lines.append(f"[{omitted_media} inline media payload{plural} omitted]")
+    return "\n".join(line for line in lines if line).strip()
+
+
+def persistable_message_content(
+    content: Any,
+    metadata: dict[str, Any] | None = None,
+) -> str:
+    """Return content safe for DB persistence and FTS indexing.
+
+    Multimodal provider blocks are collapsed to readable text plus stable
+    attachment reference lines from metadata. This avoids storing base64 media
+    in ``chat_messages.content`` while preserving enough context for reloads,
+    search, and later turns.
+    """
+    if isinstance(content, list):
+        text = _text_from_blocks(content)
+        refs = attachment_refs_from_metadata(metadata)
+        ref_lines = [_ref_line(ref) for ref in refs]
+        if ref_lines:
+            text = "\n".join([part for part in (text, "\n".join(ref_lines)) if part]).strip()
+        return text
+    if isinstance(content, str):
+        return strip_inline_data_urls(content)
+    try:
+        return strip_inline_data_urls(json.dumps(content, ensure_ascii=True, sort_keys=True))
+    except TypeError:
+        return strip_inline_data_urls(str(content))
+
+
+def search_index_text(content: Any) -> str:
+    """Best-effort searchable text for legacy stored content."""
+    if isinstance(content, str):
+        raw = content.strip()
+        if raw.startswith("[") and '"type"' in raw:
+            try:
+                parsed = json.loads(content)
+            except (TypeError, ValueError):
+                parsed = None
+            if isinstance(parsed, list):
+                return _text_from_blocks(parsed)
+        return strip_inline_data_urls(content)
+    if isinstance(content, list):
+        return _text_from_blocks(content)
+    return persistable_message_content(content)

src/builtin_actions.py

@@ -77,6 +77,7 @@ async def action_consolidate_memory(owner: str, **kwargs) -> Tuple[str, bool]:
     try:
         import json
         import re
+        from difflib import SequenceMatcher
         from src.constants import DATA_DIR
         from src.llm_core import llm_call_async_with_fallback
         from src.memory import MemoryManager
@@ -112,6 +113,64 @@ async def action_consolidate_memory(owner: str, **kwargs) -> Tuple[str, bool]:
         ai_reasons = []
         ai_used = False
 
+        def _normalized_memory_text(mem: dict) -> str:
+            text = (mem.get("text") or "").lower()
+            text = re.sub(r"[^a-z0-9@._+-]+", " ", text)
+            return " ".join(text.split())
+
+        def _memory_rank(mem: dict) -> tuple:
+            text = (mem.get("text") or "").strip()
+            return (
+                1 if mem.get("pinned") else 0,
+                1 if (mem.get("source") or "") == "user" else 0,
+                int(mem.get("uses") or 0),
+                -len(text),
+                int(mem.get("timestamp") or 0),
+            )
+
+        def _same_memory_fact(a: dict, b: dict) -> bool:
+            a_cat = (a.get("category") or "fact").strip().lower()
+            b_cat = (b.get("category") or "fact").strip().lower()
+            if a_cat != b_cat:
+                return False
+            a_text = _normalized_memory_text(a)
+            b_text = _normalized_memory_text(b)
+            if not a_text or not b_text:
+                return False
+            if a_text == b_text:
+                return True
+            shorter, longer = sorted((a_text, b_text), key=len)
+            if len(shorter) >= 24 and shorter in longer:
+                return True
+            return SequenceMatcher(None, a_text, b_text).ratio() >= 0.88
+
+        def _dedupe_group(group_memories: list) -> tuple[list, int]:
+            kept = []
+            removed = 0
+            for mem in group_memories:
+                text = (mem.get("text") or "").strip()
+                if not text:
+                    removed += 1
+                    if len(removed_examples) < 3:
+                        removed_examples.append("(empty)")
+                    continue
+                duplicate_idx = next(
+                    (idx for idx, kept_mem in enumerate(kept) if _same_memory_fact(mem, kept_mem)),
+                    None,
+                )
+                if duplicate_idx is None:
+                    kept.append(mem)
+                    continue
+                removed += 1
+                if _memory_rank(mem) > _memory_rank(kept[duplicate_idx]):
+                    if len(removed_examples) < 3:
+                        old_text = (kept[duplicate_idx].get("text") or "").strip()
+                        removed_examples.append(old_text[:60] + ("..." if len(old_text) > 60 else ""))
+                    kept[duplicate_idx] = mem
+                elif len(removed_examples) < 3:
+                    removed_examples.append(text[:60] + ("..." if len(text) > 60 else ""))
+            return kept, removed
+
         async def _try_ai_tidy_group(group_owner: str, group_memories: list) -> bool:
             nonlocal all_memories, total_removed, total_cleaned, total_scanned, ai_used
             if len(group_memories) < 2:
@@ -220,7 +279,6 @@ async def action_consolidate_memory(owner: str, **kwargs) -> Tuple[str, bool]:
                                 kept_all.append(mem)
 
                             removed = sum(1 for m in group_memories if m.get("id") in drop_ids)
-                            total_scanned += len(group_memories)
                             if removed or changed_text:
                                 all_memories = kept_all
                                 total_removed += removed
@@ -237,12 +295,23 @@ async def action_consolidate_memory(owner: str, **kwargs) -> Tuple[str, bool]:
             return False
 
         for group_owner, group_memories in memory_groups.items():
+            total_scanned += len(group_memories)
+            deduped_group, group_removed = _dedupe_group(group_memories)
+            if group_removed:
+                group_ref_ids = {id(m) for m in group_memories}
+                keep_ref_ids = {id(m) for m in deduped_group}
+                all_memories = [
+                    m for m in all_memories
+                    if id(m) not in group_ref_ids or id(m) in keep_ref_ids
+                ]
+                total_removed += group_removed
+                group_memories = deduped_group
+
             if await _try_ai_tidy_group(group_owner, group_memories):
                 continue
 
             seen = {}
             keep_refs = set()
-            total_scanned += len(group_memories)
             for mem in group_memories:
                 text = (mem.get("text") or "").strip()
                 key = " ".join(text.lower().split())
@@ -1125,14 +1194,14 @@ async def action_learn_sender_signatures(owner: str, **kwargs) -> Tuple[str, boo
             conn = _imap_connect(None, owner=owner)
             try:
                 conn.select("INBOX", readonly=True)
-                status, data = conn.search(None, "ALL")
+                status, data = conn.uid("SEARCH", None, "ALL")
                 if status != "OK" or not data or not data[0]:
                     return results
                 uids = data[0].split()[-300:][::-1]  # newest 300
                 for uid in uids:
                     try:
-                        st, msg_data = conn.fetch(
-                            uid, "(BODY.PEEK[HEADER.FIELDS (FROM)])"
+                        st, msg_data = conn.uid(
+                            "FETCH", uid, "(BODY.PEEK[HEADER.FIELDS (FROM)])"
                         )
                         if st != "OK" or not msg_data or not msg_data[0]:
                             continue
@@ -1214,7 +1283,7 @@ async def action_learn_sender_signatures(owner: str, **kwargs) -> Tuple[str, boo
                     conn2.select("INBOX", readonly=True)
                     for mm in _msgs:
                         try:
-                            st, data = conn2.fetch(mm["uid"], "(BODY.PEEK[TEXT])")
+                            st, data = conn2.uid("FETCH", mm["uid"], "(BODY.PEEK[TEXT])")
                             if st != "OK" or not data or not data[0]:
                                 continue
                             raw = data[0][1] if isinstance(data[0], tuple) else None
@@ -1356,13 +1425,13 @@ async def action_daily_brief(owner: str, **kwargs) -> Tuple[str, bool]:
             conn = _imap_connect(None)
             try:
                 conn.select("INBOX", readonly=True)
-                status, data = conn.search(None, "UNSEEN")
+                status, data = conn.uid("SEARCH", None, "UNSEEN")
                 uids = (data[0].split() if status == "OK" and data and data[0] else [])
                 unread_count = len(uids)
                 # Grab headers for the most recent 5 unread (UIDs increase with arrival)
                 for uid in uids[-5:][::-1]:
                     try:
-                        _, msg_data = conn.fetch(uid, "(BODY.PEEK[HEADER.FIELDS (FROM SUBJECT)])")
+                        _, msg_data = conn.uid("FETCH", uid, "(BODY.PEEK[HEADER.FIELDS (FROM SUBJECT)])")
                         if not msg_data or not msg_data[0]:
                             continue
                         hdr = msg_data[0][1] if isinstance(msg_data[0], tuple) else msg_data[0]

src/builtin_mcp.py

@@ -87,6 +87,7 @@ _BUILTIN_NPX_SERVERS = {
 
 # Global flag to disable MCP if there are compatibility issues
 MCP_DISABLED = os.environ.get("ODYSSEUS_DISABLE_MCP", "").lower() in ("1", "true", "yes")
+BROWSER_MCP_REQUIRE_CACHE = os.environ.get("ODYSSEUS_BROWSER_MCP_REQUIRE_CACHE", "").lower() in ("1", "true", "yes")
 
 
 # Strong references to the fire-and-forget startup tasks scheduled below.
@@ -103,6 +104,61 @@ def _spawn_bg(coro) -> asyncio.Task:
     task.add_done_callback(_BG_TASKS.discard)
     return task
 
+def _find_browser_executable() -> str:
+    """Find a browser binary for the built-in Playwright MCP server.
+
+    Docker images ship Debian's `chromium`; desktop installs may already have
+    Chrome/Chromium in a conventional location. If nothing is found, return an
+    empty string and let Playwright MCP use its own default browser/channel.
+    """
+    configured = os.environ.get("ODYSSEUS_BROWSER_EXECUTABLE", "").strip()
+    if configured:
+        return configured
+    for name in ("google-chrome", "chromium", "chromium-browser"):
+        path = shutil.which(name)
+        if path:
+            return path
+    for candidate in (
+        "/opt/google/chrome/chrome",
+        "/usr/bin/google-chrome",
+        "/usr/bin/chromium",
+        "/usr/bin/chromium-browser",
+    ):
+        if os.path.isfile(candidate):
+            return candidate
+    return ""
+
+
+def _browser_mcp_args(args: list[str]) -> list[str]:
+    """Return Playwright MCP args with a concrete browser executable when found."""
+    out = list(args or [])
+    if "--executable-path" not in out:
+        browser = _find_browser_executable()
+        if browser:
+            out.extend(["--executable-path", browser])
+    if os.environ.get("ODYSSEUS_BROWSER_ISOLATED", "1").lower() not in ("0", "false", "no"):
+        if "--isolated" not in out and "--user-data-dir" not in out:
+            out.append("--isolated")
+    if os.environ.get("ODYSSEUS_BROWSER_NO_SANDBOX", "1").lower() not in ("0", "false", "no"):
+        if "--no-sandbox" not in out and "--sandbox" not in out:
+            out.append("--no-sandbox")
+    return out
+
+
+def builtin_python_env(base_dir: str) -> dict[str, str]:
+    """Environment for built-in Python MCP subprocesses.
+
+    The app root must be importable so mcp_servers can import local modules, but
+    replacing PYTHONPATH entirely hides site-packages in container/dev launches
+    that rely on PYTHONPATH for their active environment.
+    """
+    existing = os.environ.get("PYTHONPATH", "")
+    parts = [base_dir]
+    for item in existing.split(os.pathsep):
+        if item and item not in parts:
+            parts.append(item)
+    return {"PYTHONPATH": os.pathsep.join(parts)}
+
 
 async def register_builtin_servers(mcp_manager):
     """Connect all built-in MCP servers to the manager."""
@@ -121,7 +177,7 @@ async def register_builtin_servers(mcp_manager):
                 transport="stdio",
                 command=python,
                 args=[script_path],
-                env={"PYTHONPATH": base_dir},
+                env=builtin_python_env(base_dir),
             )
             if ok:
                 logger.info(f"Built-in MCP server registered: {name}")
@@ -147,39 +203,44 @@ async def register_builtin_servers(mcp_manager):
     async def _start_npx_servers():
         await asyncio.sleep(3)  # let Python servers finish first
         for server_id, cfg in _BUILTIN_NPX_SERVERS.items():
-            # Skip the server if its npx package isn't cached. Without this
-            # check, npx would try to download/install the package on first
-            # use, which can take minutes (or hang) on fresh installs without
-            # Playwright system deps. Wrapping that in asyncio.wait_for to
-            # bound the wait sounds reasonable, but mcp.client.stdio uses an
-            # internal anyio task group that can't survive the resulting
-            # cross-task cancellation: it raises "Attempted to exit cancel
-            # scope in a different task than it was entered in" in a sibling
-            # task, which cascades cancellations into the rest of the event
-            # loop and downs the app. Detecting installed-state up-front lets
-            # us bail with a useful warning before we ever touch stdio_client.
-            args = cfg["args"]
+            # Browser automation is a shipped built-in, so the default path
+            # lets `npx -y` install @playwright/mcp on first start. Locked-down
+            # installs can opt back into the old no-network startup behavior
+            # with ODYSSEUS_BROWSER_MCP_REQUIRE_CACHE=1.
+            args = _browser_mcp_args(cfg["args"]) if server_id == "builtin_browser" else list(cfg["args"])
             pkg_spec = _npx_package_from_args(args)
-            if pkg_spec and not await _is_npx_package_cached(npx_path, pkg_spec):
+            if BROWSER_MCP_REQUIRE_CACHE and pkg_spec and not await _is_npx_package_cached(npx_path, pkg_spec):
                 logger.warning(
                     f"{cfg['name']} is not available.\n"
                     f"  Reason: npm package {pkg_spec!r} is not installed in the npx cache.\n"
                     f"  Impact: tools provided by this MCP server will be unavailable.\n"
                     f"  Fix:    {os.path.basename(npx_path)} -y {pkg_spec} --version\n"
                     f"          (run once, then restart Odysseus)\n"
-                    f"  Notes:  this server is optional; see README.md "
-                    f"'Built-in MCP servers' for details."
+                    f"  Notes:  ODYSSEUS_BROWSER_MCP_REQUIRE_CACHE=1 is set, "
+                    f"so Odysseus will not install browser automation on startup."
                 )
                 continue
 
             logger.info(f"Starting NPX server: {cfg['name']} ({npx_path} {' '.join(args)})")
             try:
+                env = None
+                if server_id == "builtin_browser":
+                    cache_home = os.environ.get(
+                        "ODYSSEUS_BROWSER_MCP_CACHE",
+                        os.path.join(base_dir, "data", "local", "playwright-mcp-cache"),
+                    )
+                    os.makedirs(cache_home, exist_ok=True)
+                    env = {
+                        "XDG_CACHE_HOME": cache_home,
+                        "PLAYWRIGHT_BROWSERS_PATH": os.path.join(cache_home, "browsers"),
+                    }
                 ok = await mcp_manager.connect_server(
                     server_id=server_id,
                     name=cfg["name"],
                     transport="stdio",
                     command=npx_path,
                     args=args,
+                    env=env,
                 )
                 if ok:
                     logger.info(f"Built-in NPX server registered: {cfg['name']}")