Merge verified Odysseus fixes

2026-08-06 13:38:40 -04:00 · 2026-07-23 14:49:08 +00:00
parent 93107c5415
commit cf4e240ad1
246 changed files with 28636 additions and 5493 deletions
@@ -0,0 +1,85 @@
+# Attachment References and Upload Storage
+
+Odysseus stores uploaded bytes once under the configured upload directory and
+passes stable references through chat history, tools, and future artifact work.
+The goal is to avoid duplicating large inline media payloads in
+`chat_messages.content` or the SQLite FTS index.
+
+## Reference Shape
+
+Attachment references use this minimum shape:
+
+```json
+{
+  "type": "attachment_ref",
+  "attachment_id": "32hex-or-32hex.ext",
+  "name": "original-filename.png",
+  "mime": "image/png",
+  "size": 12345,
+  "checksum_sha256": "hex-digest",
+  "created_at": "2026-07-09T12:00:00"
+}
+```
+
+Optional fields such as `width`, `height`, `vision`, `vision_model`, and
+`gallery_id` may be present when the uploader or preprocessing path knows them.
+
+## Persistence
+
+The live model call may still receive provider-specific multimodal blocks for
+the current turn. Persistence is different:
+
+- `chat_messages.content` stores readable text plus compact attachment reference
+  lines, never raw `data:*;base64,...` upload bytes.
+- `chat_messages.metadata.attachments` stores structured attachment reference
+  metadata for UI reloads and future processing.
+- The SQLite FTS migration recreates chat-message FTS triggers so new rows do
+  not index inline media payloads, and it scrubs legacy rows that were already
+  indexed with data URLs.
+
+## Tool Access
+
+Agent/tool context receives upload entries as `attachment_ref` manifests with an
+`odysseus://attachment/<id>` URI and `read_policy: "owner_checked_upload"`.
+
+For compatibility with existing built-in tools, a local `path` may be included
+only after all of these checks pass:
+
+- the upload ID resolves through `UploadHandler.resolve_upload`;
+- the requested owner is allowed to read the upload;
+- the file remains inside the configured upload directory;
+- the file path is inside the tool-readable roots.
+
+External MCP/custom tools should treat the URI and attachment ID as the stable
+contract and request bytes through an owner-checked server path, not by assuming
+host filesystem layout.
+
+## Retention and Deletion
+
+Current retention behavior is conservative:
+
+- uploads are indexed in `uploads.json` with owner, checksum, MIME type, size,
+  and creation time;
+- admin cleanup first scans persisted chat metadata/content, document versions,
+  PDF source markers, gallery hashes, notes, and calendar records for live
+  references;
+- cleanup fails closed if that reference scan cannot complete, and the lower-level
+  cleanup API removes nothing unless it receives a complete reference snapshot;
+- expired, unreferenced uploads are removed during the completed scan, while
+  attachment-bearing writers must first take an owner-checked reservation that
+  serializes with deletion and refreshes the upload's access timestamp;
+- deliberate removal atomically drops matching `uploads.json` rows before deleting
+  the bytes and restores those rows if filesystem removal fails;
+- deleting a chat removes the chat rows but does not immediately delete shared
+  upload bytes, because the same upload may also be referenced by gallery items,
+  documents, duplicate-upload rows, or future artifact records.
+
+There is no distinct artifact table in the current schema. Artifact-like upload
+references persisted in chat or document text are covered by the canonical
+attachment-ID scan; any future artifact store must be added to reference discovery
+before cleanup is allowed to consider its uploads unreferenced.
+
+Cleanup and write reservations share the upload-index lock. This closes the
+scan/write/delete race in the documented single-worker deployment; a future
+multi-process deployment must add an inter-process lock or move lifecycle state
+into the database before enabling destructive cleanup in more than one worker.
@@ -409,7 +409,7 @@
        <a href="#testimonials">Testimonials</a>
        <a href="#how">How it started</a>
        <a href="#start">Get started</a>
-        <a class="btn" href="https://github.com/pewdiepie-archdaemon/odysseus" target="_blank">
+        <a class="btn" href="https://github.com/odysseus-dev/odysseus" target="_blank">
          <svg width="16" height="16" viewBox="0 0 24 24" fill="currentColor"><path d="M12 .5C5.7.5.5 5.7.5 12c0 5.1 3.3 9.4 7.9 10.9.6.1.8-.2.8-.6v-2c-3.2.7-3.9-1.5-3.9-1.5-.5-1.3-1.3-1.7-1.3-1.7-1-.7.1-.7.1-.7 1.2.1 1.8 1.2 1.8 1.2 1 1.8 2.7 1.3 3.4 1 .1-.8.4-1.3.7-1.6-2.6-.3-5.3-1.3-5.3-5.7 0-1.3.5-2.3 1.2-3.1-.1-.3-.5-1.5.1-3.1 0 0 1-.3 3.3 1.2a11.5 11.5 0 0 1 6 0C17.3 4.7 18.3 5 18.3 5c.6 1.6.2 2.8.1 3.1.8.8 1.2 1.8 1.2 3.1 0 4.4-2.7 5.4-5.3 5.7.4.4.8 1.1.8 2.2v3.3c0 .4.2.7.8.6 4.6-1.5 7.9-5.8 7.9-10.9C23.5 5.7 18.3.5 12 .5z"/></svg>
          GitHub
        </a>
@@ -437,7 +437,7 @@
      </p>
      <div class="hero-cta">
        <a class="btn primary" href="#start">Get started</a>
-        <a class="btn" href="https://github.com/pewdiepie-archdaemon/odysseus" target="_blank">View on GitHub</a>
+        <a class="btn" href="https://github.com/odysseus-dev/odysseus" target="_blank">View on GitHub</a>
      </div>

    </div>
@@ -678,9 +678,9 @@
        <div class="eyebrow"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M4 17l6-5-6-5"/><path d="M12 19h8"/></svg>Get started</div>
        <h2 class="h" style="margin-bottom:6px;">Odysseus is yours.</h2>
        <p class="sub center" style="margin:0 auto;">It's open source and free. No sales team, no demo request, no Trojan horse.</p>
-        <div class="codeblock"><span><span class="prompt">$</span> git clone https://github.com/pewdiepie-archdaemon/odysseus.git &amp;&amp; cd odysseus</span><button class="copy-btn" data-copy="git clone https://github.com/pewdiepie-archdaemon/odysseus.git && cd odysseus">Copy</button></div>
+        <div class="codeblock"><span><span class="prompt">$</span> git clone https://github.com/odysseus-dev/odysseus.git &amp;&amp; cd odysseus</span><button class="copy-btn" data-copy="git clone https://github.com/odysseus-dev/odysseus.git && cd odysseus">Copy</button></div>
        <div>
-          <a class="btn primary" href="https://github.com/pewdiepie-archdaemon/odysseus" target="_blank" style="margin-top:14px;">View on GitHub</a>
+          <a class="btn primary" href="https://github.com/odysseus-dev/odysseus" target="_blank" style="margin-top:14px;">View on GitHub</a>
        </div>
        <div class="pill-row">
          <span class="pill">Self-hosted</span>
@@ -7,10 +7,9 @@ benefit.

 ## What runs, and why

-Most checks live in files under `.github/workflows/`. CodeQL is configured
-through GitHub's code scanning default setup, so it appears as a dynamic GitHub
-workflow instead of a checked-in workflow file. They run automatically; you do
-not start them.
+Most checks live in files under `.github/workflows/`. CodeQL uses the
+checked-in advanced configuration in `.github/workflows/codeql.yml`. They run
+automatically; you do not start them.

 | Check | What it protects against | Blocks a merge? |
 |---|---|---|
@@ -90,14 +89,9 @@ let the workflows run on one pull request first, then add them here.
 2. Turn on **Dependency graph** (usually on by default for public repos) -- this
   powers Dependency review and Dependabot.
 3. Turn on **Dependabot alerts** and **Dependabot security updates**.
-4. Under **Code scanning**, use **Set up -> Default** for CodeQL. GitHub then
-   runs CodeQL as a dynamic workflow without the fork-token limitations that
-   affect checked-in advanced workflows.
-
-   Do not also add a checked-in CodeQL workflow while default setup is enabled:
-   GitHub rejects advanced CodeQL uploads when default setup is active. If the
-   project later needs an advanced CodeQL workflow, disable default setup first
-   and keep only one CodeQL publishing path active.
+4. Under **Code scanning**, keep **Default setup** disabled. CodeQL is
+   configured by `.github/workflows/codeql.yml`; enabling default setup at the
+   same time causes GitHub to reject uploads from the checked-in workflow.

 ## Keeping it current

@@ -4,7 +4,7 @@ This page keeps the detailed install, deployment, troubleshooting, and configura

 ## Quick Start

-> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main).
+> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/odysseus-dev/odysseus/tree/main).

 Defaults work out of the box: clone, run, then configure models/search/email
 inside **Settings**. Only edit `.env` for deployment-level overrides like
@@ -20,7 +20,7 @@ pull request guidelines.

 ### Docker (recommended)
 ```bash
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 cp .env.example .env       # optional, but recommended for explicit defaults
 docker compose up -d --build
@@ -38,7 +38,7 @@ only when you intentionally want LAN/reverse-proxy access.

 ### Native Linux / macOS
 ```bash
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 python3 -m venv venv
 source venv/bin/activate
@@ -56,7 +56,7 @@ Docker on macOS cannot use the Metal GPU. For GPU-accelerated Cookbook on an
 M-series Mac, run Odysseus natively:

 ```bash
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 ./start-macos.sh
 ```
@@ -150,6 +150,79 @@ scripts/check-docker-gpu.sh --enable-nvidia-overlay
 # Full assisted setup — install toolkit, then enable overlay if passthrough works:
 scripts/check-docker-gpu.sh --install-nvidia-toolkit --enable-nvidia-overlay
 ```
+#### Arch Linux NVIDIA Docker notes
+
+On Arch Linux, verify the host NVIDIA driver and Docker GPU passthrough before enabling the Odysseus NVIDIA overlay.
+
+Install the required packages:
+
+```bash
+sudo pacman -Syu
+sudo pacman -S docker docker-compose nvidia-container-toolkit nvidia-utils
+sudo systemctl enable --now docker
+```
+
+Configure Docker to use the NVIDIA container runtime:
+
+```bash
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+```
+
+Verify the host GPU:
+
+```bash
+nvidia-smi
+```
+
+Verify Docker GPU passthrough:
+
+```bash
+docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi
+```
+
+Then enable the Odysseus NVIDIA compose overlay:
+
+```env
+COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
+```
+
+Rebuild and verify the GPU inside the Odysseus container:
+
+```bash
+docker compose up -d --build
+docker compose exec odysseus nvidia-smi -L
+```
+
+For first-time local model testing on 8 GB laptop GPUs, start with GGUF/Q4 models on llama.cpp before trying GPTQ/AWQ models on vLLM or SGLang. This keeps the first run simpler while confirming GPU passthrough works.
+
+**WSL2 + snap Docker.** If the NVIDIA check fails with this error, Docker may be
+installed via snap:
+
+```text
+failed to fulfil mount request: open /usr/lib/wsl/lib/libdxcore.so: no such file or directory
+```
+
+Check with `snap list docker` or:
+
+```bash
+docker info --format '{{.DockerRootDir}}'
+```
+
+A Docker root under `/var/snap/docker/` means snap confinement can prevent
+Docker from seeing WSL2's `/usr/lib/wsl/lib` GPU libraries even when the files
+exist on the host. Reinstalling or reconfiguring `nvidia-container-toolkit` will
+not fix that. Remove snap Docker, install the official apt-based Docker Engine
+([Docker docs](https://docs.docker.com/engine/install/ubuntu/)), then configure
+the NVIDIA runtime again:
+
+```bash
+sudo snap remove docker
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+```
+
+Then re-run `scripts/check-docker-gpu.sh`.

 Safety notes:
 - The app never installs host GPU runtime automatically.
@@ -257,7 +330,7 @@ do not run on macOS. MLX-only models are not served by Odysseus.
 server; safe to re-run):

 ```powershell
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
 ```
@@ -265,7 +338,7 @@ powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
 Or do it by hand:

 ```powershell
-git clone https://github.com/pewdiepie-archdaemon/odysseus.git
+git clone https://github.com/odysseus-dev/odysseus.git
 cd odysseus
 py -3.11 -m venv venv
 venv\Scripts\Activate.ps1