Compare commits
29 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 68f19a889a | |||
| 422f23fb12 | |||
| 0f966d6b9f | |||
| 7b09491557 | |||
| fafaf089c5 | |||
| b58af4267b | |||
| 8ff76f083c | |||
| 2196869c86 | |||
| dd2e23c9af | |||
| facc50cb0f | |||
| 074a1e6eff | |||
| 2fab378c6a | |||
| 5bafc30622 | |||
| d6d2e17214 | |||
| f4e8990635 | |||
| fc3a5e555e | |||
| 270b8570fc | |||
| 0750486654 | |||
| d38e2cbc07 | |||
| 7fd937fa57 | |||
| c41caac438 | |||
| 1747c13133 | |||
| ffd0aaf69b | |||
| 81e7074d93 | |||
| f66a23d19d | |||
| f602819523 | |||
| 85a773ea02 | |||
| fb0a64fe4f | |||
| bcf46dafb9 |
@@ -1,476 +1,65 @@
|
||||
# Odysseus
|
||||
<p align="center">
|
||||
<img src="docs/odysseus-wordmark.png" alt="Odysseus" width="280">
|
||||
</p>
|
||||
|
||||
> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main).
|
||||
<p align="center">
|
||||
A self-hosted AI workspace for chat, agents, research, documents, email, notes, calendar, and local model workflows.
|
||||
</p>
|
||||
|
||||
```
|
||||
───────────────────────────────────────────────
|
||||
⊹ ࣪ ˖ ૮( ˶ᵔ ᵕ ᵔ˶ )っ Odysseus vers. 1.0
|
||||
───────────────────────────────────────────────
|
||||
```
|
||||
<p align="center">
|
||||
<a href="#quick-start">Quick Start</a> ·
|
||||
<a href="docs/setup.md">Setup Guide</a> ·
|
||||
<a href="CONTRIBUTING.md">Contributing</a> ·
|
||||
<a href="ROADMAP.md">Roadmap</a>
|
||||
</p>
|
||||
|
||||

|
||||
<p align="center">
|
||||
<a href="https://repology.org/project/odysseus-ai/versions"><img src="https://repology.org/badge/vertical-allrepos/odysseus-ai.svg" alt="Packaging status"></a>
|
||||
</p>
|
||||
|
||||
A self-hosted AI workspace -- meant to be the self-hosted version of the UI experience you get from ChatGPT and Claude. But with more jank and fun. Running on your own hardware, with your own data -- local-first, privacy-first, and no trojan.
|
||||
<p align="center">
|
||||
<img src="docs/odysseus.jpg" alt="Odysseus interface">
|
||||
</p>
|
||||
|
||||
[](https://repology.org/project/odysseus-ai/versions)
|
||||
|
||||
## Features
|
||||
- **Chat** -- chat with any local model or API; adding them is super simple.<br> <sub>vLLM · llama.cpp · Ollama · OpenRouter · OpenAI · GitHub Copilot</sub>
|
||||
- **Agent** -- hand it tools and let it run the whole task itself.<br> <sub>built on [opencode](https://github.com/anomalyco/opencode) · MCP · web · files · shell · skills · memory</sub>
|
||||
- **Cookbook** -- Scans your hardware, recommends models, click to download and serve.. easy!<br> <sub>built on [llmfit](https://github.com/AlexsJones/llmfit) · VRAM-aware · GGUF / FP8 / AWQ · fit scoring · vLLM / llama.cpp serving</sub>
|
||||
- **Deep Research** -- multi-step runs that gather, read, and synthesize sources into a nice visual report.<br> <sub>adapted from [Tongyi DeepResearch](https://github.com/Alibaba-NLP/DeepResearch)</sub>
|
||||
- **Compare** -- a fun tool to compare models side by side. Test completely blind, no bias!<br> <sub>multi-model · blind test · synthesis</sub>
|
||||
- **Documents** -- YOU write the text, AI is there to assist, not the opposite.<br> <sub>multi-tab editor · markdown · HTML · CSV · syntax highlighting · AI edits · suggestions</sub>
|
||||
- **Memory / Skills** -- Persistent memory and skills, your agent evolves over time as it better understands you and your tasks!<br> <sub>ChromaDB · fastembed (ONNX) · vector + keyword retrieval · import/export</sub>
|
||||
- **Email** -- IMAP/SMTP inbox with AI triage built in: urgency reminders, auto-tag, auto-summary, auto-reply drafts, auto-spam.<br> <sub>IMAP · SMTP · per-account routing · CalDAV-aware</sub>
|
||||
- **Notes & Tasks** -- Quick notes with reminders, a todo list, and scheduled tasks the agent can act on.<br> <sub>note pings · checklist · cron-style tasks · ntfy / browser / email channels</sub>
|
||||
- **Calendar** -- Local-first calendar with CalDAV sync to Radicale / Nextcloud / Apple / Fastmail.<br> <sub>CalDAV pull · .ics import/export · per-calendar colors · agent-aware</sub>
|
||||
- **Works on mobile** -- looks and runs great on your phone, not just desktop.<br> <sub>responsive · installable (PWA) · touch gestures</sub>
|
||||
- **Extras** -- more to explore, happy if you give it a go!<br> <sub>image editor · theme editor · file uploads (vision + PDF) · web search · presets · sessions · 2FA</sub>
|
||||
|
||||
## Demo
|
||||
A full, hover-to-play tour lives on the landing page (`docs/index.html`).
|
||||
|
||||
<details>
|
||||
<summary>Screenshots / clips</summary>
|
||||
|
||||
### Chat & Agents
|
||||

|
||||
### Deep Research
|
||||

|
||||
### Compare
|
||||

|
||||
### Documents
|
||||

|
||||
### Notes & Tasks
|
||||

|
||||
|
||||
</details>
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
Defaults work out of the box: clone, run, then configure models/search/email
|
||||
inside **Settings**. Only edit `.env` for deployment-level overrides like
|
||||
`APP_BIND`, `APP_PORT`, `AUTH_ENABLED`, `DATABASE_URL`, or a pre-seeded admin password.
|
||||
> `dev` is the default branch and gets the newest changes first. Use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main) if you want the more curated branch.
|
||||
|
||||
On first setup, Odysseus creates an admin account (`admin` unless
|
||||
`ODYSSEUS_ADMIN_USER` is set) and prints a temporary password in the terminal.
|
||||
For Docker installs, the same line is in `docker compose logs odysseus`.
|
||||
Use that for the first login, then change it in **Settings**.
|
||||
|
||||
Contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, testing, and
|
||||
pull request guidelines.
|
||||
|
||||
### Docker (recommended)
|
||||
```bash
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
cp .env.example .env # optional, but recommended for explicit defaults
|
||||
cp .env.example .env
|
||||
docker compose up -d --build
|
||||
```
|
||||
To include optional extras in the image (PDF viewer, Office extraction; includes AGPL PyMuPDF), build with `docker compose build --build-arg INSTALL_OPTIONAL=true` before `up`.
|
||||
|
||||
Open `http://localhost:7000` when the containers are healthy. Docker Compose
|
||||
binds the web UI to `127.0.0.1` by default. If the port is taken, set
|
||||
`APP_PORT=7001` in `.env` and recreate the container. Set `APP_BIND=0.0.0.0`
|
||||
only when you intentionally want LAN/reverse-proxy access.
|
||||
Open `http://localhost:7000` when the containers are healthy. The first admin password is printed in `docker compose logs odysseus`.
|
||||
|
||||
> **On Apple Silicon (M-series) Macs:** Docker can't reach the Metal GPU, so
|
||||
> Cookbook serves local models on CPU only. For GPU-accelerated model serving,
|
||||
> run natively instead — see [Apple Silicon](#apple-silicon) below.
|
||||
Native installs, GPU notes, Windows/macOS instructions, HTTPS, and configuration live in the [setup guide](docs/setup.md).
|
||||
|
||||
### Native Linux / macOS
|
||||
```bash
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
python setup.py
|
||||
python -m uvicorn app:app --host 127.0.0.1 --port 7000
|
||||
```
|
||||
Requirements: Python 3.11+. Cookbook also needs `tmux` for background model
|
||||
downloads and serves. The app itself is lightweight; local model serving is the
|
||||
heavy part and depends on the model, runtime, GPU, and VRAM, so small hosts can
|
||||
connect to API or remote model servers instead. Use `--host 0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
|
||||
## Features
|
||||
|
||||
### Apple Silicon
|
||||
Docker on macOS cannot use the Metal GPU. For GPU-accelerated Cookbook on an
|
||||
M-series Mac, run Odysseus natively:
|
||||
- **Chat + Agents** — local/API models, tools, MCP, files, shell, skills, and memory.
|
||||
- **Cookbook** — hardware-aware model recommendations, downloads, and serving.
|
||||
- **Deep Research** — multi-step web research with source reading and report generation.
|
||||
- **Compare** — blind side-by-side model testing and synthesis.
|
||||
- **Documents** — writing-first editor with AI edits, suggestions, Markdown, HTML, CSV, and syntax highlighting.
|
||||
- **Email** — IMAP/SMTP inbox with triage, tags, summaries, reminders, and reply drafts.
|
||||
- **Notes, Tasks + Calendar** — reminders, todos, scheduled agent tasks, and CalDAV sync.
|
||||
- **Extras** — gallery/image editor, themes, uploads, web search, presets, sessions, and 2FA.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
./start-macos.sh
|
||||
```
|
||||
## Demo
|
||||
|
||||
It launches at `http://127.0.0.1:7860`. To expose it to your phone over a trusted LAN/VPN such as Tailscale, bind all interfaces:
|
||||
|
||||
```bash
|
||||
ODYSSEUS_HOST=0.0.0.0 ./start-macos.sh
|
||||
# then open http://<tailscale-ip>:7860
|
||||
```
|
||||
|
||||
The script also reads `.env` at startup, so `APP_BIND=0.0.0.0` and `APP_PORT`
|
||||
set there are picked up automatically without a command-line override each run.
|
||||
|
||||
Keep `AUTH_ENABLED=true` (the default) before binding outside loopback. Do not
|
||||
expose this port directly to the public internet. To build a clickable app wrapper:
|
||||
|
||||
```bash
|
||||
./build-macos-app.sh
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Cookbook, GPU, Ollama, and troubleshooting notes</summary>
|
||||
|
||||
**Docker bundled services.** Compose starts Odysseus, ChromaDB, SearXNG, and
|
||||
ntfy. Odysseus and the bundled service ports bind to `127.0.0.1` by default, so
|
||||
they are reachable from the host but not exposed to your LAN/public internet
|
||||
unless you opt in.
|
||||
|
||||
**Cookbook storage in Docker.** Downloads live in `./data/huggingface`
|
||||
(`~/.cache/huggingface` in the container). Cookbook-installed Python CLIs and
|
||||
serve engines live in `./data/local` (`~/.local` in the container), so they
|
||||
survive container recreation.
|
||||
|
||||
**Remote servers.** In **Cookbook -> Settings -> Servers**, generate the
|
||||
Odysseus SSH key and add the public key to the remote server's
|
||||
`~/.ssh/authorized_keys`. From the host you can also run:
|
||||
|
||||
```bash
|
||||
ssh-copy-id -i data/ssh/id_ed25519.pub user@server
|
||||
```
|
||||
|
||||
**Docker GPU overlays.** CPU-only users can skip this section. Cookbook can
|
||||
only detect GPUs that Docker exposes to the container — if the host runtime or
|
||||
device passthrough is not configured, Cookbook sees the iGPU, another card, or
|
||||
CPU instead of your intended GPU.
|
||||
|
||||
For NVIDIA, `scripts/check-docker-gpu.sh` diagnoses GPU passthrough and can
|
||||
optionally install the host runtime or update `.env`.
|
||||
|
||||
```bash
|
||||
# Read-only diagnostic (default — installs nothing, never edits .env):
|
||||
scripts/check-docker-gpu.sh
|
||||
|
||||
# Print OS-specific install commands without running them:
|
||||
scripts/check-docker-gpu.sh --print-install-commands
|
||||
|
||||
# Install NVIDIA Container Toolkit on Ubuntu/Debian (requires sudo):
|
||||
scripts/check-docker-gpu.sh --install-nvidia-toolkit
|
||||
|
||||
# Write COMPOSE_FILE to .env (only when GPU passthrough is confirmed working):
|
||||
scripts/check-docker-gpu.sh --enable-nvidia-overlay
|
||||
|
||||
# Full assisted setup — install toolkit, then enable overlay if passthrough works:
|
||||
scripts/check-docker-gpu.sh --install-nvidia-toolkit --enable-nvidia-overlay
|
||||
```
|
||||
|
||||
Safety notes:
|
||||
- The app never installs host GPU runtime automatically.
|
||||
- The app never edits `.env` automatically.
|
||||
- `.env` is only modified when `--enable-nvidia-overlay` is explicitly passed,
|
||||
and only after GPU passthrough succeeds. `--yes` skips prompts but does not
|
||||
bypass the passthrough gate.
|
||||
- `.env.bak.*` backups created by `--enable-nvidia-overlay` are ignored by
|
||||
Git and the Docker build context.
|
||||
|
||||
To enable manually without the script, add this to `.env`:
|
||||
|
||||
```bash
|
||||
COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
|
||||
```
|
||||
|
||||
**AMD / ROCm.** AMD setup is read-only diagnostic plus manual `.env` edit. Run:
|
||||
|
||||
```bash
|
||||
scripts/check-docker-amd-gpu.sh
|
||||
```
|
||||
|
||||
Then add the reported values to `.env`, replacing `RENDER_GID` with your host's
|
||||
numeric render group id:
|
||||
|
||||
```bash
|
||||
COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
|
||||
RENDER_GID=989
|
||||
```
|
||||
|
||||
For NVIDIA/AMD GPU support, also read the comments in the selected overlay file: docker/gpu.nvidia.yml or docker/gpu.amd.yml.
|
||||
|
||||
**Stack-management UIs (Portainer, Coolify, Dockhand, etc.).** These tools
|
||||
often accept only a single Compose file and do not reliably honor `COMPOSE_FILE`
|
||||
or multiple `-f` overlays. CLI users should keep using the `COMPOSE_FILE`
|
||||
overlay workflow above. For stack UIs, point the stack at one of the standalone
|
||||
files instead, which bundle the base stack plus the GPU settings:
|
||||
|
||||
- `docker-compose.gpu-nvidia.yml` — still requires the NVIDIA Container Toolkit
|
||||
on the host.
|
||||
- `docker-compose.gpu-amd.yml` — still requires host ROCm/kfd/DRI setup, the
|
||||
`video`/`render` group membership, and `RENDER_GID` when needed.
|
||||
|
||||
The base `docker-compose.yml` plus the `docker/gpu.*.yml` overlays remain the
|
||||
source of truth; the standalone files mirror them for single-file deployments.
|
||||
|
||||
Verify after enabling either overlay:
|
||||
|
||||
```bash
|
||||
docker compose exec odysseus nvidia-smi -L # NVIDIA
|
||||
docker compose exec odysseus sh -lc 'test -e /dev/kfd && test -d /dev/dri && ls -l /dev/kfd /dev/dri/renderD*' # AMD
|
||||
```
|
||||
|
||||
> **GPU passthrough ≠ llama.cpp CUDA.** `nvidia-smi` passing inside the
|
||||
> container confirms Docker GPU access, but llama.cpp also needs `cudart` and
|
||||
> the CUDA Toolkit at runtime. If Cookbook logs show `Unable to find cudart
|
||||
> library`, `Could NOT find CUDAToolkit`, `CUDA Toolkit not found`, or
|
||||
> tensors/layers assigned to CPU, that is a Cookbook/llama.cpp build issue —
|
||||
> not a Docker passthrough failure. Reinstall the serve engine via
|
||||
> **Cookbook → Dependencies** to get a CUDA-enabled build.
|
||||
>
|
||||
> The same split applies to AMD/ROCm: seeing `/dev/kfd` and `/dev/dri` inside
|
||||
> the container confirms device passthrough, not ROCm userspace or a
|
||||
> ROCm-enabled vLLM/llama.cpp build. `rocm-smi` and `rocminfo` are not expected
|
||||
> inside the slim Odysseus image.
|
||||
|
||||
**Ollama with Docker.** If Ollama runs on the host, add this endpoint in
|
||||
Settings:
|
||||
|
||||
```text
|
||||
http://host.docker.internal:11434/v1
|
||||
```
|
||||
|
||||
Ollama must listen outside its own loopback interface:
|
||||
|
||||
```bash
|
||||
OLLAMA_HOST=0.0.0.0:11434 ollama serve
|
||||
```
|
||||
|
||||
This connects Odysseus in Docker to an Ollama server that is already running on
|
||||
your host machine; it does not start Ollama inside the container.
|
||||
`host.docker.internal` is Docker's hostname for the host machine from inside the
|
||||
container. Cookbook **Serve** is a separate workflow for serving downloaded
|
||||
models through Odysseus/llama.cpp, so Windows users with an existing Ollama
|
||||
install usually only need to add the endpoint in Settings.
|
||||
|
||||
**Useful checks.**
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
docker compose logs --tail=120 odysseus
|
||||
docker compose logs odysseus | grep -E 'ChromaDB|MemoryVectorStore|DEGRADED'
|
||||
```
|
||||
|
||||
**macOS details.** `start-macos.sh` installs Homebrew deps, creates the venv,
|
||||
runs setup, and starts uvicorn on port `7860` because AirPlay often holds
|
||||
`7000`. It uses llama.cpp/Ollama for Metal. vLLM/SGLang are CUDA/ROCm-only and
|
||||
do not run on macOS. MLX-only models are not served by Odysseus.
|
||||
|
||||
</details>
|
||||
|
||||
### Native Windows
|
||||
|
||||
**One-command launcher** (creates the venv, installs deps, runs setup, starts the
|
||||
server; safe to re-run):
|
||||
|
||||
```powershell
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
|
||||
```
|
||||
|
||||
Or do it by hand:
|
||||
|
||||
```powershell
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
py -3.11 -m venv venv
|
||||
venv\Scripts\Activate.ps1
|
||||
pip install -r requirements.txt
|
||||
python setup.py
|
||||
python -m uvicorn app:app --host 127.0.0.1 --port 7000
|
||||
```
|
||||
|
||||
If `python` points at an older interpreter, use `py -3.12` (or another installed
|
||||
3.11+ version) for the venv step.
|
||||
|
||||
**Requirements:** Python 3.11+. The core app (chat, agent, memory, documents,
|
||||
email, calendar, deep research) runs fully native. For full **Cookbook** background
|
||||
model downloads and the agent shell tool, also install
|
||||
[Git for Windows](https://git-scm.com/download/win) (provides `bash.exe`).
|
||||
Local GPU *serving* of vLLM/SGLang needs Linux/WSL2; for a local model on Windows,
|
||||
[Ollama](https://ollama.com/download) is the easiest path — point Odysseus at
|
||||
`http://localhost:11434/v1` in Settings.
|
||||
|
||||
Open `http://localhost:7000`, log in with the generated admin password,
|
||||
and configure everything else inside **Settings**.
|
||||
|
||||
## Troubleshooting & Advanced Setup
|
||||
|
||||
### `chromadb-client` conflicts with embedded ChromaDB
|
||||
If `chromadb-client` (the lightweight HTTP-only package) is installed alongside the full `chromadb` package, Odysseus starts but ChromaDB silently falls back to HTTP-only mode and fails.
|
||||
|
||||
**Fix:** uninstall `chromadb-client` and force-reinstall the full package:
|
||||
```bash
|
||||
./venv/bin/pip uninstall chromadb-client -y
|
||||
./venv/bin/pip install --force-reinstall chromadb
|
||||
```
|
||||
|
||||
### HTTPS + LAN/Tailscale exposure
|
||||
To expose Odysseus on a local network or Tailscale with HTTPS:
|
||||
1. Change the bind address to `0.0.0.0` in `.env` (`APP_BIND=0.0.0.0` or `ODYSSEUS_HOST=0.0.0.0`).
|
||||
2. Generate a locally-trusted cert for your LAN/Tailscale IPs using [mkcert](https://github.com/FiloSottile/mkcert):
|
||||
```bash
|
||||
mkcert -install
|
||||
mkcert -cert-file cert.pem -key-file key.pem 192.168.1.100 tailscale-ip
|
||||
```
|
||||
3. Run `uvicorn` with the generated certs:
|
||||
```bash
|
||||
python -m uvicorn app:app --host 0.0.0.0 --port 7000 --ssl-certfile=cert.pem --ssl-keyfile=key.pem
|
||||
```
|
||||
4. Install the `mkcert` CA on any other device you want to access Odysseus from (e.g., for iOS, email the `rootCA.pem` to yourself, install the profile, and trust it in Certificate Trust Settings).
|
||||
|
||||
### Optional Dependencies
|
||||
`requirements-optional.txt` contains packages that unlock extra features. It is not installed by default.
|
||||
|
||||
| Package | Feature unlocked |
|
||||
|---------|-----------------|
|
||||
| `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
|
||||
| `ddgs` | DuckDuckGo as a search provider option. |
|
||||
| `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
|
||||
| `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
|
||||
|
||||
### Faster, reproducible installs with uv (optional)
|
||||
[uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
|
||||
venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
|
||||
|
||||
```bash
|
||||
uv venv venv --python 3.13
|
||||
uv pip install -r requirements.txt
|
||||
# then continue as usual: python setup.py, uvicorn, ...
|
||||
```
|
||||
|
||||
`requirements.txt` is intentionally unpinned, so two installs at different times can produce different package versions. If you want a reproducible environment (e.g. across your own machines, or to roll back after a bad upgrade), snapshot and restore exact versions with:
|
||||
|
||||
```bash
|
||||
uv pip compile requirements.txt -o requirements.lock # snapshot current resolution
|
||||
uv pip sync requirements.lock # reproduce it exactly later
|
||||
```
|
||||
|
||||
`requirements.lock` is gitignored and platform-specific (compile it on the OS you deploy to). Regenerate it deliberately when you want to take upgrades. The plain `uv pip install -r requirements.txt` keeps following the unpinned requirements like pip does.
|
||||
|
||||
### Outlook / Office 365 email
|
||||
Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
|
||||
and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
|
||||
passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
|
||||
current limitation and the planned integration direction.
|
||||
|
||||
## Security Notes
|
||||
Odysseus is a self-hosted workspace with powerful local tools: shell access, file uploads, model downloads, web research, email/calendar integrations, and API tokens. Treat it like an admin console.
|
||||
|
||||
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
|
||||
- Keep `LOCALHOST_BYPASS=false` outside local development.
|
||||
- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
|
||||
- Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
|
||||
- Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
|
||||
- Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
|
||||
- Non-admin users do not get shell/Python/file read/write by default, and admin-only routes/tools such as MCP management, API tokens, webhooks, model/cookbook serving, backup/vault, and app settings are admin-gated. Other features are controlled by per-user privileges, so review each user's privileges before exposing a deployment.
|
||||
- Rotate any API keys or tokens that were ever pasted into a shared chat, demo, screenshot, or log.
|
||||
- If you enable API tokens or webhooks, create separate tokens per integration and delete unused ones.
|
||||
- Prefer binding manual development runs to `127.0.0.1`; bind to `0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
|
||||
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
|
||||
- Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
|
||||
|
||||
### Private or proxied deployments
|
||||
Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
|
||||
|
||||
1. Keep Odysseus on localhost, for example `127.0.0.1:7000`.
|
||||
2. Terminate HTTPS at a trusted reverse proxy or private access gateway.
|
||||
3. Put the authenticated Odysseus web/API entrypoint behind that layer.
|
||||
4. Keep raw service and model ports internal-only.
|
||||
|
||||
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
|
||||
`ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
|
||||
|
||||
Common internal-only ports from the default docs/compose setup:
|
||||
|
||||
| Port | Service |
|
||||
|---|---|
|
||||
| `7000` | Odysseus raw app port |
|
||||
| `8080` | SearXNG |
|
||||
| `8091` | ntfy |
|
||||
| `8100` | ChromaDB host port for manual/compose access |
|
||||
| `11434` | Ollama |
|
||||
| `8000-8020` | Common local model/provider APIs |
|
||||
A full hover-to-play tour lives on the landing page: [`docs/index.html`](docs/index.html).
|
||||
|
||||
## Contributing
|
||||
Help is welcome. The best entry points are fresh-install testing, provider setup
|
||||
bugs, mobile/editor polish, docs, and small focused refactors. See
|
||||
[ROADMAP.md](ROADMAP.md) for the current help-wanted list.
|
||||
|
||||
## Configuration
|
||||
Most setup is done inside the app with `/setup` or **Settings**. Use `.env`
|
||||
for deployment-level defaults and secrets you want present before first boot.
|
||||
Key settings:
|
||||
Help is welcome. The best entry points are fresh-install testing, provider setup bugs, mobile/editor polish, docs, and small focused refactors. See [CONTRIBUTING.md](CONTRIBUTING.md) and [ROADMAP.md](ROADMAP.md).
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `LLM_HOST` | `localhost` | Your LLM server (e.g. `llm-host.local:8000`) |
|
||||
| `LLM_HOSTS` | -- | Comma-separated list for model discovery |
|
||||
| `OPENAI_API_KEY` | -- | Optional OpenAI key. Prefer adding providers in the app unless pre-seeding. |
|
||||
| `SEARXNG_INSTANCE` | `http://localhost:8080` | SearXNG URL. Docker overrides this to `http://searxng:8080`. |
|
||||
| `SEARXNG_SECRET` | generated on first Docker boot | Optional SearXNG cookie/CSRF secret. Leave blank unless you need to pin it. |
|
||||
| `APP_BIND` | `127.0.0.1` | Docker Compose host bind address for the web UI. Use `0.0.0.0` only for intentional LAN/reverse-proxy access. |
|
||||
| `APP_PORT` | `7000` | Docker Compose host port for the web UI. |
|
||||
| `APP_DATA_DIR` | `./data` | Docker Compose host directory for application data volumes. |
|
||||
| `APP_LOGS_DIR` | `./logs` | Docker Compose host directory for application logs. |
|
||||
| `AUTH_ENABLED` | `true` | Enable/disable login |
|
||||
| `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
|
||||
| `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
|
||||
| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
|
||||
| `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
|
||||
| `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
|
||||
| `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |
|
||||
| `EMBEDDING_URL` | -- | OpenAI-compatible embeddings endpoint |
|
||||
| `ODYSSEUS_CHAT_UPLOAD_MAX_BYTES` | `10485760` | Chat/agent attachment cap in bytes. Raise for larger local PDFs or text documents. |
|
||||
| `ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES` | `104857600` | Gallery image upload cap in bytes (100 MB). |
|
||||
| `ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES` | `26214400` | Gallery transform input cap in bytes (25 MB). |
|
||||
| `ODYSSEUS_MEMORY_IMPORT_MAX_BYTES` | `10485760` | Memory import file cap in bytes (10 MB). |
|
||||
| `ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES` | `26214400` | Personal document upload cap in bytes (25 MB). |
|
||||
| `ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES` | `26214400` | Email compose attachment cap in bytes (25 MB). |
|
||||
| `ODYSSEUS_STT_MAX_AUDIO_BYTES` | `26214400` | Speech-to-text audio cap in bytes (25 MB). |
|
||||
| `ODYSSEUS_ICS_MAX_BYTES` | `10485760` | Calendar `.ics` import cap in bytes (10 MB). |
|
||||
## Security
|
||||
|
||||
All upload-limit vars are validated (must be a positive integer) and optional; an invalid value fails fast at startup.
|
||||
|
||||
### Built-in MCP servers (optional setup)
|
||||
|
||||
Odysseus auto-registers a few built-in MCP servers at startup. The npx-based ones (currently the browser server, `@playwright/mcp`) only start when their npm package is already in the local npx cache. If a package isn't cached, that server is skipped with a startup log message explaining what to do, so a fresh install does not block on a multi-minute npm download or hang if Playwright system deps are missing.
|
||||
|
||||
To enable the browser MCP (page navigation, screenshots, vision), run once:
|
||||
|
||||
```bash
|
||||
npx -y @playwright/mcp@latest --version
|
||||
```
|
||||
|
||||
That installs `@playwright/mcp` plus Playwright (~300MB total). Restart Odysseus and the server will register at startup.
|
||||
|
||||
## Architecture
|
||||
```
|
||||
app.py # FastAPI entry point
|
||||
core/ auth, database, middleware, constants
|
||||
src/ llm_core, agent_loop, agent_tools, chat_processor, search/
|
||||
routes/ chat, session, document, memory, model … endpoints
|
||||
services/ docs, memory, search, hwfit (Cookbook) …
|
||||
static/ index.html + app.js + style.css + js/ (modular front-end)
|
||||
docs/ landing page (index.html) + preview clips
|
||||
```
|
||||
|
||||
## Data
|
||||
All user data lives in `data/` (gitignored): `app.db` (sessions, messages, documents),
|
||||
`memory.json`, `presets.json`, `uploads/`, `personal_docs/`, `chroma/`, `settings.json`.
|
||||
|
||||
To back up or restore everything in `data/`, see the
|
||||
[Backup & Restore guide](docs/backup-restore.md).
|
||||
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly. Deployment details are in the [setup guide](docs/setup.md#security-notes).
|
||||
|
||||
## Star History
|
||||
|
||||
@@ -483,19 +72,5 @@ To back up or restore everything in `data/`, see the
|
||||
</a>
|
||||
|
||||
## License
|
||||
AGPL-3.0-or-later -- see [LICENSE](LICENSE) and [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md).
|
||||
|
||||
```
|
||||
|
|
||||
|||
|
||||
|||||
|
||||
| | | |||||||
|
||||
)_) )_) )_) ~|~
|
||||
)___))___))___)\ |
|
||||
)____)____)_____)\\|
|
||||
_____|____|____|_____\\\__
|
||||
\ /
|
||||
~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~
|
||||
~^~ all aboard! ~^~
|
||||
~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~
|
||||
```
|
||||
AGPL-3.0-or-later -- see [LICENSE](LICENSE) and [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md).
|
||||
|
||||
@@ -331,8 +331,8 @@ if AUTH_ENABLED:
|
||||
request.state.current_user = "internal-tool"
|
||||
request.state.api_token = False
|
||||
return await call_next(request)
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as _e:
|
||||
logger.warning("Internal tool auth header check failed", exc_info=_e)
|
||||
# Allow DIRECT localhost requests (internal service calls from
|
||||
# heartbeats etc.). Tunnel/proxy-forwarded requests are excluded by
|
||||
# _is_trusted_loopback so LOCALHOST_BYPASS can't be abused over a
|
||||
@@ -385,11 +385,10 @@ if AUTH_ENABLED:
|
||||
_db.close()
|
||||
try:
|
||||
await _asyncio.to_thread(_do)
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as _e:
|
||||
logger.debug("Failed to update token last_used_at", exc_info=_e)
|
||||
_asyncio.create_task(_touch_last_used(matched_id))
|
||||
# Keep bearer-token callers out of normal cookie/user
|
||||
# routes. API-aware routes can read api_token_owner.
|
||||
request.state.current_user = "api"
|
||||
request.state.api_token = True
|
||||
request.state.api_token_id = matched_id
|
||||
@@ -464,8 +463,8 @@ async def serve_generated_image(filename: str, request: Request):
|
||||
_db.close()
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as _e:
|
||||
logger.warning("Image ownership verification failed for %r", filename, exc_info=_e)
|
||||
ext = filename.rsplit('.', 1)[-1].lower()
|
||||
mime = {
|
||||
"png": "image/png", "jpg": "image/jpeg", "jpeg": "image/jpeg",
|
||||
|
||||
@@ -5,8 +5,9 @@ offers and pair to it, without duplicating any LLM logic.
|
||||
|
||||
Auth is enforced globally by AuthMiddleware (app.py), so reaching a handler here
|
||||
means the caller is authenticated by either a cookie session or a Bearer `ody_`
|
||||
API token. The read endpoints (ping/info/models) accept either; the pairing
|
||||
endpoints are admin-cookie only.
|
||||
API token. Ping/info accept either credential type, models requires a chat-
|
||||
scoped API token for bearer callers, and the pairing endpoints are admin-cookie
|
||||
only.
|
||||
|
||||
Pairing CSRF posture: minting happens ONLY on POST. The session cookie is
|
||||
SameSite=Lax (routes/auth_routes.py), which a browser does not send on a
|
||||
@@ -18,7 +19,7 @@ on a GET would be unsafe (Lax cookies ride top-level GET navigations), so GET
|
||||
|
||||
import html
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from fastapi.responses import HTMLResponse
|
||||
|
||||
from core.middleware import require_admin
|
||||
@@ -52,6 +53,18 @@ def owner_can_see(row_owner, owner) -> bool:
|
||||
return row_owner is None or row_owner == owner
|
||||
|
||||
|
||||
def require_models_scope(request: Request) -> None:
|
||||
"""Require the companion chat scope for bearer-token model inventory."""
|
||||
if not getattr(request.state, "api_token", False):
|
||||
return
|
||||
scopes = getattr(request.state, "api_token_scopes", None) or []
|
||||
if isinstance(scopes, str):
|
||||
scopes = [scope.strip() for scope in scopes.split(",")]
|
||||
scope_set = {str(scope).strip() for scope in scopes if str(scope).strip()}
|
||||
if _pairing.COMPANION_SCOPE not in scope_set:
|
||||
raise HTTPException(403, "API token requires chat scope")
|
||||
|
||||
|
||||
def mint_pairing_token(owner: str, invalidate=None) -> tuple[str, str]:
|
||||
"""Mint a pairing token AND invalidate the auth middleware's in-memory token
|
||||
cache, so the new token is accepted on the very next request without a server
|
||||
@@ -103,6 +116,7 @@ def setup_companion_routes() -> APIRouter:
|
||||
rows -- the same rule as owner_filter. Read-only; never returns api_key
|
||||
material.
|
||||
"""
|
||||
require_models_scope(request)
|
||||
import json as _json
|
||||
|
||||
from core.database import SessionLocal, ModelEndpoint
|
||||
|
||||
@@ -2,12 +2,15 @@ import os
|
||||
import logging
|
||||
import sqlite3
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from sqlalchemy import event, create_engine, Column, String, Text, Boolean, DateTime, Integer, ForeignKey, JSON, Index, func, text
|
||||
from sqlalchemy.engine import Engine
|
||||
from sqlalchemy.types import TypeDecorator
|
||||
from sqlalchemy.ext.declarative import declarative_base, declared_attr
|
||||
from sqlalchemy.orm import relationship, sessionmaker, backref
|
||||
|
||||
from src.runtime_paths import get_app_root
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Create base class for declarative models
|
||||
@@ -29,9 +32,26 @@ class TimestampMixin:
|
||||
def updated_at(cls):
|
||||
return Column(DateTime, default=utcnow_naive, onupdate=utcnow_naive, nullable=False)
|
||||
|
||||
# Get database URL from environment, default to SQLite in DATA_DIR
|
||||
# Ensure the writable data directory exists before SQLite connects.
|
||||
from src.constants import DATA_DIR, AUTH_FILE, MEMORY_FILE, USER_PREFS_FILE, SETTINGS_FILE
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{DATA_DIR}/app.db")
|
||||
Path(DATA_DIR).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def _default_database_url() -> str:
|
||||
return f"sqlite:///{Path(DATA_DIR) / 'app.db'}"
|
||||
|
||||
|
||||
def _normalize_sqlite_url(url: str) -> str:
|
||||
if not url.startswith("sqlite:///"):
|
||||
return url
|
||||
db_path = url.replace("sqlite:///", "", 1)
|
||||
if db_path == ":memory:" or os.path.isabs(db_path):
|
||||
return url
|
||||
return f"sqlite:///{(Path(get_app_root()) / db_path).resolve().as_posix()}"
|
||||
|
||||
|
||||
# Get database URL from environment, default to SQLite in DATA_DIR
|
||||
DATABASE_URL = _normalize_sqlite_url(os.getenv("DATABASE_URL", _default_database_url()))
|
||||
|
||||
# Create engine
|
||||
engine = create_engine(
|
||||
@@ -324,6 +344,13 @@ class EmailAccount(TimestampMixin, Base):
|
||||
smtp_password = Column(String, default="")
|
||||
|
||||
from_address = Column(String, default="")
|
||||
display_name = Column(String, nullable=True) # "Hriday Ranka" — used in From: header
|
||||
|
||||
# OAuth2 (Google / Google Workspace). Tokens stored encrypted via secret_storage.
|
||||
oauth_provider = Column(String, nullable=True) # "google" or None
|
||||
oauth_access_token = Column(String, nullable=True) # encrypted
|
||||
oauth_refresh_token = Column(String, nullable=True) # encrypted
|
||||
oauth_token_expiry = Column(String, nullable=True) # unix timestamp string
|
||||
|
||||
__table_args__ = (
|
||||
Index('ix_email_accounts_owner_default', 'owner', 'is_default'),
|
||||
@@ -1427,6 +1454,25 @@ def _migrate_add_task_automation_columns():
|
||||
except Exception as e:
|
||||
logging.getLogger(__name__).warning(f"task automation migration: {e}")
|
||||
|
||||
def _migrate_add_email_oauth_columns():
|
||||
"""Add Google OAuth and display_name columns to email_accounts if missing."""
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
cols = [r[1] for r in conn.execute(text("PRAGMA table_info(email_accounts)"))]
|
||||
for col, typedef in [
|
||||
("oauth_provider", "TEXT"),
|
||||
("oauth_access_token", "TEXT"),
|
||||
("oauth_refresh_token", "TEXT"),
|
||||
("oauth_token_expiry", "TEXT"),
|
||||
("display_name", "TEXT"),
|
||||
]:
|
||||
if col not in cols:
|
||||
conn.execute(text(f"ALTER TABLE email_accounts ADD COLUMN {col} {typedef}"))
|
||||
conn.commit()
|
||||
except Exception as e:
|
||||
logging.getLogger(__name__).warning(f"email oauth columns migration: {e}")
|
||||
|
||||
|
||||
def _migrate_add_oauth_config():
|
||||
"""Add oauth_config column to mcp_servers table if missing."""
|
||||
try:
|
||||
@@ -1771,6 +1817,7 @@ def init_db():
|
||||
_migrate_add_tidy_verdict()
|
||||
_migrate_add_doc_source_email_cols()
|
||||
_migrate_add_oauth_config()
|
||||
_migrate_add_email_oauth_columns()
|
||||
_migrate_add_task_automation_columns()
|
||||
_migrate_add_disabled_tools()
|
||||
_migrate_add_mcp_oauth_tokens_column()
|
||||
|
||||
|
Before Width: | Height: | Size: 3.0 MiB |
|
Before Width: | Height: | Size: 3.4 MiB |
|
Before Width: | Height: | Size: 1.1 MiB |
|
Before Width: | Height: | Size: 1003 KiB |
|
After Width: | Height: | Size: 16 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 52 KiB |
|
Before Width: | Height: | Size: 2.5 MiB |
@@ -1,14 +1,16 @@
|
||||
# Security CI guide
|
||||
|
||||
This project runs a set of automated security checks on every pull request and
|
||||
on every push to `main`. This page explains what each one does, whether it can
|
||||
This project runs a set of automated security checks on pull requests and
|
||||
selected branch pushes. This page explains what each one does, whether it can
|
||||
block a merge, and the few one-time settings you should turn on to get the full
|
||||
benefit.
|
||||
|
||||
## What runs, and why
|
||||
|
||||
Each check lives in its own file under `.github/workflows/`. They run
|
||||
automatically; you do not start them.
|
||||
Most checks live in files under `.github/workflows/`. CodeQL is configured
|
||||
through GitHub's code scanning default setup, so it appears as a dynamic GitHub
|
||||
workflow instead of a checked-in workflow file. They run automatically; you do
|
||||
not start them.
|
||||
|
||||
| Check | What it protects against | Blocks a merge? |
|
||||
|---|---|---|
|
||||
@@ -88,11 +90,14 @@ let the workflows run on one pull request first, then add them here.
|
||||
2. Turn on **Dependency graph** (usually on by default for public repos) -- this
|
||||
powers Dependency review and Dependabot.
|
||||
3. Turn on **Dependabot alerts** and **Dependabot security updates**.
|
||||
4. Under **Code scanning**, you have two ways to scan the app code with CodeQL:
|
||||
- The included `codeql.yml` workflow already scans `main` and runs weekly.
|
||||
- To also scan **pull requests** (recommended, since most contributions come
|
||||
from forks), click **Set up -> Default** under Code scanning. GitHub then
|
||||
runs CodeQL on pull requests for you, with no token limitations.
|
||||
4. Under **Code scanning**, use **Set up -> Default** for CodeQL. GitHub then
|
||||
runs CodeQL as a dynamic workflow without the fork-token limitations that
|
||||
affect checked-in advanced workflows.
|
||||
|
||||
Do not also add a checked-in CodeQL workflow while default setup is enabled:
|
||||
GitHub rejects advanced CodeQL uploads when default setup is active. If the
|
||||
project later needs an advanced CodeQL workflow, disable default setup first
|
||||
and keep only one CodeQL publishing path active.
|
||||
|
||||
## Keeping it current
|
||||
|
||||
|
||||
@@ -0,0 +1,425 @@
|
||||
# Odysseus Setup Guide
|
||||
|
||||
This page keeps the detailed install, deployment, troubleshooting, and configuration notes out of the front README.
|
||||
|
||||
## Quick Start
|
||||
|
||||
> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main).
|
||||
|
||||
Defaults work out of the box: clone, run, then configure models/search/email
|
||||
inside **Settings**. Only edit `.env` for deployment-level overrides like
|
||||
`APP_BIND`, `APP_PORT`, `AUTH_ENABLED`, `DATABASE_URL`, or a pre-seeded admin password.
|
||||
|
||||
On first setup, Odysseus creates an admin account (`admin` unless
|
||||
`ODYSSEUS_ADMIN_USER` is set) and prints a temporary password in the terminal.
|
||||
For Docker installs, the same line is in `docker compose logs odysseus`.
|
||||
Use that for the first login, then change it in **Settings**.
|
||||
|
||||
Contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, testing, and
|
||||
pull request guidelines.
|
||||
|
||||
### Docker (recommended)
|
||||
```bash
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
cp .env.example .env # optional, but recommended for explicit defaults
|
||||
docker compose up -d --build
|
||||
```
|
||||
To include optional extras in the image (PDF viewer, Office extraction; includes AGPL PyMuPDF), build with `docker compose build --build-arg INSTALL_OPTIONAL=true` before `up`.
|
||||
|
||||
Open `http://localhost:7000` when the containers are healthy. Docker Compose
|
||||
binds the web UI to `127.0.0.1` by default. If the port is taken, set
|
||||
`APP_PORT=7001` in `.env` and recreate the container. Set `APP_BIND=0.0.0.0`
|
||||
only when you intentionally want LAN/reverse-proxy access.
|
||||
|
||||
> **On Apple Silicon (M-series) Macs:** Docker can't reach the Metal GPU, so
|
||||
> Cookbook serves local models on CPU only. For GPU-accelerated model serving,
|
||||
> run natively instead — see [Apple Silicon](#apple-silicon) below.
|
||||
|
||||
### Native Linux / macOS
|
||||
```bash
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
python setup.py
|
||||
python -m uvicorn app:app --host 127.0.0.1 --port 7000
|
||||
```
|
||||
Requirements: Python 3.11+. Cookbook also needs `tmux` for background model
|
||||
downloads and serves. The app itself is lightweight; local model serving is the
|
||||
heavy part and depends on the model, runtime, GPU, and VRAM, so small hosts can
|
||||
connect to API or remote model servers instead. Use `--host 0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
|
||||
|
||||
### Apple Silicon
|
||||
Docker on macOS cannot use the Metal GPU. For GPU-accelerated Cookbook on an
|
||||
M-series Mac, run Odysseus natively:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
./start-macos.sh
|
||||
```
|
||||
|
||||
It launches at `http://127.0.0.1:7860`. To expose it to your phone over a trusted LAN/VPN such as Tailscale, bind all interfaces:
|
||||
|
||||
```bash
|
||||
ODYSSEUS_HOST=0.0.0.0 ./start-macos.sh
|
||||
# then open http://<tailscale-ip>:7860
|
||||
```
|
||||
|
||||
The script also reads `.env` at startup, so `APP_BIND=0.0.0.0` and `APP_PORT`
|
||||
set there are picked up automatically without a command-line override each run.
|
||||
|
||||
Keep `AUTH_ENABLED=true` (the default) before binding outside loopback. Do not
|
||||
expose this port directly to the public internet. To build a clickable app wrapper:
|
||||
|
||||
```bash
|
||||
./build-macos-app.sh
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Cookbook, GPU, Ollama, and troubleshooting notes</summary>
|
||||
|
||||
**Docker bundled services.** Compose starts Odysseus, ChromaDB, SearXNG, and
|
||||
ntfy. Odysseus and the bundled service ports bind to `127.0.0.1` by default, so
|
||||
they are reachable from the host but not exposed to your LAN/public internet
|
||||
unless you opt in.
|
||||
|
||||
**Cookbook storage in Docker.** Downloads live in `./data/huggingface`
|
||||
(`~/.cache/huggingface` in the container). Cookbook-installed Python CLIs and
|
||||
serve engines live in `./data/local` (`~/.local` in the container), so they
|
||||
survive container recreation.
|
||||
|
||||
**Remote servers.** In **Cookbook -> Settings -> Servers**, generate the
|
||||
Odysseus SSH key and add the public key to the remote server's
|
||||
`~/.ssh/authorized_keys`. From the host you can also run:
|
||||
|
||||
```bash
|
||||
ssh-copy-id -i data/ssh/id_ed25519.pub user@server
|
||||
```
|
||||
|
||||
**Docker GPU overlays.** CPU-only users can skip this section. Cookbook can
|
||||
only detect GPUs that Docker exposes to the container — if the host runtime or
|
||||
device passthrough is not configured, Cookbook sees the iGPU, another card, or
|
||||
CPU instead of your intended GPU.
|
||||
|
||||
For NVIDIA, `scripts/check-docker-gpu.sh` diagnoses GPU passthrough and can
|
||||
optionally install the host runtime or update `.env`.
|
||||
|
||||
```bash
|
||||
# Read-only diagnostic (default — installs nothing, never edits .env):
|
||||
scripts/check-docker-gpu.sh
|
||||
|
||||
# Print OS-specific install commands without running them:
|
||||
scripts/check-docker-gpu.sh --print-install-commands
|
||||
|
||||
# Install NVIDIA Container Toolkit on Ubuntu/Debian (requires sudo):
|
||||
scripts/check-docker-gpu.sh --install-nvidia-toolkit
|
||||
|
||||
# Write COMPOSE_FILE to .env (only when GPU passthrough is confirmed working):
|
||||
scripts/check-docker-gpu.sh --enable-nvidia-overlay
|
||||
|
||||
# Full assisted setup — install toolkit, then enable overlay if passthrough works:
|
||||
scripts/check-docker-gpu.sh --install-nvidia-toolkit --enable-nvidia-overlay
|
||||
```
|
||||
|
||||
Safety notes:
|
||||
- The app never installs host GPU runtime automatically.
|
||||
- The app never edits `.env` automatically.
|
||||
- `.env` is only modified when `--enable-nvidia-overlay` is explicitly passed,
|
||||
and only after GPU passthrough succeeds. `--yes` skips prompts but does not
|
||||
bypass the passthrough gate.
|
||||
- `.env.bak.*` backups created by `--enable-nvidia-overlay` are ignored by
|
||||
Git and the Docker build context.
|
||||
|
||||
To enable manually without the script, add this to `.env`:
|
||||
|
||||
```bash
|
||||
COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
|
||||
```
|
||||
|
||||
**AMD / ROCm.** AMD setup is read-only diagnostic plus manual `.env` edit. Run:
|
||||
|
||||
```bash
|
||||
scripts/check-docker-amd-gpu.sh
|
||||
```
|
||||
|
||||
Then add the reported values to `.env`, replacing `RENDER_GID` with your host's
|
||||
numeric render group id:
|
||||
|
||||
```bash
|
||||
COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
|
||||
RENDER_GID=989
|
||||
```
|
||||
|
||||
For NVIDIA/AMD GPU support, also read the comments in the selected overlay file: docker/gpu.nvidia.yml or docker/gpu.amd.yml.
|
||||
|
||||
**Stack-management UIs (Portainer, Coolify, Dockhand, etc.).** These tools
|
||||
often accept only a single Compose file and do not reliably honor `COMPOSE_FILE`
|
||||
or multiple `-f` overlays. CLI users should keep using the `COMPOSE_FILE`
|
||||
overlay workflow above. For stack UIs, point the stack at one of the standalone
|
||||
files instead, which bundle the base stack plus the GPU settings:
|
||||
|
||||
- `docker-compose.gpu-nvidia.yml` — still requires the NVIDIA Container Toolkit
|
||||
on the host.
|
||||
- `docker-compose.gpu-amd.yml` — still requires host ROCm/kfd/DRI setup, the
|
||||
`video`/`render` group membership, and `RENDER_GID` when needed.
|
||||
|
||||
The base `docker-compose.yml` plus the `docker/gpu.*.yml` overlays remain the
|
||||
source of truth; the standalone files mirror them for single-file deployments.
|
||||
|
||||
Verify after enabling either overlay:
|
||||
|
||||
```bash
|
||||
docker compose exec odysseus nvidia-smi -L # NVIDIA
|
||||
docker compose exec odysseus sh -lc 'test -e /dev/kfd && test -d /dev/dri && ls -l /dev/kfd /dev/dri/renderD*' # AMD
|
||||
```
|
||||
|
||||
> **GPU passthrough ≠ llama.cpp CUDA.** `nvidia-smi` passing inside the
|
||||
> container confirms Docker GPU access, but llama.cpp also needs `cudart` and
|
||||
> the CUDA Toolkit at runtime. If Cookbook logs show `Unable to find cudart
|
||||
> library`, `Could NOT find CUDAToolkit`, `CUDA Toolkit not found`, or
|
||||
> tensors/layers assigned to CPU, that is a Cookbook/llama.cpp build issue —
|
||||
> not a Docker passthrough failure. Reinstall the serve engine via
|
||||
> **Cookbook → Dependencies** to get a CUDA-enabled build.
|
||||
>
|
||||
> The same split applies to AMD/ROCm: seeing `/dev/kfd` and `/dev/dri` inside
|
||||
> the container confirms device passthrough, not ROCm userspace or a
|
||||
> ROCm-enabled vLLM/llama.cpp build. `rocm-smi` and `rocminfo` are not expected
|
||||
> inside the slim Odysseus image.
|
||||
|
||||
**Ollama with Docker.** If Ollama runs on the host, add this endpoint in
|
||||
Settings:
|
||||
|
||||
```text
|
||||
http://host.docker.internal:11434/v1
|
||||
```
|
||||
|
||||
Ollama must listen outside its own loopback interface:
|
||||
|
||||
```bash
|
||||
OLLAMA_HOST=0.0.0.0:11434 ollama serve
|
||||
```
|
||||
|
||||
This connects Odysseus in Docker to an Ollama server that is already running on
|
||||
your host machine; it does not start Ollama inside the container.
|
||||
`host.docker.internal` is Docker's hostname for the host machine from inside the
|
||||
container. Cookbook **Serve** is a separate workflow for serving downloaded
|
||||
models through Odysseus/llama.cpp, so Windows users with an existing Ollama
|
||||
install usually only need to add the endpoint in Settings.
|
||||
|
||||
**Useful checks.**
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
docker compose logs --tail=120 odysseus
|
||||
docker compose logs odysseus | grep -E 'ChromaDB|MemoryVectorStore|DEGRADED'
|
||||
```
|
||||
|
||||
**macOS details.** `start-macos.sh` installs Homebrew deps, creates the venv,
|
||||
runs setup, and starts uvicorn on port `7860` because AirPlay often holds
|
||||
`7000`. It uses llama.cpp/Ollama for Metal. vLLM/SGLang are CUDA/ROCm-only and
|
||||
do not run on macOS. MLX-only models are not served by Odysseus.
|
||||
|
||||
</details>
|
||||
|
||||
### Native Windows
|
||||
|
||||
**One-command launcher** (creates the venv, installs deps, runs setup, starts the
|
||||
server; safe to re-run):
|
||||
|
||||
```powershell
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
|
||||
```
|
||||
|
||||
Or do it by hand:
|
||||
|
||||
```powershell
|
||||
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
|
||||
cd odysseus
|
||||
py -3.11 -m venv venv
|
||||
venv\Scripts\Activate.ps1
|
||||
pip install -r requirements.txt
|
||||
python setup.py
|
||||
python -m uvicorn app:app --host 127.0.0.1 --port 7000
|
||||
```
|
||||
|
||||
If `python` points at an older interpreter, use `py -3.12` (or another installed
|
||||
3.11+ version) for the venv step.
|
||||
|
||||
**Requirements:** Python 3.11+. The core app (chat, agent, memory, documents,
|
||||
email, calendar, deep research) runs fully native. For full **Cookbook** background
|
||||
model downloads and the agent shell tool, also install
|
||||
[Git for Windows](https://git-scm.com/download/win) (provides `bash.exe`).
|
||||
Local GPU *serving* of vLLM/SGLang needs Linux/WSL2; for a local model on Windows,
|
||||
[Ollama](https://ollama.com/download) is the easiest path — point Odysseus at
|
||||
`http://localhost:11434/v1` in Settings.
|
||||
|
||||
Open `http://localhost:7000`, log in with the generated admin password,
|
||||
and configure everything else inside **Settings**.
|
||||
|
||||
## Troubleshooting & Advanced Setup
|
||||
|
||||
### `chromadb-client` conflicts with embedded ChromaDB
|
||||
If `chromadb-client` (the lightweight HTTP-only package) is installed alongside the full `chromadb` package, Odysseus starts but ChromaDB silently falls back to HTTP-only mode and fails.
|
||||
|
||||
**Fix:** uninstall `chromadb-client` and force-reinstall the full package:
|
||||
```bash
|
||||
./venv/bin/pip uninstall chromadb-client -y
|
||||
./venv/bin/pip install --force-reinstall chromadb
|
||||
```
|
||||
|
||||
### HTTPS + LAN/Tailscale exposure
|
||||
To expose Odysseus on a local network or Tailscale with HTTPS:
|
||||
1. Change the bind address to `0.0.0.0` in `.env` (`APP_BIND=0.0.0.0` or `ODYSSEUS_HOST=0.0.0.0`).
|
||||
2. Generate a locally-trusted cert for your LAN/Tailscale IPs using [mkcert](https://github.com/FiloSottile/mkcert):
|
||||
```bash
|
||||
mkcert -install
|
||||
mkcert -cert-file cert.pem -key-file key.pem 192.168.1.100 tailscale-ip
|
||||
```
|
||||
3. Run `uvicorn` with the generated certs:
|
||||
```bash
|
||||
python -m uvicorn app:app --host 0.0.0.0 --port 7000 --ssl-certfile=cert.pem --ssl-keyfile=key.pem
|
||||
```
|
||||
4. Install the `mkcert` CA on any other device you want to access Odysseus from (e.g., for iOS, email the `rootCA.pem` to yourself, install the profile, and trust it in Certificate Trust Settings).
|
||||
|
||||
### Optional Dependencies
|
||||
`requirements-optional.txt` contains packages that unlock extra features. It is not installed by default.
|
||||
|
||||
| Package | Feature unlocked |
|
||||
|---------|-----------------|
|
||||
| `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
|
||||
| `ddgs` | DuckDuckGo as a search provider option. |
|
||||
| `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
|
||||
| `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
|
||||
|
||||
### Faster, reproducible installs with uv (optional)
|
||||
[uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
|
||||
venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
|
||||
|
||||
```bash
|
||||
uv venv venv --python 3.13
|
||||
uv pip install -r requirements.txt
|
||||
# then continue as usual: python setup.py, uvicorn, ...
|
||||
```
|
||||
|
||||
`requirements.txt` is intentionally unpinned, so two installs at different times can produce different package versions. If you want a reproducible environment (e.g. across your own machines, or to roll back after a bad upgrade), snapshot and restore exact versions with:
|
||||
|
||||
```bash
|
||||
uv pip compile requirements.txt -o requirements.lock # snapshot current resolution
|
||||
uv pip sync requirements.lock # reproduce it exactly later
|
||||
```
|
||||
|
||||
`requirements.lock` is gitignored and platform-specific (compile it on the OS you deploy to). Regenerate it deliberately when you want to take upgrades. The plain `uv pip install -r requirements.txt` keeps following the unpinned requirements like pip does.
|
||||
|
||||
### Outlook / Office 365 email
|
||||
Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
|
||||
and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
|
||||
passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
|
||||
current limitation and the planned integration direction.
|
||||
|
||||
## Security Notes
|
||||
Odysseus is a self-hosted workspace with powerful local tools: shell access, file uploads, model downloads, web research, email/calendar integrations, and API tokens. Treat it like an admin console.
|
||||
|
||||
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
|
||||
- Keep `LOCALHOST_BYPASS=false` outside local development.
|
||||
- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
|
||||
- Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
|
||||
- Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
|
||||
- Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
|
||||
- Non-admin users do not get shell/Python/file read/write by default, and admin-only routes/tools such as MCP management, API tokens, webhooks, model/cookbook serving, backup/vault, and app settings are admin-gated. Other features are controlled by per-user privileges, so review each user's privileges before exposing a deployment.
|
||||
- Rotate any API keys or tokens that were ever pasted into a shared chat, demo, screenshot, or log.
|
||||
- If you enable API tokens or webhooks, create separate tokens per integration and delete unused ones.
|
||||
- Prefer binding manual development runs to `127.0.0.1`; bind to `0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
|
||||
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
|
||||
- Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
|
||||
|
||||
### Private or proxied deployments
|
||||
Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
|
||||
|
||||
1. Keep Odysseus on localhost, for example `127.0.0.1:7000`.
|
||||
2. Terminate HTTPS at a trusted reverse proxy or private access gateway.
|
||||
3. Put the authenticated Odysseus web/API entrypoint behind that layer.
|
||||
4. Keep raw service and model ports internal-only.
|
||||
|
||||
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
|
||||
`ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
|
||||
|
||||
Common internal-only ports from the default docs/compose setup:
|
||||
|
||||
| Port | Service |
|
||||
|---|---|
|
||||
| `7000` | Odysseus raw app port |
|
||||
| `8080` | SearXNG |
|
||||
| `8091` | ntfy |
|
||||
| `8100` | ChromaDB host port for manual/compose access |
|
||||
| `11434` | Ollama |
|
||||
| `8000-8020` | Common local model/provider APIs |
|
||||
|
||||
## Configuration
|
||||
Most setup is done inside the app with `/setup` or **Settings**. Use `.env`
|
||||
for deployment-level defaults and secrets you want present before first boot.
|
||||
Key settings:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `LLM_HOST` | `localhost` | Your LLM server (e.g. `llm-host.local:8000`) |
|
||||
| `LLM_HOSTS` | -- | Comma-separated list for model discovery |
|
||||
| `OPENAI_API_KEY` | -- | Optional OpenAI key. Prefer adding providers in the app unless pre-seeding. |
|
||||
| `SEARXNG_INSTANCE` | `http://localhost:8080` | SearXNG URL. Docker overrides this to `http://searxng:8080`. |
|
||||
| `SEARXNG_SECRET` | generated on first Docker boot | Optional SearXNG cookie/CSRF secret. Leave blank unless you need to pin it. |
|
||||
| `APP_BIND` | `127.0.0.1` | Docker Compose host bind address for the web UI. Use `0.0.0.0` only for intentional LAN/reverse-proxy access. |
|
||||
| `APP_PORT` | `7000` | Docker Compose host port for the web UI. |
|
||||
| `APP_DATA_DIR` | `./data` | Docker Compose host directory for application data volumes. |
|
||||
| `APP_LOGS_DIR` | `./logs` | Docker Compose host directory for application logs. |
|
||||
| `AUTH_ENABLED` | `true` | Enable/disable login |
|
||||
| `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
|
||||
| `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
|
||||
| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
|
||||
| `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
|
||||
| `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
|
||||
| `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |
|
||||
| `EMBEDDING_URL` | -- | OpenAI-compatible embeddings endpoint |
|
||||
| `ODYSSEUS_CHAT_UPLOAD_MAX_BYTES` | `10485760` | Chat/agent attachment cap in bytes. Raise for larger local PDFs or text documents. |
|
||||
| `ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES` | `104857600` | Gallery image upload cap in bytes (100 MB). |
|
||||
| `ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES` | `26214400` | Gallery transform input cap in bytes (25 MB). |
|
||||
| `ODYSSEUS_MEMORY_IMPORT_MAX_BYTES` | `10485760` | Memory import file cap in bytes (10 MB). |
|
||||
| `ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES` | `26214400` | Personal document upload cap in bytes (25 MB). |
|
||||
| `ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES` | `26214400` | Email compose attachment cap in bytes (25 MB). |
|
||||
| `ODYSSEUS_STT_MAX_AUDIO_BYTES` | `26214400` | Speech-to-text audio cap in bytes (25 MB). |
|
||||
| `ODYSSEUS_ICS_MAX_BYTES` | `10485760` | Calendar `.ics` import cap in bytes (10 MB). |
|
||||
|
||||
All upload-limit vars are validated (must be a positive integer) and optional; an invalid value fails fast at startup.
|
||||
|
||||
### Built-in MCP servers (optional setup)
|
||||
|
||||
Odysseus auto-registers a few built-in MCP servers at startup. The npx-based ones (currently the browser server, `@playwright/mcp`) only start when their npm package is already in the local npx cache. If a package isn't cached, that server is skipped with a startup log message explaining what to do, so a fresh install does not block on a multi-minute npm download or hang if Playwright system deps are missing.
|
||||
|
||||
To enable the browser MCP (page navigation, screenshots, vision), run once:
|
||||
|
||||
```bash
|
||||
npx -y @playwright/mcp@latest --version
|
||||
```
|
||||
|
||||
That installs `@playwright/mcp` plus Playwright (~300MB total). Restart Odysseus and the server will register at startup.
|
||||
|
||||
## Architecture
|
||||
```
|
||||
app.py # FastAPI entry point
|
||||
core/ auth, database, middleware, constants
|
||||
src/ llm_core, agent_loop, agent_tools, chat_processor, search/
|
||||
routes/ chat, session, document, memory, model … endpoints
|
||||
services/ docs, memory, search, hwfit (Cookbook) …
|
||||
static/ index.html + app.js + style.css + js/ (modular front-end)
|
||||
docs/ landing page (index.html) + preview clips
|
||||
```
|
||||
|
||||
## Data
|
||||
All user data lives in `data/` (gitignored): `app.db` (sessions, messages, documents),
|
||||
`memory.json`, `presets.json`, `uploads/`, `personal_docs/`, `chroma/`, `settings.json`.
|
||||
|
||||
To back up or restore everything in `data/`, see the
|
||||
[Backup & Restore guide](docs/backup-restore.md).
|
||||
@@ -6,6 +6,7 @@ Imports MemoryManager and MemoryVectorStore from the Odysseus codebase.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
@@ -23,6 +24,55 @@ _memory_manager = None
|
||||
_memory_vector = None
|
||||
_initialized = False
|
||||
|
||||
_OWNER_ENV_KEYS = ("ODYSSEUS_MCP_MEMORY_OWNER", "ODYSSEUS_MEMORY_OWNER")
|
||||
_OWNER_SCOPE_ERROR = (
|
||||
"Error: Memory MCP owner is not configured for an owner-scoped memory store. "
|
||||
"Set ODYSSEUS_MCP_MEMORY_OWNER for this server or use the owner-aware native memory tool."
|
||||
)
|
||||
|
||||
|
||||
def _configured_owner() -> str | None:
|
||||
for key in _OWNER_ENV_KEYS:
|
||||
owner = os.environ.get(key, "").strip()
|
||||
if owner:
|
||||
return owner
|
||||
return None
|
||||
|
||||
|
||||
def _entry_owner(entry: dict) -> str | None:
|
||||
owner = entry.get("owner")
|
||||
if owner is None:
|
||||
return None
|
||||
owner_text = str(owner).strip()
|
||||
return owner_text or None
|
||||
|
||||
|
||||
def _owner_scoped_store(entries: list[dict]) -> bool:
|
||||
return any(_entry_owner(entry) for entry in entries if isinstance(entry, dict))
|
||||
|
||||
|
||||
def _scope_entries() -> tuple[str | None, list[dict], list[dict], str | None]:
|
||||
"""Return configured owner, all entries, visible entries, and optional error."""
|
||||
entries = _memory_manager.load_all()
|
||||
owner = _configured_owner()
|
||||
if owner is None and _owner_scoped_store(entries):
|
||||
return None, entries, [], _OWNER_SCOPE_ERROR
|
||||
if owner is None:
|
||||
visible = [
|
||||
entry for entry in entries
|
||||
if isinstance(entry, dict) and _entry_owner(entry) is None
|
||||
]
|
||||
else:
|
||||
visible = [
|
||||
entry for entry in entries
|
||||
if isinstance(entry, dict) and _entry_owner(entry) == owner
|
||||
]
|
||||
return owner, entries, visible, None
|
||||
|
||||
|
||||
def _text_result(text: str) -> list[TextContent]:
|
||||
return [TextContent(type="text", text=text)]
|
||||
|
||||
|
||||
def _ensure_init():
|
||||
"""Lazy-init memory managers on first use."""
|
||||
@@ -75,24 +125,26 @@ async def list_tools() -> list[Tool]:
|
||||
@server.call_tool()
|
||||
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
|
||||
if name != "manage_memory":
|
||||
return [TextContent(type="text", text=f"Unknown tool: {name}")]
|
||||
return _text_result(f"Unknown tool: {name}")
|
||||
|
||||
_ensure_init()
|
||||
if not _memory_manager:
|
||||
return [TextContent(type="text", text="Error: Memory manager not available")]
|
||||
return _text_result("Error: Memory manager not available")
|
||||
|
||||
action = arguments.get("action", "")
|
||||
|
||||
if action == "list":
|
||||
category_filter = arguments.get("category", "")
|
||||
memories = _memory_manager.load()
|
||||
_owner, _all_memories, memories, scope_error = _scope_entries()
|
||||
if scope_error:
|
||||
return _text_result(scope_error)
|
||||
if category_filter:
|
||||
memories = [m for m in memories if m.get("category", "").lower() == category_filter.lower()]
|
||||
if not memories:
|
||||
msg = "No memories found"
|
||||
if category_filter:
|
||||
msg += f" in category '{category_filter}'"
|
||||
return [TextContent(type="text", text=msg + ".")]
|
||||
return _text_result(msg + ".")
|
||||
|
||||
lines = [f"Found {len(memories)} memory entries:\n"]
|
||||
for m in memories:
|
||||
@@ -102,15 +154,17 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
|
||||
if len(text) > 150:
|
||||
text = text[:150] + "..."
|
||||
lines.append(f"- [{cat}] `{mid}` — {text}")
|
||||
return [TextContent(type="text", text="\n".join(lines))]
|
||||
return _text_result("\n".join(lines))
|
||||
|
||||
elif action == "add":
|
||||
text = arguments.get("text", "")
|
||||
category = arguments.get("category", "fact")
|
||||
if not text:
|
||||
return [TextContent(type="text", text="Error: Memory text cannot be empty")]
|
||||
entry = _memory_manager.add_entry(text, source="ai_agent", category=category)
|
||||
memories = _memory_manager.load_all()
|
||||
return _text_result("Error: Memory text cannot be empty")
|
||||
owner, memories, _visible, scope_error = _scope_entries()
|
||||
if scope_error:
|
||||
return _text_result(scope_error)
|
||||
entry = _memory_manager.add_entry(text, source="ai_agent", category=category, owner=owner)
|
||||
memories.append(entry)
|
||||
_memory_manager.save(memories)
|
||||
if _memory_vector and _memory_vector.healthy:
|
||||
@@ -118,25 +172,28 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
|
||||
_memory_vector.add(entry["id"], text)
|
||||
except Exception:
|
||||
pass
|
||||
return [TextContent(type="text", text=f"Memory added: [{category}] {text} (id: {entry['id'][:8]})")]
|
||||
return _text_result(f"Memory added: [{category}] {text} (id: {entry['id'][:8]})")
|
||||
|
||||
elif action == "edit":
|
||||
memory_id = arguments.get("memory_id", "")
|
||||
new_text = arguments.get("text", "")
|
||||
if not memory_id or not new_text:
|
||||
return [TextContent(type="text", text="Error: edit needs memory_id and text")]
|
||||
memories = _memory_manager.load_all()
|
||||
found = False
|
||||
return _text_result("Error: edit needs memory_id and text")
|
||||
_owner, memories, visible, scope_error = _scope_entries()
|
||||
if scope_error:
|
||||
return _text_result(scope_error)
|
||||
full_id = None
|
||||
for m in memories:
|
||||
for m in visible:
|
||||
if m.get("id", "").startswith(memory_id):
|
||||
m["text"] = new_text
|
||||
m["timestamp"] = int(time.time())
|
||||
found = True
|
||||
full_id = m["id"]
|
||||
break
|
||||
if not found:
|
||||
return [TextContent(type="text", text=f"Error: Memory '{memory_id}' not found")]
|
||||
if not full_id:
|
||||
return _text_result(f"Error: Memory '{memory_id}' not found")
|
||||
for m in memories:
|
||||
if m.get("id") == full_id:
|
||||
m["text"] = new_text
|
||||
m["timestamp"] = int(time.time())
|
||||
break
|
||||
_memory_manager.save(memories)
|
||||
if _memory_vector and _memory_vector.healthy and full_id:
|
||||
try:
|
||||
@@ -144,24 +201,26 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
|
||||
_memory_vector.add(full_id, new_text)
|
||||
except Exception:
|
||||
pass
|
||||
return [TextContent(type="text", text=f"Memory updated: {new_text}")]
|
||||
return _text_result(f"Memory updated: {new_text}")
|
||||
|
||||
elif action == "delete":
|
||||
memory_id = arguments.get("memory_id", "")
|
||||
if not memory_id:
|
||||
return [TextContent(type="text", text="Error: delete needs memory_id")]
|
||||
memories = _memory_manager.load_all()
|
||||
return _text_result("Error: delete needs memory_id")
|
||||
_owner, memories, visible, scope_error = _scope_entries()
|
||||
if scope_error:
|
||||
return _text_result(scope_error)
|
||||
full_id = None
|
||||
deleted_text = ""
|
||||
deleted_category = ""
|
||||
for m in memories:
|
||||
for m in visible:
|
||||
if m.get("id", "").startswith(memory_id):
|
||||
full_id = m["id"]
|
||||
deleted_text = m.get("text", "")
|
||||
deleted_category = m.get("category", "")
|
||||
break
|
||||
if not full_id:
|
||||
return [TextContent(type="text", text=f"Error: Memory '{memory_id}' not found")]
|
||||
return _text_result(f"Error: Memory '{memory_id}' not found")
|
||||
memories = [m for m in memories if m.get("id") != full_id]
|
||||
_memory_manager.save(memories)
|
||||
if _memory_vector and _memory_vector.healthy and full_id:
|
||||
@@ -171,30 +230,32 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
|
||||
pass
|
||||
cat = f"[{deleted_category}] " if deleted_category else ""
|
||||
snippet = deleted_text if len(deleted_text) <= 120 else deleted_text[:117] + "..."
|
||||
return [TextContent(type="text", text=f"Memory deleted: {cat}{snippet} (id: {memory_id})")]
|
||||
return _text_result(f"Memory deleted: {cat}{snippet} (id: {memory_id})")
|
||||
|
||||
elif action == "search":
|
||||
query = arguments.get("text", "")
|
||||
if not query:
|
||||
return [TextContent(type="text", text="Error: search needs text (query)")]
|
||||
memories = _memory_manager.load()
|
||||
return _text_result("Error: search needs text (query)")
|
||||
_owner, _all_memories, memories, scope_error = _scope_entries()
|
||||
if scope_error:
|
||||
return _text_result(scope_error)
|
||||
if hasattr(_memory_manager, 'get_relevant_memories'):
|
||||
results = _memory_manager.get_relevant_memories(query, memories, threshold=0.05, max_items=20)
|
||||
else:
|
||||
query_lower = query.lower()
|
||||
results = [m for m in memories if query_lower in m.get("text", "").lower()][:20]
|
||||
if not results:
|
||||
return [TextContent(type="text", text=f"No memories found matching '{query}'.")]
|
||||
return _text_result(f"No memories found matching '{query}'.")
|
||||
lines = [f"Found {len(results)} matching memories:\n"]
|
||||
for m in results:
|
||||
cat = m.get("category", "fact")
|
||||
mid = m.get("id", "?")[:8]
|
||||
text = m.get("text", "")
|
||||
lines.append(f"- [{cat}] `{mid}` — {text}")
|
||||
return [TextContent(type="text", text="\n".join(lines))]
|
||||
return _text_result("\n".join(lines))
|
||||
|
||||
else:
|
||||
return [TextContent(type="text", text=f"Error: Unknown action '{action}'. Use: list, add, edit, delete, search")]
|
||||
return _text_result(f"Error: Unknown action '{action}'. Use: list, add, edit, delete, search")
|
||||
|
||||
|
||||
async def run():
|
||||
|
||||
@@ -160,6 +160,8 @@ def setup_api_token_routes() -> APIRouter:
|
||||
payload = await request.json()
|
||||
except Exception:
|
||||
payload = {}
|
||||
if not isinstance(payload, dict):
|
||||
payload = {}
|
||||
with get_db_session() as db:
|
||||
token = db.query(ApiToken).filter(ApiToken.id == token_id).first()
|
||||
if not token:
|
||||
|
||||
@@ -14,7 +14,7 @@ from core.database import Session as DBSession, ModelEndpoint
|
||||
from src.llm_core import normalize_model_id
|
||||
from src.endpoint_resolver import normalize_base
|
||||
from src.context_compactor import maybe_compact, trim_for_context
|
||||
from src.auth_helpers import get_current_user
|
||||
from src.auth_helpers import effective_user
|
||||
from src.prompt_security import untrusted_context_message
|
||||
from routes.prefs_routes import _load_for_user as load_prefs_for_user
|
||||
|
||||
@@ -78,7 +78,7 @@ def _enforce_chat_privileges(request, sess) -> None:
|
||||
which means unrestricted allowed_models / zero cap -> no-op for them.
|
||||
"""
|
||||
try:
|
||||
user = get_current_user(request)
|
||||
user = effective_user(request)
|
||||
except Exception:
|
||||
user = None
|
||||
if not user:
|
||||
@@ -346,11 +346,11 @@ def add_user_message(sess, chat_handler, preprocessed: PreprocessedMessage, inco
|
||||
def fire_message_event(request, webhook_manager, session_id: str, sess, message: str, compare_mode: bool = False):
|
||||
"""Fire webhook and event_bus events for a new user message."""
|
||||
if webhook_manager and not compare_mode:
|
||||
asyncio.create_task(webhook_manager.fire("chat.message", {
|
||||
webhook_manager.fire_and_forget("chat.message", {
|
||||
"session_id": session_id, "model": sess.model, "message": message[:2000],
|
||||
}))
|
||||
})
|
||||
from src.event_bus import fire_event
|
||||
user = get_current_user(request)
|
||||
user = effective_user(request)
|
||||
fire_event("message_sent", user)
|
||||
|
||||
|
||||
@@ -577,7 +577,7 @@ async def build_chat_context(
|
||||
fire_message_event(request, webhook_manager, session_id, sess, message, compare_mode)
|
||||
|
||||
# Resolve user prefs
|
||||
user = get_current_user(request)
|
||||
user = effective_user(request)
|
||||
uprefs = load_prefs_for_user(user)
|
||||
|
||||
# Memory enabled?
|
||||
@@ -1120,10 +1120,10 @@ def run_post_response_tasks(
|
||||
|
||||
# Webhook
|
||||
if webhook_manager and not compare_mode:
|
||||
asyncio.create_task(webhook_manager.fire("chat.completed", {
|
||||
webhook_manager.fire_and_forget("chat.completed", {
|
||||
"session_id": session_id, "model": sess.model,
|
||||
"user_message": message, "response": full_response[:2000],
|
||||
}))
|
||||
})
|
||||
|
||||
# Auto-name
|
||||
if needs_auto_name(sess.name):
|
||||
|
||||
@@ -23,7 +23,7 @@ from src.endpoint_resolver import normalize_base as _normalize_base, build_chat_
|
||||
from src.session_search import search_session_messages
|
||||
from src.prompt_security import untrusted_context_message
|
||||
from core.exceptions import SessionNotFoundError
|
||||
from src.auth_helpers import get_current_user
|
||||
from src.auth_helpers import effective_user, get_current_user
|
||||
from routes.session_routes import _verify_session_owner
|
||||
from routes.document_helpers import _owner_session_filter
|
||||
from core.database import SessionLocal, get_session_mode, set_session_mode
|
||||
@@ -126,7 +126,8 @@ def _clear_orphaned_session_endpoint(sess, owner: str | None = None) -> bool:
|
||||
sess.model = ""
|
||||
sess.headers = {}
|
||||
return True
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
logger.warning("Failed to clear orphaned session endpoint", exc_info=e)
|
||||
db.rollback()
|
||||
return False
|
||||
finally:
|
||||
@@ -144,7 +145,8 @@ def _endpoint_cache_contains_model(endpoint, model: str) -> bool:
|
||||
return True
|
||||
try:
|
||||
models = json.loads(raw) if isinstance(raw, str) else raw
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
logger.warning("Failed to parse cached models list, treating as containing model", exc_info=e)
|
||||
return True
|
||||
if not isinstance(models, list) or not models:
|
||||
return True
|
||||
@@ -236,7 +238,8 @@ def _recover_empty_session_model(sess, session_id: str, owner: str | None = None
|
||||
is_chatgpt_subscription = False
|
||||
try:
|
||||
cached = json.loads(ep.cached_models) if isinstance(ep.cached_models, str) else (ep.cached_models or [])
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
logger.warning("Failed to parse cached_models for endpoint %r", getattr(ep, "id", "?"), exc_info=e)
|
||||
cached = []
|
||||
if not cached:
|
||||
visible = []
|
||||
@@ -360,7 +363,7 @@ def setup_chat_routes(
|
||||
sess = session_manager.get_session(session)
|
||||
except KeyError:
|
||||
raise HTTPException(404, f"Session '{session}' not found")
|
||||
owner = get_current_user(request)
|
||||
owner = effective_user(request)
|
||||
if _clear_orphaned_session_endpoint(sess, owner=owner):
|
||||
raise HTTPException(400, "Selected model endpoint was removed. Pick another model in Settings.")
|
||||
|
||||
@@ -600,7 +603,7 @@ def setup_chat_routes(
|
||||
# but BEFORE loading. Prevents cross-user session hijack.
|
||||
_verify_session_owner(request, session)
|
||||
sess = session_manager.get_session(session)
|
||||
owner = get_current_user(request)
|
||||
owner = effective_user(request)
|
||||
if _clear_orphaned_session_endpoint(sess, owner=owner):
|
||||
raise HTTPException(400, "Selected model endpoint was removed. Pick another model in Settings.")
|
||||
# Issue #587: picker shows a model from the endpoint cache but
|
||||
@@ -631,7 +634,7 @@ def setup_chat_routes(
|
||||
_enforce_chat_privileges(request, sess)
|
||||
|
||||
# Ensure session has auth headers
|
||||
resolve_session_auth(sess, session, owner=get_current_user(request))
|
||||
resolve_session_auth(sess, session, owner=effective_user(request))
|
||||
|
||||
# Check for research_pending BEFORE mode persist overwrites it
|
||||
do_research = str(use_research).lower() == "true"
|
||||
@@ -646,8 +649,8 @@ def setup_chat_routes(
|
||||
elif attachments:
|
||||
try:
|
||||
att_ids = [str(x) for x in json.loads(attachments)]
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning("Failed to parse attachments JSON, ignoring attachments", exc_info=e)
|
||||
|
||||
no_memory = str(form_data.get("no_memory", "")).lower() == "true"
|
||||
pre_context_tool_policy = build_effective_tool_policy(
|
||||
@@ -1482,7 +1485,7 @@ def setup_chat_routes(
|
||||
if not q or not q.strip():
|
||||
return []
|
||||
|
||||
_user = get_current_user(request)
|
||||
_user = effective_user(request)
|
||||
return [
|
||||
result.to_dict()
|
||||
for result in search_session_messages(
|
||||
|
||||
@@ -790,7 +790,7 @@ def setup_codex_routes(
|
||||
norm = dict(body or {})
|
||||
sess = (norm.get("tmux_session") or norm.get("session_id") or "").strip()
|
||||
model = (norm.get("model") or norm.get("repo_id") or "").strip()
|
||||
host = (norm.get("host") or norm.get("remote_host") or "").strip()
|
||||
host = validate_remote_host((norm.get("host") or norm.get("remote_host") or "").strip() or None) or ""
|
||||
port = norm.get("port") or 8000
|
||||
import re as _re
|
||||
if not sess or not _re.fullmatch(r"[a-zA-Z0-9_-]+", sess):
|
||||
|
||||
@@ -505,6 +505,8 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
|
||||
" if u.startswith('KB'): return int(n * 1024)",
|
||||
" return int(n)",
|
||||
"def scan_ollama():",
|
||||
" if any(m.get('is_ollama') for m in models): return",
|
||||
" if os.name == 'nt' and not os.environ.get('ODYSSEUS_ALLOW_OLLAMA_CLI_SCAN'): return",
|
||||
" if not shutil.which('ollama'): return",
|
||||
" try:",
|
||||
" p = subprocess.run(['ollama', 'list'], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, timeout=6)",
|
||||
@@ -535,8 +537,8 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
|
||||
" models.append({'repo_id':name,'size_bytes':size_bytes,'nb_files':1,'has_incomplete':False,'path':'ollama','backend':'ollama','is_ollama':True})",
|
||||
" return",
|
||||
"for _hf_cache in hf_cache_paths(): scan_hf(_hf_cache)",
|
||||
"scan_ollama()",
|
||||
"scan_ollama_api()",
|
||||
"scan_ollama()",
|
||||
]
|
||||
for model_dir in model_dirs or []:
|
||||
lines.append(f"scan_dir(os.path.expanduser({model_dir!r}))")
|
||||
|
||||
@@ -503,7 +503,8 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
|
||||
user = get_current_user(request)
|
||||
try:
|
||||
data = await request.json()
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
logger.warning("Failed to parse export request body, defaulting to empty", exc_info=e)
|
||||
data = {}
|
||||
ids = data.get("ids") or []
|
||||
if not ids:
|
||||
@@ -645,8 +646,8 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
|
||||
try:
|
||||
from src.agent_tools.document_tools import clear_active_document
|
||||
clear_active_document(doc_id)
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning("Failed to clear active document %r on detach", doc_id, exc_info=e)
|
||||
db.commit()
|
||||
db.refresh(doc)
|
||||
return _doc_to_dict(doc)
|
||||
|
||||
@@ -13,6 +13,8 @@ and `email_pollers.py` (the background loops):
|
||||
"""
|
||||
|
||||
import os
|
||||
import base64
|
||||
import time
|
||||
import imaplib
|
||||
import smtplib
|
||||
import email as email_mod
|
||||
@@ -38,6 +40,106 @@ from src.secret_storage import decrypt as _decrypt
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _xoauth2_raw(user: str, access_token: str) -> str:
|
||||
"""The SASL XOAUTH2 initial-response string (unencoded).
|
||||
|
||||
Both smtplib.SMTP.auth() and imaplib.IMAP4.authenticate() base64-encode
|
||||
the value their callback returns, so callers pass this raw form — never
|
||||
pre-encoded — to avoid double base64.
|
||||
"""
|
||||
return f"user={user}\x01auth=Bearer {access_token}\x01\x01"
|
||||
|
||||
|
||||
def _xoauth2_bytes(user: str, access_token: str) -> bytes:
|
||||
"""Raw XOAUTH2 bytes for imaplib's authenticate() callback."""
|
||||
return _xoauth2_raw(user, access_token).encode()
|
||||
|
||||
|
||||
def make_oauth_state(account_id: str, owner: str) -> str:
|
||||
"""Return an HMAC-signed, base64-encoded OAuth state token.
|
||||
|
||||
Encodes account_id + owner + a random nonce, signed with the app secret
|
||||
so the callback can validate that the flow was initiated by an
|
||||
authenticated, owning user (CSRF / state-forgery protection).
|
||||
"""
|
||||
import hmac as _hmac, hashlib as _hl, secrets as _sec
|
||||
from src.secret_storage import _load_or_create_key
|
||||
nonce = _sec.token_hex(16)
|
||||
payload = json.dumps({"a": account_id, "o": owner, "n": nonce}, separators=(",", ":"))
|
||||
sig = _hmac.new(_load_or_create_key(), payload.encode(), _hl.sha256).hexdigest()
|
||||
return base64.urlsafe_b64encode(f"{payload}|{sig}".encode()).decode()
|
||||
|
||||
|
||||
def verify_oauth_state(state: str) -> dict | None:
|
||||
"""Verify an OAuth state token's HMAC signature.
|
||||
|
||||
Returns the decoded payload dict ({"a", "o", "n"}) on success, or None if
|
||||
the token is malformed, tampered, or signed with a different key.
|
||||
"""
|
||||
import hmac as _hmac, hashlib as _hl
|
||||
from src.secret_storage import _load_or_create_key
|
||||
try:
|
||||
decoded = base64.urlsafe_b64decode(state.encode()).decode()
|
||||
payload, sig = decoded.rsplit("|", 1)
|
||||
expected = _hmac.new(_load_or_create_key(), payload.encode(), _hl.sha256).hexdigest()
|
||||
if not _hmac.compare_digest(sig, expected):
|
||||
return None
|
||||
return json.loads(payload)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _refresh_google_token(account_id: str) -> str | None:
|
||||
"""Exchange the stored refresh token for a new access token and persist it."""
|
||||
import httpx
|
||||
from core.database import SessionLocal as _SL, EmailAccount as _EA
|
||||
from src.secret_storage import encrypt as _enc, decrypt as _dec
|
||||
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
|
||||
client_secret = os.environ.get("GOOGLE_OAUTH_CLIENT_SECRET", "")
|
||||
if not client_id or not client_secret:
|
||||
return None
|
||||
db = _SL()
|
||||
try:
|
||||
row = db.get(_EA, account_id)
|
||||
if not row or not row.oauth_refresh_token:
|
||||
return None
|
||||
refresh_token = _dec(row.oauth_refresh_token or "")
|
||||
if not refresh_token:
|
||||
return None
|
||||
resp = httpx.post("https://oauth2.googleapis.com/token", data={
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"refresh_token": refresh_token,
|
||||
"grant_type": "refresh_token",
|
||||
}, timeout=10)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
access_token = data["access_token"]
|
||||
row.oauth_access_token = _enc(access_token)
|
||||
row.oauth_token_expiry = str(int(time.time()) + data.get("expires_in", 3600))
|
||||
db.commit()
|
||||
return access_token
|
||||
except Exception:
|
||||
logger.warning(f"Google token refresh failed for account {account_id}")
|
||||
return None
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def _get_valid_google_token(account_id: str, cfg: dict) -> str | None:
|
||||
"""Return a valid Google access token, refreshing if expired or missing."""
|
||||
from src.secret_storage import decrypt as _dec
|
||||
access_token = _dec(cfg.get("oauth_access_token") or "")
|
||||
expiry_str = cfg.get("oauth_token_expiry") or ""
|
||||
if access_token and expiry_str:
|
||||
try:
|
||||
if int(expiry_str) - 60 > time.time():
|
||||
return access_token
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
return _refresh_google_token(account_id)
|
||||
|
||||
|
||||
def _smtp_security_mode(cfg: dict) -> str:
|
||||
raw = str(cfg.get("smtp_security") or "").strip().lower()
|
||||
if raw in {"ssl", "starttls", "none"}:
|
||||
@@ -54,20 +156,29 @@ def _send_smtp_message(cfg: dict, from_addr: str, recipients: list[str], message
|
||||
port = int(cfg.get("smtp_port") or 465)
|
||||
user = cfg.get("smtp_user") or ""
|
||||
password = cfg.get("smtp_password") or ""
|
||||
|
||||
def _auth_smtp(smtp):
|
||||
if cfg.get("oauth_provider") == "google":
|
||||
token = _get_valid_google_token(cfg.get("account_id"), cfg)
|
||||
if not token:
|
||||
raise RuntimeError("Google OAuth token unavailable — reconnect the account")
|
||||
smtp.ehlo()
|
||||
smtp.auth("XOAUTH2", lambda challenge=None: _xoauth2_raw(user, token), initial_response_ok=True)
|
||||
elif user and password:
|
||||
smtp.login(user, password)
|
||||
|
||||
security = _smtp_security_mode(cfg)
|
||||
|
||||
if security == "ssl":
|
||||
with smtplib.SMTP_SSL(host, port, timeout=timeout) as smtp:
|
||||
if user and password:
|
||||
smtp.login(user, password)
|
||||
_auth_smtp(smtp)
|
||||
smtp.sendmail(from_addr, recipients, message)
|
||||
return
|
||||
|
||||
with smtplib.SMTP(host, port, timeout=timeout) as smtp:
|
||||
if security == "starttls":
|
||||
smtp.starttls()
|
||||
if user and password:
|
||||
smtp.login(user, password)
|
||||
_auth_smtp(smtp)
|
||||
smtp.sendmail(from_addr, recipients, message)
|
||||
|
||||
|
||||
@@ -701,10 +812,16 @@ def _get_email_config(account_id: str | None = None, owner: str = "") -> dict:
|
||||
"imap_password": _decrypt(row.imap_password or ""),
|
||||
"imap_starttls": bool(row.imap_starttls),
|
||||
"from_address": row.from_address or row.imap_user or "",
|
||||
"oauth_provider": row.oauth_provider or "",
|
||||
"oauth_access_token": row.oauth_access_token or "",
|
||||
"oauth_refresh_token": row.oauth_refresh_token or "",
|
||||
"oauth_token_expiry": row.oauth_token_expiry or "",
|
||||
"display_name": row.display_name or "",
|
||||
}
|
||||
if not (cfg["smtp_host"] and cfg["smtp_user"] and cfg["smtp_password"]):
|
||||
is_oauth = bool(cfg.get("oauth_provider"))
|
||||
if not is_oauth and not (cfg["smtp_host"] and cfg["smtp_user"] and cfg["smtp_password"]):
|
||||
logger.warning(f"SMTP not configured for account {row.name!r}")
|
||||
if not (cfg["imap_host"] and cfg["imap_user"] and cfg["imap_password"]):
|
||||
if not is_oauth and not (cfg["imap_host"] and cfg["imap_user"] and cfg["imap_password"]):
|
||||
logger.warning(f"IMAP not configured for account {row.name!r}")
|
||||
return cfg
|
||||
finally:
|
||||
@@ -825,12 +942,19 @@ def _imap_connect(account_id: str | None = None, owner: str = "",
|
||||
timeout=timeout,
|
||||
)
|
||||
try:
|
||||
conn.login(cfg["imap_user"], cfg["imap_password"])
|
||||
if cfg.get("oauth_provider") == "google":
|
||||
token = _get_valid_google_token(cfg.get("account_id"), cfg)
|
||||
if not token:
|
||||
raise RuntimeError("Google OAuth token unavailable — reconnect the account in Settings → Integrations")
|
||||
conn.authenticate("XOAUTH2", lambda x: _xoauth2_bytes(cfg["imap_user"], token))
|
||||
else:
|
||||
conn.login(cfg["imap_user"], cfg["imap_password"])
|
||||
except Exception:
|
||||
# A failed AUTHENTICATE (e.g. an Office 365 app password on an
|
||||
# MFA-enabled tenant, #3174) otherwise orphans the already-connected
|
||||
# socket; close it before propagating so a misconfigured account
|
||||
# can't leak one descriptor per retry / background poller pass.
|
||||
# MFA-enabled tenant, #3174, or an expired/revoked OAuth token)
|
||||
# otherwise orphans the already-connected socket; close it before
|
||||
# propagating so a misconfigured account can't leak one descriptor
|
||||
# per retry / background poller pass.
|
||||
try:
|
||||
conn.shutdown()
|
||||
except Exception:
|
||||
|
||||
@@ -13,7 +13,9 @@ handlers need. The split is mechanical — no behavior change.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sqlite3 as _sql3
|
||||
import time
|
||||
import email as email_mod
|
||||
import email.header
|
||||
import email.utils
|
||||
@@ -43,6 +45,7 @@ from routes.email_helpers import (
|
||||
_load_settings, _save_settings, _get_email_config,
|
||||
_send_smtp_message, _smtp_security_mode,
|
||||
_IMAP_TIMEOUT_SECONDS, _open_imap_connection,
|
||||
make_oauth_state, verify_oauth_state,
|
||||
_imap_connect, _imap, _decode_header, _detect_sent_folder, _detect_drafts_folder,
|
||||
_extract_attachment_text, _list_attachments_from_msg,
|
||||
_extract_attachment_to_disk, _extract_html, _extract_text,
|
||||
@@ -76,15 +79,16 @@ def _email_tag_owner_aliases(account_id: str | None, owner: str = "") -> list[st
|
||||
cfg.get("smtp_user") or "",
|
||||
cfg.get("from_address") or "",
|
||||
])
|
||||
except Exception:
|
||||
except Exception as _e:
|
||||
logger.warning("Failed to resolve email account alias", exc_info=_e)
|
||||
resolved_account_id = None
|
||||
row = db.get(_EA, resolved_account_id) if resolved_account_id else None
|
||||
if row:
|
||||
aliases.extend([row.owner or "", row.imap_user or "", row.from_address or ""])
|
||||
finally:
|
||||
db.close()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as _e:
|
||||
logger.warning("Failed to load email aliases", exc_info=_e)
|
||||
out = []
|
||||
for a in aliases:
|
||||
a = (a or "").strip()
|
||||
@@ -285,7 +289,9 @@ def _group_uid_fetch_records(msg_data) -> list:
|
||||
|
||||
|
||||
def _smtp_ready(cfg: dict) -> bool:
|
||||
return bool(cfg.get("smtp_host") and cfg.get("smtp_user") and cfg.get("smtp_password"))
|
||||
if not cfg.get("smtp_host") or not cfg.get("smtp_user"):
|
||||
return False
|
||||
return bool(cfg.get("smtp_password") or cfg.get("oauth_provider"))
|
||||
|
||||
|
||||
def _resolve_send_config(account_id: str | None = None, owner: str = "") -> dict:
|
||||
@@ -2021,7 +2027,7 @@ def setup_email_routes():
|
||||
outer = MIMEMultipart("alternative")
|
||||
body_container = outer
|
||||
|
||||
outer["From"] = cfg["from_address"]
|
||||
outer["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
|
||||
outer["To"] = to
|
||||
if cc:
|
||||
outer["Cc"] = cc
|
||||
@@ -2285,6 +2291,7 @@ def setup_email_routes():
|
||||
try:
|
||||
cfg = _resolve_send_config(req.account_id, owner=owner)
|
||||
except Exception as e:
|
||||
logger.warning(f"No SMTP-capable account resolved: {e}")
|
||||
return {"success": False, "error": str(e) or "No SMTP-capable email account configured"}
|
||||
|
||||
# Use 'mixed' if we have attachments, 'alternative' otherwise
|
||||
@@ -2297,7 +2304,7 @@ def setup_email_routes():
|
||||
outer = MIMEMultipart("alternative")
|
||||
body_container = outer
|
||||
|
||||
outer["From"] = cfg["from_address"]
|
||||
outer["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
|
||||
outer["To"] = req.to
|
||||
if req.cc:
|
||||
outer["Cc"] = req.cc
|
||||
@@ -2348,6 +2355,10 @@ def setup_email_routes():
|
||||
|
||||
_account_id = cfg.get("account_id") or req.account_id # capture for the IMAP append in the closure
|
||||
_in_reply_to = (req.in_reply_to or "").strip()
|
||||
_oauth_provider = cfg.get("oauth_provider") or ""
|
||||
_oauth_access_token = cfg.get("oauth_access_token") or ""
|
||||
_oauth_refresh_token = cfg.get("oauth_refresh_token") or ""
|
||||
_oauth_token_expiry = cfg.get("oauth_token_expiry") or ""
|
||||
|
||||
def _deliver():
|
||||
try:
|
||||
@@ -2358,6 +2369,11 @@ def setup_email_routes():
|
||||
"smtp_security": _smtp_security,
|
||||
"smtp_user": _smtp_user,
|
||||
"smtp_password": _smtp_pw,
|
||||
"account_id": _account_id,
|
||||
"oauth_provider": _oauth_provider,
|
||||
"oauth_access_token": _oauth_access_token,
|
||||
"oauth_refresh_token": _oauth_refresh_token,
|
||||
"oauth_token_expiry": _oauth_token_expiry,
|
||||
},
|
||||
_from,
|
||||
_recipients,
|
||||
@@ -2470,7 +2486,7 @@ def setup_email_routes():
|
||||
msg.attach(MIMEText(_draft_html, "html", "utf-8"))
|
||||
else:
|
||||
msg = MIMEText(req.body, "plain", "utf-8")
|
||||
msg["From"] = cfg["from_address"]
|
||||
msg["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
|
||||
msg["To"] = req.to
|
||||
if req.cc:
|
||||
msg["Cc"] = req.cc
|
||||
@@ -3122,6 +3138,8 @@ def setup_email_routes():
|
||||
"from_address": r.from_address or "",
|
||||
"has_imap_password": bool(r.imap_password),
|
||||
"has_smtp_password": bool(r.smtp_password),
|
||||
"oauth_provider": r.oauth_provider or "",
|
||||
"display_name": r.display_name or "",
|
||||
})
|
||||
return {"accounts": out}
|
||||
finally:
|
||||
@@ -3154,6 +3172,7 @@ def setup_email_routes():
|
||||
smtp_user=(data.get("smtp_user") or "").strip(),
|
||||
smtp_password=_enc(data.get("smtp_password") or ""),
|
||||
from_address=(data.get("from_address") or "").strip(),
|
||||
display_name=(data.get("display_name") or "").strip(),
|
||||
# SECURITY: stamp the creator so all subsequent reads / mutations
|
||||
# can filter by user. Without this every new account leaks to
|
||||
# every other user.
|
||||
@@ -3188,7 +3207,7 @@ def setup_email_routes():
|
||||
if not row:
|
||||
return {"ok": False, "error": "Account not found"}
|
||||
# Simple fields
|
||||
for key in ("name", "imap_host", "imap_user", "smtp_host", "smtp_user", "from_address"):
|
||||
for key in ("name", "imap_host", "imap_user", "smtp_host", "smtp_user", "from_address", "display_name"):
|
||||
if key in data:
|
||||
setattr(row, key, (data[key] or "").strip())
|
||||
for key in ("imap_port", "smtp_port"):
|
||||
@@ -3377,4 +3396,123 @@ def setup_email_routes():
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
# ── Google OAuth2 routes ──
|
||||
|
||||
@router.get("/oauth/google/authorize")
|
||||
async def google_oauth_authorize(account_id: str = Query(...), request: Request = None, owner: str = Depends(require_user)):
|
||||
import urllib.parse
|
||||
_assert_owns_account(account_id, owner)
|
||||
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
|
||||
if not client_id:
|
||||
raise HTTPException(400, "GOOGLE_OAUTH_CLIENT_ID not set — add it to .env")
|
||||
redirect_uri = (
|
||||
os.environ.get("GOOGLE_OAUTH_REDIRECT_URI")
|
||||
or f"http://{request.headers.get('host', 'localhost:7000')}/api/email/oauth/google/callback"
|
||||
)
|
||||
state = make_oauth_state(account_id, owner)
|
||||
params = urllib.parse.urlencode({
|
||||
"client_id": client_id,
|
||||
"redirect_uri": redirect_uri,
|
||||
"response_type": "code",
|
||||
"scope": "https://mail.google.com/ email",
|
||||
"access_type": "offline",
|
||||
"prompt": "consent",
|
||||
"state": state,
|
||||
})
|
||||
from fastapi.responses import RedirectResponse as _RR
|
||||
return _RR(f"https://accounts.google.com/o/oauth2/v2/auth?{params}")
|
||||
|
||||
@router.get("/oauth/google/callback")
|
||||
async def google_oauth_callback(
|
||||
code: str = Query(None),
|
||||
state: str = Query(None),
|
||||
error: str = Query(None),
|
||||
request: Request = None,
|
||||
):
|
||||
import urllib.parse
|
||||
from fastapi.responses import RedirectResponse as _RR
|
||||
if error:
|
||||
return _RR("/?section=integrations&email_oauth_error=google_error")
|
||||
if not code or not state:
|
||||
return _RR("/?section=integrations&email_oauth_error=missing_code")
|
||||
state_data = verify_oauth_state(state)
|
||||
if not state_data:
|
||||
return _RR("/?section=integrations&email_oauth_error=invalid_state")
|
||||
account_id = state_data.get("a", "")
|
||||
owner = state_data.get("o", "")
|
||||
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
|
||||
client_secret = os.environ.get("GOOGLE_OAUTH_CLIENT_SECRET", "")
|
||||
redirect_uri = (
|
||||
os.environ.get("GOOGLE_OAUTH_REDIRECT_URI")
|
||||
or f"http://{request.headers.get('host', 'localhost:7000')}/api/email/oauth/google/callback"
|
||||
)
|
||||
import httpx as _httpx
|
||||
try:
|
||||
resp = _httpx.post("https://oauth2.googleapis.com/token", data={
|
||||
"code": code,
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"redirect_uri": redirect_uri,
|
||||
"grant_type": "authorization_code",
|
||||
}, timeout=10)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
except Exception:
|
||||
logger.warning("Google token exchange failed")
|
||||
return _RR("/?section=integrations&email_oauth_error=token_exchange_failed")
|
||||
access_token = data.get("access_token", "")
|
||||
refresh_token = data.get("refresh_token", "")
|
||||
expiry = str(int(time.time()) + data.get("expires_in", 3600))
|
||||
# Fetch the email address from userinfo so we can auto-fill imap_user.
|
||||
email_addr = ""
|
||||
display_name = ""
|
||||
try:
|
||||
ui = _httpx.get("https://www.googleapis.com/oauth2/v1/userinfo",
|
||||
headers={"Authorization": f"Bearer {access_token}"}, timeout=10)
|
||||
if ui.is_success:
|
||||
ui_data = ui.json()
|
||||
email_addr = ui_data.get("email", "")
|
||||
display_name = ui_data.get("name", "")
|
||||
except Exception:
|
||||
pass
|
||||
from core.database import SessionLocal, EmailAccount
|
||||
from src.secret_storage import encrypt as _enc
|
||||
db = SessionLocal()
|
||||
try:
|
||||
row = db.query(EmailAccount).filter(EmailAccount.id == account_id).first()
|
||||
if not row:
|
||||
return _RR("/?section=integrations&email_oauth_error=account_not_found")
|
||||
# SECURITY: verify the account belongs to the initiating user.
|
||||
if owner and row.owner and row.owner != owner:
|
||||
logger.warning("OAuth callback owner mismatch — rejecting token write")
|
||||
return _RR("/?section=integrations&email_oauth_error=ownership_error")
|
||||
row.oauth_provider = "google"
|
||||
row.oauth_access_token = _enc(access_token)
|
||||
if refresh_token:
|
||||
row.oauth_refresh_token = _enc(refresh_token)
|
||||
row.oauth_token_expiry = expiry
|
||||
# Auto-fill Google IMAP/SMTP settings if not already configured.
|
||||
if not row.imap_host:
|
||||
row.imap_host = "imap.gmail.com"
|
||||
row.imap_port = 993
|
||||
row.imap_starttls = False
|
||||
if not row.smtp_host:
|
||||
row.smtp_host = "smtp.gmail.com"
|
||||
row.smtp_port = 587
|
||||
if email_addr:
|
||||
if not row.imap_user:
|
||||
row.imap_user = email_addr
|
||||
if not row.smtp_user:
|
||||
row.smtp_user = email_addr
|
||||
if not row.from_address:
|
||||
row.from_address = email_addr
|
||||
if not row.name or row.name == row.id:
|
||||
row.name = email_addr
|
||||
if display_name and not row.display_name:
|
||||
row.display_name = display_name
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
return _RR("/?section=integrations&email_oauth_success=1")
|
||||
|
||||
return router
|
||||
|
||||
@@ -9,6 +9,7 @@ from pathlib import Path
|
||||
from fastapi import APIRouter, HTTPException, Form, Depends
|
||||
from core.constants import EMBEDDING_ENDPOINT_FILE, FASTEMBED_CACHE_DIR
|
||||
from core.middleware import require_admin
|
||||
from src.runtime_paths import get_app_root
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -232,8 +232,6 @@ def setup_gallery_routes() -> APIRouter:
|
||||
@router.post("/api/gallery/{image_id}/replace")
|
||||
async def gallery_replace(request: Request, image_id: str):
|
||||
"""Replace an existing gallery image file with a new one."""
|
||||
from pathlib import Path
|
||||
|
||||
user = get_current_user(request)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
@@ -249,9 +247,8 @@ def setup_gallery_routes() -> APIRouter:
|
||||
raise HTTPException(400, "No image provided")
|
||||
|
||||
content = await read_upload_limited(file, GALLERY_UPLOAD_MAX_BYTES, "Gallery replacement")
|
||||
img_dir = Path(GENERATED_IMAGES_DIR)
|
||||
img_dir.mkdir(parents=True, exist_ok=True)
|
||||
img_path = img_dir / _sanitize_gallery_filename(img.filename)
|
||||
GALLERY_IMAGE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
img_path = _gallery_image_path(img.filename)
|
||||
img_path.write_bytes(content)
|
||||
|
||||
# Refresh dimensions in case the editor resized the canvas.
|
||||
|
||||
@@ -26,7 +26,7 @@ from src.endpoint_resolver import (
|
||||
build_models_url,
|
||||
build_headers,
|
||||
)
|
||||
from src.auth_helpers import _auth_disabled, owner_filter
|
||||
from src.auth_helpers import _auth_disabled, effective_user, owner_filter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -1255,13 +1255,16 @@ def setup_model_routes(model_discovery):
|
||||
# Require auth; "" is the unconfigured single-user mode, treated as
|
||||
# "see everything" by _fetch_models.
|
||||
try:
|
||||
from src.auth_helpers import get_current_user as _gcu
|
||||
owner = _gcu(request) or ""
|
||||
except Exception:
|
||||
owner = ""
|
||||
# Reject anonymous in configured deployments — no leaking the model
|
||||
# list to unauthenticated callers.
|
||||
try:
|
||||
if getattr(request.state, "api_token", False):
|
||||
scopes = set(getattr(request.state, "api_token_scopes", []) or [])
|
||||
if "chat" not in scopes:
|
||||
raise HTTPException(403, "API token is not scoped for chat")
|
||||
if not getattr(request.state, "api_token_owner", None):
|
||||
raise HTTPException(403, "API token has no owner")
|
||||
owner = effective_user(request) or ""
|
||||
|
||||
# Reject anonymous in configured deployments — no leaking the model
|
||||
# list to unauthenticated callers.
|
||||
auth_mgr = getattr(request.app.state, "auth_manager", None)
|
||||
if not owner and not _auth_disabled() and auth_mgr is not None and getattr(auth_mgr, "is_configured", False):
|
||||
raise HTTPException(401, "Not authenticated")
|
||||
|
||||
@@ -10,7 +10,7 @@ from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core.database import SessionLocal, Note
|
||||
from src.auth_helpers import get_current_user
|
||||
from src.auth_helpers import require_user
|
||||
from src.constants import DATA_DIR
|
||||
from sqlalchemy.orm.attributes import flag_modified
|
||||
|
||||
@@ -570,7 +570,16 @@ def setup_note_routes(task_scheduler=None):
|
||||
router = APIRouter(prefix="/api/notes", tags=["notes"])
|
||||
|
||||
def _owner(request: Request) -> Optional[str]:
|
||||
return get_current_user(request)
|
||||
# require_user, not bare get_current_user: a request that reaches
|
||||
# these owner-scoped routes with NO identity (auth-middleware
|
||||
# regression, SSRF from a sibling service) must fail closed (401)
|
||||
# when auth is configured — not be treated as the single-user mode
|
||||
# and handed blanket access to every account's notes. The documented
|
||||
# anonymous modes (AUTH_ENABLED=false, LOCALHOST_BYPASS on loopback,
|
||||
# unconfigured first-run) still resolve to None, the single-user
|
||||
# path. fire_reminder below already gated this way; the CRUD routes
|
||||
# did not.
|
||||
return require_user(request) or None
|
||||
|
||||
def _is_admin_or_single_user(request: Request, user: str | None) -> bool:
|
||||
if user == "internal-tool":
|
||||
@@ -805,8 +814,7 @@ def setup_note_routes(task_scheduler=None):
|
||||
Returns {synthesis, email_sent}.
|
||||
"""
|
||||
# Gate against anonymous callers — LLM synthesis can burn tokens.
|
||||
from src.auth_helpers import require_user as _ru
|
||||
user = _ru(request)
|
||||
user = require_user(request)
|
||||
body = await request.json()
|
||||
note_id = str(body.get("note_id") or "").strip()
|
||||
if not note_id:
|
||||
|
||||
@@ -278,8 +278,8 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
|
||||
# Delete file from disk if it's in uploads dir
|
||||
deleted_from_disk = False
|
||||
try:
|
||||
abs_target = os.path.abspath(filepath)
|
||||
base_abs = os.path.abspath(UPLOADS_DIR)
|
||||
abs_target = os.path.realpath(filepath)
|
||||
base_abs = os.path.realpath(UPLOADS_DIR)
|
||||
in_uploads = (
|
||||
abs_target == base_abs
|
||||
or os.path.commonpath([abs_target, base_abs]) == base_abs
|
||||
|
||||
@@ -11,7 +11,7 @@ from core.session_manager import SessionManager
|
||||
from core.models import ChatMessage
|
||||
from src.request_models import SessionResponse
|
||||
from core.database import Session as DbSession, SessionLocal, Document, GalleryImage, utcnow_naive
|
||||
from src.auth_helpers import get_current_user, effective_user, _auth_disabled, owner_filter
|
||||
from src.auth_helpers import effective_user, _auth_disabled, owner_filter
|
||||
from src.session_actions import is_session_recently_active
|
||||
|
||||
|
||||
@@ -328,7 +328,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
|
||||
endpoint_id: str = Form(""),
|
||||
):
|
||||
skip_val = str(skip_validation).lower() == "true"
|
||||
user = get_current_user(request)
|
||||
user = effective_user(request)
|
||||
endpoint_api_key = ""
|
||||
endpoint_base_url = ""
|
||||
_reject_raw_endpoint_url_for_non_admin(request, user, endpoint_id, endpoint_url)
|
||||
@@ -477,7 +477,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
|
||||
db.close()
|
||||
# Switch model/endpoint mid-session
|
||||
if model is not None and endpoint_url is not None:
|
||||
user = get_current_user(request)
|
||||
user = effective_user(request)
|
||||
_reject_raw_endpoint_url_for_non_admin(request, user, endpoint_id, endpoint_url)
|
||||
endpoint_api_key = ""
|
||||
endpoint_base_url = ""
|
||||
|
||||
@@ -7,7 +7,7 @@ from fastapi import APIRouter, Request, File, UploadFile, HTTPException
|
||||
from typing import List
|
||||
import logging
|
||||
from core.middleware import require_admin
|
||||
from src.auth_helpers import get_current_user
|
||||
from src.auth_helpers import effective_user
|
||||
from src.upload_handler import count_recent_uploads
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -78,7 +78,7 @@ def setup_upload_routes(upload_handler):
|
||||
|
||||
for u in files:
|
||||
try:
|
||||
meta = upload_handler.save_upload(u, client_ip, owner=get_current_user(request))
|
||||
meta = upload_handler.save_upload(u, client_ip, owner=effective_user(request))
|
||||
out.append({
|
||||
"id": meta["id"],
|
||||
"name": meta["name"],
|
||||
@@ -138,7 +138,7 @@ def setup_upload_routes(upload_handler):
|
||||
original_name = info.get("name", file_id)
|
||||
auth_mgr = getattr(request.app.state, "auth_manager", None)
|
||||
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
|
||||
current_user = get_current_user(request)
|
||||
current_user = effective_user(request)
|
||||
file_owner = info.get("owner") if info else None
|
||||
if auth_configured:
|
||||
if not current_user:
|
||||
@@ -204,7 +204,7 @@ def setup_upload_routes(upload_handler):
|
||||
info = _load_upload_info(file_id)
|
||||
auth_mgr = getattr(request.app.state, "auth_manager", None)
|
||||
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
|
||||
current_user = get_current_user(request)
|
||||
current_user = effective_user(request)
|
||||
file_owner = info.get("owner") if info else None
|
||||
if auth_configured:
|
||||
if not current_user:
|
||||
@@ -247,7 +247,7 @@ def setup_upload_routes(upload_handler):
|
||||
raise HTTPException(404, "File not found")
|
||||
auth_mgr = getattr(request.app.state, "auth_manager", None)
|
||||
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
|
||||
current_user = get_current_user(request)
|
||||
current_user = effective_user(request)
|
||||
file_owner = info.get("owner")
|
||||
if auth_configured:
|
||||
if not current_user:
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
"""Webhook, API Token, and sync chat routes."""
|
||||
|
||||
import asyncio
|
||||
import uuid
|
||||
import logging
|
||||
from typing import Optional
|
||||
@@ -385,10 +384,10 @@ def setup_webhook_routes(
|
||||
sess.add_message(ChatMessage("assistant", reply))
|
||||
session_manager.save_sessions()
|
||||
|
||||
asyncio.create_task(webhook_manager.fire("chat.completed", {
|
||||
webhook_manager.fire_and_forget("chat.completed", {
|
||||
"session_id": session_id, "model": sess.model,
|
||||
"user_message": message[:2000], "response": reply[:2000],
|
||||
}))
|
||||
})
|
||||
|
||||
return {"response": reply, "session_id": session_id, "model": sess.model}
|
||||
|
||||
|
||||
@@ -19,6 +19,10 @@ GPU_BANDWIDTH = {
|
||||
"6950 xt": 576, "6900 xt": 512, "6800 xt": 512, "6800": 512, "6700 xt": 384, "6600 xt": 256, "6600": 224,
|
||||
"mi300x": 5300, "mi300": 5300, "mi250x": 3277, "mi250": 3277, "mi210": 1638, "mi100": 1229,
|
||||
"9070 xt": 624, "9070": 488, "9060 xt": 322, "9060": 322,
|
||||
# NVIDIA GB10 Grace-Blackwell superchip (DGX Spark). Unified LPDDR5X memory,
|
||||
# not Apple Silicon, so it lives in the generic GPU table — the Apple-only
|
||||
# lookup never matches it (its name carries no "apple").
|
||||
"gb10": 273,
|
||||
}
|
||||
|
||||
# Pre-sort keys by length descending for correct substring matching
|
||||
|
||||
@@ -15,6 +15,8 @@ from urllib.parse import urljoin, urlparse
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
from src.constants import WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES, WEB_FETCH_USER_AGENT
|
||||
|
||||
from .analytics import RateLimitError, error_logger
|
||||
from .cache import (
|
||||
CONTENT_CACHE_DIR,
|
||||
@@ -89,18 +91,128 @@ def _public_http_url(url: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _get_public_url(url: str, headers: dict, timeout: int, max_redirects: int = 5) -> httpx.Response:
|
||||
class BodyTooLargeError(Exception):
|
||||
"""The server declared a body larger than the hard fetch ceiling."""
|
||||
|
||||
def __init__(self, url: str, declared_bytes: int):
|
||||
self.url = url
|
||||
self.declared_bytes = declared_bytes
|
||||
super().__init__(
|
||||
f"response body is {declared_bytes:,} bytes, over the "
|
||||
f"{WEB_FETCH_HARD_MAX_BYTES:,}-byte hard cap"
|
||||
)
|
||||
|
||||
|
||||
class _CappedFetch:
|
||||
"""Result of a size-capped streaming GET.
|
||||
|
||||
Carries just what fetch_webpage_content needs from an httpx.Response,
|
||||
plus the cap bookkeeping: the (possibly truncated) body, whether the
|
||||
cap cut it short, and the size the server declared via Content-Length
|
||||
(wire bytes; None when absent).
|
||||
"""
|
||||
|
||||
__slots__ = ("status_code", "headers", "content", "truncated",
|
||||
"declared_bytes", "encoding", "url")
|
||||
|
||||
def __init__(self, status_code, headers, content, truncated,
|
||||
declared_bytes, encoding, url):
|
||||
self.status_code = status_code
|
||||
self.headers = headers
|
||||
self.content = content
|
||||
self.truncated = truncated
|
||||
self.declared_bytes = declared_bytes
|
||||
self.encoding = encoding
|
||||
self.url = url
|
||||
|
||||
@property
|
||||
def text(self) -> str:
|
||||
return self.content.decode(self.encoding or "utf-8", errors="replace")
|
||||
|
||||
def raise_for_status(self):
|
||||
if self.status_code >= 400:
|
||||
request = httpx.Request("GET", self.url)
|
||||
raise httpx.HTTPStatusError(
|
||||
f"HTTP {self.status_code} for {self.url}",
|
||||
request=request,
|
||||
response=httpx.Response(self.status_code, request=request),
|
||||
)
|
||||
|
||||
|
||||
def _get_public_url(url: str, headers: dict, timeout: int, max_redirects: int = 5,
|
||||
max_bytes: int = None) -> "_CappedFetch":
|
||||
"""Capped streaming GET with SSRF-guarded manual redirects.
|
||||
|
||||
The body is streamed and buffering stops at ``max_bytes`` (default: the
|
||||
soft cap), so an oversized resource cannot be pulled into memory or the
|
||||
content cache in full. When Content-Length already declares a body over
|
||||
the hard ceiling, the fetch is refused before any body bytes are read.
|
||||
"""
|
||||
cap = min(max_bytes or WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES)
|
||||
current = url
|
||||
for _ in range(max_redirects + 1):
|
||||
if not _public_http_url(current):
|
||||
raise httpx.RequestError("Blocked private/internal URL", request=httpx.Request("GET", current))
|
||||
response = httpx.get(current, headers=headers, timeout=timeout, follow_redirects=False)
|
||||
if response.status_code not in (301, 302, 303, 307, 308):
|
||||
return response
|
||||
location = response.headers.get("location")
|
||||
if not location:
|
||||
return response
|
||||
current = urljoin(str(response.url), location)
|
||||
# Force identity transfer-encoding. With gzip/deflate the wire bytes
|
||||
# (and Content-Length) can be a small fraction of the decoded body, so
|
||||
# a tiny compressed response could pass the hard-cap preflight and then
|
||||
# expand past the ceiling in a single decoded chunk before the streamed
|
||||
# cap below can slice it. Identity makes Content-Length the true body
|
||||
# size and keeps each streamed chunk bounded by the network read.
|
||||
req_headers = dict(headers or {})
|
||||
req_headers["Accept-Encoding"] = "identity"
|
||||
with httpx.stream("GET", current, headers=req_headers, timeout=timeout,
|
||||
follow_redirects=False) as response:
|
||||
if response.status_code in (301, 302, 303, 307, 308):
|
||||
location = response.headers.get("location")
|
||||
if not location:
|
||||
return _CappedFetch(response.status_code, response.headers, b"",
|
||||
False, None, response.encoding, str(response.url))
|
||||
current = urljoin(str(response.url), location)
|
||||
continue
|
||||
|
||||
# A server can ignore the identity request and still return a
|
||||
# compressed body; httpx.iter_bytes would then decode it, and a tiny
|
||||
# gzip can balloon into one decoded chunk far past the cap before we
|
||||
# slice. Refuse a compressed Content-Encoding so the streamed cap
|
||||
# stays a real memory bound (Content-Length is the compressed wire
|
||||
# length here, so the preflight and size metadata are unreliable too).
|
||||
enc = (response.headers.get("content-encoding") or "").strip().lower()
|
||||
if enc and enc != "identity":
|
||||
raise httpx.RequestError(
|
||||
f"Refusing compressed response (Content-Encoding: {enc}) after "
|
||||
"requesting identity: cannot bound decoded body size",
|
||||
request=httpx.Request("GET", current),
|
||||
)
|
||||
|
||||
declared = None
|
||||
raw_len = response.headers.get("content-length")
|
||||
if raw_len and raw_len.isdigit():
|
||||
declared = int(raw_len)
|
||||
# Refuse before buffering anything when the server already tells
|
||||
# us the body exceeds the absolute ceiling (Content-Length is wire
|
||||
# bytes; the decompressed body can only be larger).
|
||||
if declared is not None and declared > WEB_FETCH_HARD_MAX_BYTES:
|
||||
raise BodyTooLargeError(current, declared)
|
||||
|
||||
chunks = []
|
||||
read = 0
|
||||
truncated = False
|
||||
# We requested identity above, so iter_bytes yields the raw body in
|
||||
# network-read-sized chunks (no decompression expansion); the cap
|
||||
# therefore bounds what we actually buffer.
|
||||
for chunk in response.iter_bytes():
|
||||
read += len(chunk)
|
||||
if read > cap:
|
||||
keep = cap - (read - len(chunk))
|
||||
if keep > 0:
|
||||
chunks.append(chunk[:keep])
|
||||
truncated = True
|
||||
break
|
||||
chunks.append(chunk)
|
||||
return _CappedFetch(response.status_code, response.headers,
|
||||
b"".join(chunks), truncated, declared,
|
||||
response.encoding, str(response.url))
|
||||
raise httpx.RequestError("Too many redirects", request=httpx.Request("GET", current))
|
||||
|
||||
# PDF extraction (optional dependency)
|
||||
@@ -222,9 +334,19 @@ def _empty_result(url: str, error: str = "") -> dict:
|
||||
# ----------------------------------------------------------------------
|
||||
# Main content fetcher
|
||||
# ----------------------------------------------------------------------
|
||||
def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) -> dict:
|
||||
"""Fetch and extract meaningful content from a webpage with caching."""
|
||||
cache_key = generate_cache_key(url)
|
||||
def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0,
|
||||
max_bytes: int = None) -> dict:
|
||||
"""Fetch and extract meaningful content from a webpage with caching.
|
||||
|
||||
``max_bytes`` raises the download budget per call (clamped to the hard
|
||||
cap); the default is the soft cap. When the body is cut short the result
|
||||
carries ``truncated``/``fetched_bytes``/``total_bytes`` so callers can
|
||||
tell the model the content is partial (#3812).
|
||||
"""
|
||||
effective_cap = min(max_bytes or WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES)
|
||||
# The cap is part of the cache identity: a truncated soft-cap fetch must
|
||||
# not be served to a later full-budget request for the same URL.
|
||||
cache_key = generate_cache_key(f"{url}#cap={effective_cap}")
|
||||
cache_file = CONTENT_CACHE_DIR / f"{cache_key}.cache"
|
||||
|
||||
# Check cache
|
||||
@@ -247,18 +369,24 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
|
||||
# Fetch
|
||||
try:
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
|
||||
"User-Agent": WEB_FETCH_USER_AGENT,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.5",
|
||||
"Accept-Encoding": "gzip, deflate",
|
||||
# identity so the streamed size cap in _get_public_url stays honest
|
||||
# (a compressed body can decode to far more than Content-Length).
|
||||
"Accept-Encoding": "identity",
|
||||
"Connection": "keep-alive",
|
||||
}
|
||||
response = _get_public_url(url, headers=headers, timeout=timeout)
|
||||
response = _get_public_url(url, headers=headers, timeout=timeout,
|
||||
max_bytes=effective_cap)
|
||||
|
||||
if response.status_code == 429:
|
||||
raise RateLimitError(f"Rate limit hit for {url} (attempt {retry_attempt})")
|
||||
|
||||
response.raise_for_status()
|
||||
except BodyTooLargeError as e:
|
||||
error_logger.warning(f"Refused oversized body for {url}: {e}")
|
||||
return _empty_result(url, f"TooLarge: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
error_logger.warning(f"HTTP {e.response.status_code} fetching {url}: {e}")
|
||||
return _empty_result(url, f"HTTP {e.response.status_code}: {e}")
|
||||
@@ -269,9 +397,27 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
|
||||
error_logger.error(str(e))
|
||||
return _empty_result(url, str(e))
|
||||
|
||||
# Size bookkeeping shared by every content branch below. getattr keeps
|
||||
# plain httpx.Response stand-ins (tests) working without the cap fields.
|
||||
_size_fields = {
|
||||
"truncated": getattr(response, "truncated", False),
|
||||
"fetched_bytes": len(response.content),
|
||||
"total_bytes": getattr(response, "declared_bytes", None),
|
||||
}
|
||||
|
||||
# PDF handling
|
||||
content_type = response.headers.get("Content-Type", "").lower()
|
||||
if "application/pdf" in content_type or url.lower().endswith(".pdf"):
|
||||
if _size_fields["truncated"]:
|
||||
# A PDF cut mid-stream is not parseable; unlike text there is no
|
||||
# useful partial result, so report the budget problem instead.
|
||||
_declared = _size_fields["total_bytes"]
|
||||
return _empty_result(
|
||||
url,
|
||||
f"TooLarge: PDF exceeds the {effective_cap:,}-byte fetch budget"
|
||||
+ (f" (size {_declared:,} bytes)" if _declared else "")
|
||||
+ "; retry with a larger budget if it fits under the hard cap",
|
||||
)
|
||||
if pdf_extract_text is None:
|
||||
logger.error("pdfminer.six is not installed; cannot extract PDF text.")
|
||||
pdf_text = ""
|
||||
@@ -295,6 +441,7 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
|
||||
"js_message": "",
|
||||
"success": bool(pdf_text),
|
||||
"error": "" if pdf_text else "Failed to extract PDF text",
|
||||
**_size_fields,
|
||||
}
|
||||
_cache_result(cache_file, cache_key, result, url)
|
||||
return result
|
||||
@@ -329,6 +476,7 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
|
||||
"js_message": "",
|
||||
"success": bool(text_body),
|
||||
"error": "" if text_body else "Empty response body",
|
||||
**_size_fields,
|
||||
}
|
||||
_cache_result(cache_file, cache_key, result, url)
|
||||
return result
|
||||
@@ -391,6 +539,7 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
|
||||
"js_message": js_message,
|
||||
"success": True,
|
||||
"error": "",
|
||||
**_size_fields,
|
||||
}
|
||||
_cache_result(cache_file, cache_key, result, url)
|
||||
return result
|
||||
|
||||
@@ -9,14 +9,12 @@ from urllib.parse import urljoin, urlparse, parse_qs
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
from src.constants import SEARXNG_INSTANCE
|
||||
from src.constants import SEARXNG_INSTANCE, REQUEST_TIMEOUT, WEB_FETCH_USER_AGENT
|
||||
from .analytics import RateLimitError, error_logger
|
||||
from .query import build_enhanced_query
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
REQUEST_TIMEOUT = 20
|
||||
|
||||
# Provider registry — maps setting value to (label, needs_key, needs_url)
|
||||
PROVIDER_INFO = {
|
||||
"searxng": ("SearXNG", False, True),
|
||||
@@ -140,7 +138,7 @@ def searxng_search_api(query: str, count: Optional[int] = None, categories: str
|
||||
count = count if count is not None else _get_result_count()
|
||||
instance = _get_search_instance()
|
||||
api_key = ""
|
||||
headers = {"User-Agent": "Mozilla/5.0"}
|
||||
headers = {"User-Agent": WEB_FETCH_USER_AGENT}
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
# News/fresh queries do badly in the 'general' category — it favours
|
||||
@@ -252,7 +250,7 @@ def searxng_search(query, max_results=10):
|
||||
"""Search using SearXNG instance - parsing HTML."""
|
||||
instance = _get_search_instance()
|
||||
api_key = ""
|
||||
req_headers = {"User-Agent": "Mozilla/5.0"}
|
||||
req_headers = {"User-Agent": WEB_FETCH_USER_AGENT}
|
||||
if api_key:
|
||||
req_headers["Authorization"] = f"Bearer {api_key}"
|
||||
try:
|
||||
@@ -391,7 +389,7 @@ def duckduckgo_search(query: str, count: Optional[int] = None, time_filter: Opti
|
||||
response = httpx.get(
|
||||
"https://html.duckduckgo.com/html/",
|
||||
params={"q": query, "kp": _safesearch_for("duckduckgo_html")},
|
||||
headers={"User-Agent": "Mozilla/5.0"},
|
||||
headers={"User-Agent": WEB_FETCH_USER_AGENT},
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
@@ -524,7 +524,7 @@ def get_builtin_overrides() -> dict:
|
||||
ov = get_setting("builtin_tool_overrides", {})
|
||||
return ov if isinstance(ov, dict) else {}
|
||||
except Exception as e:
|
||||
logger.warning('Failed to load builtin tool overrides: %s', e)
|
||||
logger.warning("Failed to load builtin tool overrides, using defaults", exc_info=e)
|
||||
return {}
|
||||
|
||||
|
||||
@@ -929,8 +929,8 @@ def _build_system_prompt(
|
||||
try:
|
||||
from src.user_time import current_datetime_context_message
|
||||
_datetime_message = current_datetime_context_message()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning("Failed to build datetime context message", exc_info=e)
|
||||
|
||||
# Document context is kept as a SEPARATE message (not merged into the tool
|
||||
# prompt) so the context trimmer doesn't destroy it when truncating the
|
||||
@@ -973,8 +973,8 @@ def _build_system_prompt(
|
||||
try:
|
||||
from src.pdf_form_doc import find_source_upload_id
|
||||
_is_form_backed = bool(find_source_upload_id(active_document.current_content or ""))
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning("Failed to detect if document is form-backed, assuming plain", exc_info=e)
|
||||
|
||||
if _is_form_backed:
|
||||
doc_ctx = (
|
||||
|
||||
@@ -57,13 +57,23 @@ class WebSearchTool:
|
||||
class WebFetchTool:
|
||||
async def execute(self, content: str, ctx: dict) -> dict:
|
||||
from src.search.content import fetch_webpage_content
|
||||
from src.constants import WEB_FETCH_HARD_MAX_BYTES
|
||||
raw = content.strip()
|
||||
url = ""
|
||||
max_bytes = None
|
||||
if raw.startswith("{"):
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
if isinstance(parsed, dict):
|
||||
url = str(parsed.get("url") or "").strip()
|
||||
# Download-budget override (#3812): "full": true raises the
|
||||
# budget to the hard cap; an explicit max_bytes is clamped
|
||||
# to the hard cap downstream. Default stays the soft cap.
|
||||
if parsed.get("full") is True:
|
||||
max_bytes = WEB_FETCH_HARD_MAX_BYTES
|
||||
mb = parsed.get("max_bytes")
|
||||
if isinstance(mb, int) and mb > 0:
|
||||
max_bytes = mb
|
||||
except json.JSONDecodeError:
|
||||
url = ""
|
||||
if not url:
|
||||
@@ -78,7 +88,7 @@ class WebFetchTool:
|
||||
loop = asyncio.get_running_loop()
|
||||
try:
|
||||
result = await asyncio.wait_for(
|
||||
loop.run_in_executor(None, lambda: fetch_webpage_content(url, timeout=10)),
|
||||
loop.run_in_executor(None, lambda: fetch_webpage_content(url, timeout=10, max_bytes=max_bytes)),
|
||||
timeout=30,
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
@@ -94,8 +104,28 @@ class WebFetchTool:
|
||||
return {"error": f"web_fetch: {url}: {err}", "exit_code": 1}
|
||||
return {"error": f"web_fetch: {url}: no readable text content (not HTML, or the page needs JS/login)", "exit_code": 1}
|
||||
|
||||
# Tell the model when the download budget cut the body short and how
|
||||
# to get the rest, instead of silently presenting a partial page as
|
||||
# the whole thing.
|
||||
size_note = ""
|
||||
if result.get("truncated"):
|
||||
fetched = result.get("fetched_bytes") or 0
|
||||
total = result.get("total_bytes")
|
||||
total_txt = f" of {total:,} bytes" if total else ""
|
||||
size_note = (
|
||||
f"[partial content: download stopped at {fetched:,} bytes{total_txt}. "
|
||||
f'Re-call with {{"url": "{url}", "full": true}} to fetch up to '
|
||||
f"{WEB_FETCH_HARD_MAX_BYTES:,} bytes.]\n\n"
|
||||
)
|
||||
|
||||
# The notice must lead the output so the MAX_OUTPUT_CHARS trim below can
|
||||
# never drop it. The title is untrusted, uncapped page content, so a
|
||||
# giant title ahead of the notice could push it out of range; keep the
|
||||
# notice first and cap the title as a second guard.
|
||||
if len(title) > 300:
|
||||
title = title[:300] + "..."
|
||||
header = (f"# {title}\n" if title else "") + f"Source: {url}\n\n"
|
||||
output = header + text
|
||||
output = size_note + header + text
|
||||
if len(output) > MAX_OUTPUT_CHARS:
|
||||
output = output[:MAX_OUTPUT_CHARS] + "\n\n[...truncated]"
|
||||
return {"output": output, "exit_code": 0}
|
||||
|
||||
@@ -1613,7 +1613,9 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
|
||||
"""
|
||||
import base64
|
||||
import httpx
|
||||
import os
|
||||
from pathlib import Path
|
||||
from src.url_safety import check_outbound_url
|
||||
|
||||
lines = content.strip().split("\n")
|
||||
prompt = lines[0].strip() if lines else ""
|
||||
@@ -1779,8 +1781,15 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
|
||||
|
||||
elif img.get("url"):
|
||||
# Download external URL and save locally (DALL-E returns temp URLs)
|
||||
result_url = img["url"]
|
||||
ok, reason = check_outbound_url(
|
||||
result_url,
|
||||
block_private=os.getenv("IMAGE_BLOCK_PRIVATE_IPS", "false").lower() == "true",
|
||||
)
|
||||
if not ok:
|
||||
return {"error": f"Image API returned unsafe image URL: {reason}"}
|
||||
try:
|
||||
dl_resp = httpx.get(img["url"], timeout=60)
|
||||
dl_resp = httpx.get(result_url, timeout=60)
|
||||
if dl_resp.status_code == 200:
|
||||
img_dir = Path(GENERATED_IMAGES_DIR)
|
||||
img_dir.mkdir(parents=True, exist_ok=True)
|
||||
@@ -1790,10 +1799,10 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
|
||||
image_url = f"/api/generated-image/{filename}"
|
||||
image_id = _save_to_gallery(filename)
|
||||
else:
|
||||
image_url = img["url"] # fallback to external URL
|
||||
image_url = result_url # fallback to external URL
|
||||
except Exception as _dl_e:
|
||||
logger.warning(f"Failed to download DALL-E image: {_dl_e}")
|
||||
image_url = img["url"] # fallback to external URL
|
||||
image_url = result_url # fallback to external URL
|
||||
else:
|
||||
return {"error": "Image API returned unexpected format (no b64_json or url)"}
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@ import subprocess
|
||||
import sys
|
||||
|
||||
from core.platform_compat import IS_WINDOWS, which_tool
|
||||
from src.runtime_paths import get_app_root
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -81,7 +82,7 @@ _BUILTIN_NPX_SERVERS = {
|
||||
"name": "Built-in: Browser",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@playwright/mcp@latest", "--headless", "--caps", "vision"],
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
# Global flag to disable MCP if there are compatibility issues
|
||||
@@ -94,7 +95,7 @@ async def register_builtin_servers(mcp_manager):
|
||||
logger.info("Built-in MCP servers disabled via ODYSSEUS_DISABLE_MCP")
|
||||
return
|
||||
|
||||
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
base_dir = get_app_root()
|
||||
python = sys.executable
|
||||
|
||||
async def _connect_python_server(server_id: str, script_path: str, name: str):
|
||||
|
||||
@@ -5,6 +5,7 @@ from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
from pydantic import Field, field_validator
|
||||
|
||||
from src.constants import DATA_DIR as _DATA_DIR_CONST
|
||||
from src.runtime_paths import get_app_root
|
||||
|
||||
# Cross-platform OS flag, exposed here so callers can `from src.config import
|
||||
# IS_WINDOWS`. Defined locally (a trivial `os.name == "nt"`) rather than imported
|
||||
@@ -19,7 +20,7 @@ IS_WINDOWS = os.name == "nt"
|
||||
class DataConfig(BaseSettings):
|
||||
"""Configuration for data storage and file handling."""
|
||||
# Base directory
|
||||
base_dir: Path = Field(default=Path(__file__).parent.parent, description="Base directory for the application")
|
||||
base_dir: Path = Field(default=Path(get_app_root()), description="Base directory for the application")
|
||||
|
||||
# Data paths
|
||||
data_dir: Path = Field(default=Path(_DATA_DIR_CONST), description="Main data directory")
|
||||
@@ -138,7 +139,7 @@ class AppConfig(BaseSettings):
|
||||
if isinstance(v, dict) and "base_dir" in v:
|
||||
base_dir = v["base_dir"]
|
||||
else:
|
||||
base_dir = Path(__file__).parent.parent
|
||||
base_dir = Path(get_app_root())
|
||||
|
||||
# Convert string paths to Path objects relative to base_dir
|
||||
data_dir = Path(_DATA_DIR_CONST)
|
||||
|
||||
@@ -2,12 +2,14 @@
|
||||
"""Application-wide constants and configuration values."""
|
||||
import os
|
||||
|
||||
from src.runtime_paths import get_app_root, get_default_data_dir
|
||||
|
||||
APP_VERSION = "1.0.0"
|
||||
|
||||
# Base paths
|
||||
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + "/"
|
||||
BASE_DIR = os.path.join(get_app_root(), "")
|
||||
STATIC_DIR = os.path.join(BASE_DIR, "static")
|
||||
DATA_DIR = os.getenv("ODYSSEUS_DATA_DIR", os.path.join(BASE_DIR, "data"))
|
||||
DATA_DIR = os.getenv("ODYSSEUS_DATA_DIR", get_default_data_dir())
|
||||
|
||||
# Data file paths
|
||||
# Single source of truth: every persisted file/dir lives under DATA_DIR, which
|
||||
@@ -55,7 +57,13 @@ MEMORY_VECTORS_DIR = os.path.join(DATA_DIR, "memory_vectors")
|
||||
|
||||
# Paths with an intentional dedicated env override, defaulting under DATA_DIR.
|
||||
MAIL_ATTACHMENTS_DIR = os.getenv("ODYSSEUS_MAIL_ATTACHMENTS_DIR", os.path.join(DATA_DIR, "mail-attachments"))
|
||||
FASTEMBED_CACHE_DIR = os.getenv("FASTEMBED_CACHE_PATH", os.path.join(DATA_DIR, "fastembed_cache"))
|
||||
# `or` (not os.getenv's default arg) so a PRESENT-but-EMPTY value falls back to
|
||||
# the default. docker-compose.yml injects `FASTEMBED_CACHE_PATH=${FASTEMBED_CACHE_PATH:-}`,
|
||||
# which sets the var to "" when the host hasn't defined it. os.getenv(name, default)
|
||||
# only returns the default when the var is ABSENT, so the empty string would win →
|
||||
# os.makedirs("") raises [Errno 2] No such file or directory: '' → FastEmbed fails to
|
||||
# init and all vector features (RAG, semantic memory, tool index) silently degrade.
|
||||
FASTEMBED_CACHE_DIR = os.getenv("FASTEMBED_CACHE_PATH") or os.path.join(DATA_DIR, "fastembed_cache")
|
||||
|
||||
# Agent tool output limits (single source of truth — imported by tool_execution.py,
|
||||
# tool_implementations.py, agent_tools.py, and any other module that needs them)
|
||||
@@ -63,11 +71,26 @@ MAX_OUTPUT_CHARS = 10_000 # cap for bash/python/web_search/web_fetch outpu
|
||||
MAX_READ_CHARS = 20_000 # cap for read_file / document preview
|
||||
MAX_DIFF_LINES = 400 # cap for edit_file unified-diff display
|
||||
|
||||
# web_fetch response-size policy (#3812). MAX_OUTPUT_CHARS above only trims
|
||||
# what the agent SEES; these caps bound what the server downloads, parses,
|
||||
# and writes to the content cache. The soft cap is the default download
|
||||
# budget; the agent can raise it per call (full/max_bytes) but never past
|
||||
# the hard cap, so a model can't decide to pull a multi-GB file.
|
||||
WEB_FETCH_SOFT_MAX_BYTES = 2_000_000 # default download budget (2 MB)
|
||||
WEB_FETCH_HARD_MAX_BYTES = 20_000_000 # absolute ceiling, even with override (20 MB)
|
||||
|
||||
# API Configuration
|
||||
MAX_CONTEXT_MESSAGES = 90
|
||||
REQUEST_TIMEOUT = 20
|
||||
OPENAI_COMPAT_PATH = "/v1/chat/completions"
|
||||
|
||||
# Outbound UA for web_fetch / web_search scraping; common desktop UA so pages serve normal HTML.
|
||||
WEB_FETCH_USER_AGENT = os.environ.get(
|
||||
"WEB_FETCH_USER_AGENT",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
|
||||
)
|
||||
|
||||
# Environment variables with defaults
|
||||
DEFAULT_HOST = os.getenv("LLM_HOST", "localhost")
|
||||
LLM_HOSTS = [h.strip() for h in os.getenv("LLM_HOSTS", "").split(",") if h.strip()]
|
||||
|
||||
@@ -161,11 +161,13 @@ async def _tick() -> None:
|
||||
# Re-read state once before writing so we capture any updates from
|
||||
# concurrent UI syncs.
|
||||
stopped_any = False
|
||||
successfully_stopped_sids = set()
|
||||
for sid, host, port in to_stop:
|
||||
ok = await _stop_serve(sid, host, port)
|
||||
logger.info(f"cookbook_serve_lifecycle: stop {sid} (host={host or 'local'}): {'ok' if ok else 'failed'}")
|
||||
if ok:
|
||||
stopped_any = True
|
||||
successfully_stopped_sids.add(sid)
|
||||
# Drop the auto-registered endpoint so the model picker and
|
||||
# the chat router don't keep pointing at a dead server.
|
||||
for t in tasks:
|
||||
@@ -188,12 +190,11 @@ async def _tick() -> None:
|
||||
except Exception:
|
||||
fresh = state
|
||||
fresh_tasks = tasks
|
||||
stopped_sids = {sid for sid, _, _ in to_stop}
|
||||
for ft in fresh_tasks:
|
||||
if not isinstance(ft, dict):
|
||||
continue
|
||||
ft_sid = ft.get("sessionId") or ft.get("id")
|
||||
if ft_sid in stopped_sids:
|
||||
if ft_sid in successfully_stopped_sids:
|
||||
ft["status"] = "stopped"
|
||||
ft["_scheduledStopAtMs"] = None
|
||||
ft["_lastStatusFlipAt"] = now_ms
|
||||
|
||||
@@ -31,6 +31,8 @@ import numpy as np
|
||||
import httpx
|
||||
from typing import List, Optional
|
||||
|
||||
from src.runtime_paths import get_app_root
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DEFAULT_MODEL = "all-minilm:l6-v2"
|
||||
|
||||
@@ -283,7 +283,8 @@ def _is_ollama_native_url(url: str) -> bool:
|
||||
"""Return True for native Ollama API URLs, including Ollama Cloud."""
|
||||
try:
|
||||
parsed = urlparse(url or "")
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
logger.warning("Failed to parse URL for Ollama detection", exc_info=e)
|
||||
return False
|
||||
host = parsed.hostname or ""
|
||||
path = (parsed.path or "").rstrip("/")
|
||||
@@ -1345,8 +1346,8 @@ def list_model_ids(
|
||||
r = httpx.get(root + "/api/tags", timeout=timeout)
|
||||
r.raise_for_status()
|
||||
return [m.get("name") or m.get("model") for m in (r.json().get("models") or []) if m.get("name") or m.get("model")]
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning("Failed to fetch model list from configured endpoint", exc_info=e)
|
||||
return []
|
||||
|
||||
def normalize_model_id(
|
||||
|
||||
@@ -11,6 +11,8 @@ import os
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple
|
||||
|
||||
from src.runtime_paths import get_app_root
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
def _format_mcp_connection_error(name: str, command: str = "", args: Optional[List[str]] = None, error: Exception = None) -> str:
|
||||
@@ -508,7 +510,7 @@ class McpManager:
|
||||
return False
|
||||
|
||||
script_rel, name = _BUILTIN_SERVERS[server_id]
|
||||
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
base_dir = get_app_root()
|
||||
script_path = os.path.join(base_dir, script_rel)
|
||||
|
||||
# Clean up old connection
|
||||
|
||||
@@ -7,6 +7,7 @@ import time
|
||||
from pathlib import Path
|
||||
|
||||
from src.constants import RAG_DIR
|
||||
from src.runtime_paths import get_app_root
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Helpers for resolving runtime paths in source and frozen builds."""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
|
||||
def get_app_root() -> str:
|
||||
"""Return the app root directory.
|
||||
|
||||
In normal source runs, this is the repository root. In a frozen Windows
|
||||
build, it is the bundle content root (PyInstaller's internal directory)
|
||||
so bundled runtime folders like `static/`, `scripts/`, and `data/` stay
|
||||
together with the executable payload.
|
||||
"""
|
||||
if getattr(sys, "frozen", False):
|
||||
return getattr(sys, "_MEIPASS", os.path.dirname(os.path.abspath(sys.executable)))
|
||||
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
|
||||
def get_default_data_dir() -> str:
|
||||
"""Return the default path to the data directory.
|
||||
|
||||
In normal runs, this is a 'data' subdirectory under the app root.
|
||||
In frozen builds, it is a persistent user directory (~/.odysseus/data)
|
||||
to prevent SQLite databases and other persistent files from being
|
||||
written to the ephemeral, temporary extraction bundle directory.
|
||||
"""
|
||||
if getattr(sys, "frozen", False):
|
||||
return os.path.join(os.path.expanduser("~"), ".odysseus", "data")
|
||||
return os.path.join(get_app_root(), "data")
|
||||
@@ -236,6 +236,29 @@ def _digest_windows(now):
|
||||
]
|
||||
|
||||
|
||||
def _checkin_calendar_events(db, owner, start, end):
|
||||
"""Calendar events in [start, end] for ONE owner, for the check-in digest.
|
||||
|
||||
Ownership lives on CalendarCal.owner; events inherit it via calendar_id.
|
||||
The digest query had no owner scope, so it pulled EVERY user's events into
|
||||
one user's check-in (a cross-tenant leak of summaries/locations). Scope it
|
||||
by joining CalendarCal, mirroring routes/calendar_routes.list_events.
|
||||
"""
|
||||
from core.database import CalendarEvent as _CE, CalendarCal as _CC
|
||||
return (
|
||||
db.query(_CE)
|
||||
.join(_CC, _CE.calendar_id == _CC.id)
|
||||
.filter(
|
||||
_CC.owner == owner,
|
||||
_CE.dtstart >= start,
|
||||
_CE.dtstart <= end,
|
||||
_CE.status != "cancelled",
|
||||
)
|
||||
.order_by(_CE.dtstart)
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
class TaskScheduler:
|
||||
def __init__(self, session_manager):
|
||||
self._session_manager = session_manager
|
||||
@@ -1127,11 +1150,7 @@ class TaskScheduler:
|
||||
# Strip timezone for naive DB comparison
|
||||
_s = start.replace(tzinfo=None) if start.tzinfo else start
|
||||
_e = end.replace(tzinfo=None) if end.tzinfo else end
|
||||
evs = _db.query(_CE).filter(
|
||||
_CE.dtstart >= _s,
|
||||
_CE.dtstart <= _e,
|
||||
_CE.status != "cancelled",
|
||||
).order_by(_CE.dtstart).all()
|
||||
evs = _checkin_calendar_events(_db, task.owner, _s, _e)
|
||||
if not evs:
|
||||
continue
|
||||
# Group by importance for richer output
|
||||
|
||||
@@ -1579,10 +1579,10 @@ async def do_manage_calendar(content: str, owner: Optional[str] = None) -> Dict:
|
||||
text = str(raw).strip().lower()
|
||||
if text in {"none", "no", "off", "false"}:
|
||||
return None
|
||||
m = re.search(r"(\d+)\s*(?:m|min|minute|minutes)\b", text)
|
||||
m = re.search(r"(\d+)\s*(?:minutes?|mins?|m)\b", text)
|
||||
if m:
|
||||
return max(0, int(m.group(1)))
|
||||
m = re.search(r"(\d+)\s*(?:h|hr|hour|hours)\b", text)
|
||||
m = re.search(r"(\d+)\s*(?:hours?|hrs?|h)\b", text)
|
||||
if m:
|
||||
return max(0, int(m.group(1)) * 60)
|
||||
if text.isdigit():
|
||||
@@ -1595,7 +1595,7 @@ async def do_manage_calendar(content: str, owner: Optional[str] = None) -> Dict:
|
||||
return desc
|
||||
reminder_only = re.compile(
|
||||
r"^\s*(?:remind(?:er)?|alarm)\s*:?\s*\d+\s*"
|
||||
r"(?:m|min|minute|minutes|h|hr|hour|hours)\b.*$",
|
||||
r"(?:minutes?|mins?|m|hours?|hrs?|h)\b.*$",
|
||||
re.I,
|
||||
)
|
||||
return "" if reminder_only.match(desc) else desc
|
||||
@@ -3797,7 +3797,7 @@ async def do_resolve_contact(content: str, owner: Optional[str] = None) -> Dict:
|
||||
if not name:
|
||||
return {"error": "name is required", "exit_code": 1}
|
||||
|
||||
contacts = {} # email -> {name, source}
|
||||
contacts = {} # email_or_phone -> {name, source, phone?}
|
||||
|
||||
# 1. CardDAV (Radicale) — structured contacts. Call in-process: a
|
||||
# server-side httpx GET to /api/contacts/search carries no session
|
||||
@@ -3812,10 +3812,18 @@ async def do_resolve_contact(content: str, owner: Optional[str] = None) -> Dict:
|
||||
match = q in hay_name or any(q in (e or "").lower() for e in c.get("emails", []))
|
||||
if not match:
|
||||
continue
|
||||
has_email = False
|
||||
for email in (c.get("emails") or []):
|
||||
email = (email or "").strip().lower()
|
||||
if email and "@" in email:
|
||||
contacts[email] = {"name": c.get("name") or email, "source": "contacts"}
|
||||
has_email = True
|
||||
# Fall back to phone numbers when the contact has no email address
|
||||
if not has_email:
|
||||
for phone in (c.get("phones") or []):
|
||||
phone = (phone or "").strip()
|
||||
if phone:
|
||||
contacts[phone] = {"name": c.get("name") or phone, "source": "contacts", "phone": phone}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -3835,8 +3843,11 @@ async def do_resolve_contact(content: str, owner: Optional[str] = None) -> Dict:
|
||||
return {"output": f"No contacts found matching '{name}'.", "exit_code": 0}
|
||||
|
||||
lines = [f"Contacts matching '{name}':"]
|
||||
for email, info in contacts.items():
|
||||
lines.append(f"- {info['name']} <{email}> ({info['source']})")
|
||||
for key, info in contacts.items():
|
||||
if info.get("phone"):
|
||||
lines.append(f"- {info['name']} — phone: {info['phone']} ({info['source']})")
|
||||
else:
|
||||
lines.append(f"- {info['name']} <{key}> ({info['source']})")
|
||||
return {"output": "\n".join(lines), "exit_code": 0}
|
||||
|
||||
|
||||
|
||||
@@ -68,11 +68,12 @@ FUNCTION_TOOL_SCHEMAS = [
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "web_fetch",
|
||||
"description": "Fetch and read the text content of a specific URL the user names (e.g. 'check example.com', 'what's on this page <url>'). Use when you already have a concrete URL/domain. NOT for open-ended searches (use web_search) or 'research X' jobs (use trigger_research).",
|
||||
"description": "Fetch and read the text content of a specific URL the user names (e.g. 'check example.com', 'what's on this page <url>'). Use when you already have a concrete URL/domain. NOT for open-ended searches (use web_search) or 'research X' jobs (use trigger_research). Downloads are size-budgeted; a '[partial content: ...]' notice in the result means the body was cut short and you can re-call with full=true for the rest.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"url": {"type": "string", "description": "The URL or domain to fetch (http/https; a bare domain like example.com is fine)"}
|
||||
"url": {"type": "string", "description": "The URL or domain to fetch (http/https; a bare domain like example.com is fine)"},
|
||||
"full": {"type": "boolean", "description": "Raise the download budget to the hard cap for large pages/files. Use only after a result reported partial content."}
|
||||
},
|
||||
"required": ["url"]
|
||||
}
|
||||
@@ -1008,7 +1009,7 @@ FUNCTION_TOOL_SCHEMAS = [
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "resolve_contact",
|
||||
"description": "Look up a contact's email address by name. Searches CardDAV address book and sent email history. Use when the user says 'message [name]' or 'email [name]' without an email address.",
|
||||
"description": "Look up a contact by name. Searches CardDAV address book and sent email history. Returns email addresses (when available) or phone numbers. Use when the user says 'message [name]', 'email [name]', or asks for someone's contact details.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
@@ -87,7 +87,8 @@ import * as Modals from './modalManager.js';
|
||||
}
|
||||
|
||||
function _accountCanSend(account) {
|
||||
return !!(account && account.smtp_host && account.smtp_user && account.has_smtp_password);
|
||||
if (!account || !account.smtp_host || !account.smtp_user) return false;
|
||||
return !!(account.has_smtp_password || account.oauth_provider);
|
||||
}
|
||||
|
||||
async function _resolveComposeSendAccountId() {
|
||||
|
||||
@@ -2913,13 +2913,14 @@ async function initEmailAccountsSettings() {
|
||||
// IMAP and SMTP. Dovecot is IMAP-only here; the host is intentionally
|
||||
// blank because it may live on another machine (DNS, LAN, Tailscale).
|
||||
const PROVIDERS = {
|
||||
gmail: { label: 'Gmail', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 465 } },
|
||||
migadu: { label: 'Migadu', imap: { host: 'imap.migadu.com', port: 993, starttls: false }, smtp: { host: 'smtp.migadu.com', port: 465 } },
|
||||
icloud: { label: 'iCloud', imap: { host: 'imap.mail.me.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.me.com', port: 587 } },
|
||||
outlook: { label: 'Outlook / Office 365', imap: { host: 'outlook.office365.com', port: 993, starttls: false }, smtp: { host: 'smtp.office365.com', port: 587 } },
|
||||
fastmail: { label: 'Fastmail', imap: { host: 'imap.fastmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.fastmail.com', port: 465 } },
|
||||
yahoo: { label: 'Yahoo', imap: { host: 'imap.mail.yahoo.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.yahoo.com', port: 465 } },
|
||||
dovecot: { label: 'Dovecot IMAP (no SMTP)', imap: { host: '', port: 31143, starttls: false }, smtp: { host: '', port: 465 } },
|
||||
gmail: { label: 'Gmail', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 465 } },
|
||||
google_workspace: { label: 'Google Workspace / .edu', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 587 }, oauth: 'google' },
|
||||
migadu: { label: 'Migadu', imap: { host: 'imap.migadu.com', port: 993, starttls: false }, smtp: { host: 'smtp.migadu.com', port: 465 } },
|
||||
icloud: { label: 'iCloud', imap: { host: 'imap.mail.me.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.me.com', port: 587 } },
|
||||
outlook: { label: 'Outlook / Office 365', imap: { host: 'outlook.office365.com', port: 993, starttls: false }, smtp: { host: 'smtp.office365.com', port: 587 } },
|
||||
fastmail: { label: 'Fastmail', imap: { host: 'imap.fastmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.fastmail.com', port: 465 } },
|
||||
yahoo: { label: 'Yahoo', imap: { host: 'imap.mail.yahoo.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.yahoo.com', port: 465 } },
|
||||
dovecot: { label: 'Dovecot IMAP (no SMTP)', imap: { host: '', port: 31143, starttls: false }, smtp: { host: '', port: 465 } },
|
||||
};
|
||||
const _providerOptions = Object.entries(PROVIDERS)
|
||||
.map(([k, v]) => `<option value="${k}">${esc(v.label)}</option>`)
|
||||
@@ -2932,11 +2933,17 @@ async function initEmailAccountsSettings() {
|
||||
<div id="eaf-provider-note" style="display:none;font-size:11px;line-height:1.5;padding:8px 10px;margin:2px 0 4px;border:1px solid color-mix(in srgb, var(--fg) 15%, transparent);border-left:3px solid var(--accent, var(--red));border-radius:4px;background:color-mix(in srgb, var(--fg) 4%, transparent);"></div>
|
||||
<div class="settings-row"><label class="settings-label">Name${_hint('Optional label for this account (e.g. “Work” or “Personal”). Leave blank to use the email address.')}</label><input id="eaf-name" class="settings-input" placeholder="(optional — leave blank to use email)" value="${esc(a.name || '')}"></div>
|
||||
<div class="settings-row"><label class="settings-label">Email${_hint('Your email address. Used as the From: header on outgoing mail and as the display label when Name is blank.')}</label><input id="eaf-from" class="settings-input" placeholder="you@example.com" value="${esc(a.from_address || '')}"></div>
|
||||
<div class="settings-row"><label class="settings-label">Display Name${_hint('Your name as it appears in the From: field of emails you send, e.g. Jane Smith. Auto-filled from Google during OAuth.')}</label><input id="eaf-display-name" class="settings-input" placeholder="Your Name" value="${esc(a.display_name || '')}"></div>
|
||||
<div id="eaf-oauth-section" style="display:none;margin:8px 0;padding:10px;border:1px solid var(--border);border-radius:6px;background:color-mix(in srgb,var(--accent,#50fa7b) 6%,transparent)">
|
||||
<div style="font-size:11px;font-weight:600;margin-bottom:6px">Google OAuth2 — required for Workspace / .edu accounts</div>
|
||||
<div id="eaf-oauth-status" style="font-size:11px;opacity:0.7;margin-bottom:6px">${a.oauth_provider === 'google' ? '✓ Connected via Google OAuth' : 'Not connected — click below to authorize'}</div>
|
||||
<button type="button" id="eaf-oauth-btn" class="admin-btn-add" style="font-size:11px">${a.oauth_provider === 'google' ? 'Reconnect with Google' : 'Connect with Google'}</button>
|
||||
</div>
|
||||
<div style="font-size:11px;font-weight:600;opacity:0.6;margin:6px 0 2px">IMAP (Receiving)</div>
|
||||
<div class="settings-row"><label class="settings-label">Host${_hint('Your IMAP server, e.g. imap.gmail.com, imap.migadu.com, a LAN host, or a Tailscale IP for Dovecot.')}</label><input id="eaf-imap-host" class="settings-input" value="${esc(a.imap_host || '')}"></div>
|
||||
<div class="settings-row"><label class="settings-label">Port${_hint('993 for IMAPS (most providers), 143 for plain or STARTTLS. Local servers often use a custom port like 31143.')}</label><input id="eaf-imap-port" class="settings-input" type="number" value="${esc(a.imap_port || 993)}" style="max-width:100px"></div>
|
||||
<div class="settings-row"><label class="settings-label">Username${_hint('Usually your full email address.')}</label><input id="eaf-imap-user" class="settings-input" value="${esc(a.imap_user || '')}"></div>
|
||||
<div class="settings-row"><label class="settings-label">Password${_hint('Your IMAP login password. Use an app-specific password if your provider requires 2FA. Outlook / Office 365 generally requires OAuth and will not work with a normal password here.')}</label><input id="eaf-imap-pass" class="settings-input" type="password" placeholder="${isEdit && a.has_imap_password ? '(unchanged)' : ''}"></div>
|
||||
<div class="eaf-password-section"><div class="settings-row"><label class="settings-label">Password${_hint('Your IMAP login password. Use an app-specific password if your provider requires 2FA. Outlook / Office 365 generally requires OAuth and will not work with a normal password here.')}</label><input id="eaf-imap-pass" class="settings-input" type="password" placeholder="${isEdit && a.has_imap_password ? '(unchanged)' : ''}"></div></div>
|
||||
<div class="settings-row"><label class="settings-label">STARTTLS${_hint('Turn ON for port 143/587 to upgrade plain to TLS. Turn OFF for port 993 (IMAPS — already encrypted) or a local server with no TLS configured.')}</label><label class="admin-switch"><input type="checkbox" id="eaf-imap-starttls" ${a.imap_starttls !== false ? 'checked' : ''}><span class="admin-slider"></span></label></div>
|
||||
<div style="font-size:11px;font-weight:600;opacity:0.6;margin:8px 0 2px">SMTP (Sending) <span style="font-weight:normal;opacity:0.7">— optional, leave blank for read-only</span></div>
|
||||
<div class="settings-row"><label class="settings-label">Host${_hint('Your outgoing-mail server, e.g. smtp.gmail.com, smtp.migadu.com. Leave blank to make this account read-only.')}</label><input id="eaf-smtp-host" class="settings-input" value="${esc(a.smtp_host || '')}"></div>
|
||||
@@ -2959,6 +2966,16 @@ async function initEmailAccountsSettings() {
|
||||
</div>
|
||||
`;
|
||||
|
||||
// Show/hide OAuth section and password fields based on provider selection.
|
||||
function _syncOauthUI(providerKey) {
|
||||
const p = PROVIDERS[providerKey];
|
||||
const isOauth = !!(p && p.oauth);
|
||||
el('eaf-oauth-section').style.display = isOauth ? '' : 'none';
|
||||
formEl.querySelectorAll('.eaf-password-section').forEach(r => {
|
||||
r.style.display = isOauth ? 'none' : '';
|
||||
});
|
||||
}
|
||||
|
||||
const eafProviderNotes = {
|
||||
outlook: {
|
||||
title: 'Outlook / Office 365 needs OAuth',
|
||||
@@ -2983,13 +3000,41 @@ async function initEmailAccountsSettings() {
|
||||
el('eaf-provider').addEventListener('change', (e) => {
|
||||
_renderEafProviderNote(e.target.value);
|
||||
const p = PROVIDERS[e.target.value];
|
||||
if (!p) return;
|
||||
if (!p) { _syncOauthUI(''); return; }
|
||||
el('eaf-imap-host').value = p.imap.host;
|
||||
el('eaf-imap-port').value = p.imap.port;
|
||||
el('eaf-imap-starttls').checked = !!p.imap.starttls;
|
||||
el('eaf-smtp-host').value = p.smtp.host;
|
||||
el('eaf-smtp-port').value = p.smtp.port;
|
||||
el('eaf-smtp-security').value = p.smtp.security || ((parseInt(p.smtp.port || 465) === 587) ? 'starttls' : 'ssl');
|
||||
_syncOauthUI(e.target.value);
|
||||
});
|
||||
|
||||
// Init OAuth UI for accounts already connected via OAuth.
|
||||
if (a.oauth_provider === 'google') _syncOauthUI('google_workspace');
|
||||
|
||||
// "Connect with Google" button — save the account first, then redirect to OAuth.
|
||||
el('eaf-oauth-btn').addEventListener('click', async () => {
|
||||
// Must save the account first to get an account_id to pass to the OAuth flow.
|
||||
const body = {
|
||||
name: el('eaf-name').value.trim() || el('eaf-from').value.trim(),
|
||||
from_address: el('eaf-from').value.trim(),
|
||||
imap_host: el('eaf-imap-host').value.trim(),
|
||||
imap_port: parseInt(el('eaf-imap-port').value) || 993,
|
||||
imap_user: el('eaf-imap-user').value.trim(),
|
||||
imap_starttls: el('eaf-imap-starttls').checked,
|
||||
smtp_host: el('eaf-smtp-host').value.trim(),
|
||||
smtp_port: parseInt(el('eaf-smtp-port').value) || 587,
|
||||
smtp_user: el('eaf-imap-user').value.trim(),
|
||||
};
|
||||
if (!body.name) { el('eaf-msg').textContent = 'Enter a Name or Email first'; el('eaf-msg').style.color = 'var(--red)'; return; }
|
||||
const url = isEdit ? `/api/email/accounts/${a.id}` : '/api/email/accounts';
|
||||
const method = isEdit ? 'PUT' : 'POST';
|
||||
const r = await fetch(url, { method, credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
|
||||
const d = await r.json();
|
||||
if (!d.ok) { el('eaf-msg').textContent = d.error || 'Save failed'; el('eaf-msg').style.color = 'var(--red)'; return; }
|
||||
const accId = isEdit ? a.id : d.id;
|
||||
window.location.href = `/api/email/oauth/google/authorize?account_id=${encodeURIComponent(accId)}`;
|
||||
});
|
||||
el('eaf-smtp-security').value = _smtpSecurity(a);
|
||||
|
||||
@@ -3009,6 +3054,7 @@ async function initEmailAccountsSettings() {
|
||||
const body = {
|
||||
name: el('eaf-name').value.trim(),
|
||||
from_address: el('eaf-from').value.trim(),
|
||||
display_name: el('eaf-display-name').value.trim(),
|
||||
imap_host: el('eaf-imap-host').value.trim(),
|
||||
imap_port: parseInt(el('eaf-imap-port').value) || 993,
|
||||
imap_user: el('eaf-imap-user').value.trim(),
|
||||
@@ -4317,6 +4363,7 @@ async function initUnifiedIntegrations() {
|
||||
// it may be remote (DNS, LAN, Tailscale), not localhost.
|
||||
const PROVIDERS = {
|
||||
gmail: { label: 'Gmail', emailEx: 'you@gmail.com', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 465 } },
|
||||
google_workspace: { label: 'Google Workspace / .edu', emailEx: 'you@yourschool.edu', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 587 }, oauth: 'google' },
|
||||
migadu: { label: 'Migadu', emailEx: 'you@yourdomain.com', imap: { host: 'imap.migadu.com', port: 993, starttls: false }, smtp: { host: 'smtp.migadu.com', port: 465 } },
|
||||
icloud: { label: 'iCloud', emailEx: 'you@icloud.com', imap: { host: 'imap.mail.me.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.me.com', port: 587 } },
|
||||
outlook: { label: 'Outlook / Office 365', emailEx: 'you@outlook.com', imap: { host: 'outlook.office365.com', port: 993, starttls: false }, smtp: { host: 'smtp.office365.com', port: 587 } },
|
||||
@@ -4334,6 +4381,7 @@ async function initUnifiedIntegrations() {
|
||||
const PROV_LOGO = {
|
||||
'': _customLogo,
|
||||
gmail: _letterLogo('G', '#ea4335'),
|
||||
google_workspace: _letterLogo('G', '#ea4335'),
|
||||
migadu: _letterLogo('M', '#3aa39d'),
|
||||
icloud: _letterLogo('i', '#3693f3'),
|
||||
outlook: _letterLogo('O', '#0078d4'),
|
||||
@@ -4362,11 +4410,17 @@ async function initUnifiedIntegrations() {
|
||||
<div id="uf-email-provider-note" style="display:none;font-size:11px;line-height:1.5;padding:8px 10px;margin:2px 0 4px;border:1px solid color-mix(in srgb, var(--fg) 15%, transparent);border-left:3px solid var(--accent, var(--red));border-radius:4px;background:color-mix(in srgb, var(--fg) 4%, transparent);"></div>
|
||||
<div class="settings-row"><label class="settings-label">Name${_hint('Optional label for this account (e.g. “Work” or “Personal”). Leave blank to use the email address.')}</label><input id="uf-email-name" class="settings-input" placeholder="(optional — leave blank to use email)"></div>
|
||||
<div class="settings-row"><label class="settings-label">Email${_hint('Your email address. Used as the From: header on outgoing mail and as the display label when Name is blank.')}</label><input id="uf-email-from" class="settings-input" placeholder="you@example.com"></div>
|
||||
<div class="settings-row"><label class="settings-label">Display Name${_hint('Your name as it appears in the From: field of emails you send, e.g. Jane Smith. Auto-filled from Google during OAuth.')}</label><input id="uf-display-name" class="settings-input" placeholder="Your Name"></div>
|
||||
<div id="uf-oauth-section" style="display:none;margin:8px 0;padding:10px;border:1px solid var(--border);border-radius:6px;background:color-mix(in srgb,var(--accent,#50fa7b) 6%,transparent)">
|
||||
<div style="font-size:11px;font-weight:600;margin-bottom:6px">Google OAuth2 — required for Workspace / .edu accounts</div>
|
||||
<div id="uf-oauth-status" style="font-size:11px;opacity:0.7;margin-bottom:6px">${existing && existing.oauth_provider === 'google' ? '✓ Connected via Google OAuth' : 'Not connected — click below to authorize'}</div>
|
||||
<button type="button" id="uf-oauth-btn" class="admin-btn-add" style="font-size:11px">${existing && existing.oauth_provider === 'google' ? 'Reconnect with Google' : 'Connect with Google'}</button>
|
||||
</div>
|
||||
<div style="font-size:11px;font-weight:600;opacity:0.6;margin:4px 0 2px;display:flex;align-items:center;gap:5px;"><svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="color:var(--accent, var(--red));flex-shrink:0;" aria-hidden="true"><polyline points="22 12 16 12 14 15 10 15 8 12 2 12"/><path d="M5.45 5.11 2 12v6a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2v-6l-3.45-6.89A2 2 0 0 0 16.76 4H7.24a2 2 0 0 0-1.79 1.11z"/></svg>IMAP (Receiving)</div>
|
||||
<div class="settings-row"><label class="settings-label">Host${_hint('Your IMAP server, e.g. imap.gmail.com, imap.migadu.com, a LAN host, or a Tailscale IP for Dovecot.')}</label><input id="uf-imap-host" class="settings-input" placeholder="imap.example.com"></div>
|
||||
<div class="settings-row"><label class="settings-label">Port${_hint('993 for IMAPS (most providers), 143 for plain or STARTTLS. Local servers often use a custom port like 31143.')}</label><input id="uf-imap-port" class="settings-input" type="number" placeholder="993" style="max-width:100px"></div>
|
||||
<div class="settings-row"><label class="settings-label">Username${_hint('Yes — your full email address goes here too (e.g. you@gmail.com). Same as the Email field above for almost every provider.')}</label><input id="uf-imap-user" class="settings-input" placeholder="you@example.com"></div>
|
||||
<div class="settings-row"><label class="settings-label">Password${_hint('For Gmail, iCloud, and Yahoo: paste your App Password (NOT your normal account password). For Migadu and Fastmail, your mailbox password usually works. Outlook / Office 365 generally requires OAuth and will not work with this password form.')}</label><input id="uf-imap-pass" class="settings-input" type="password" placeholder="${placeholderPass}"></div>
|
||||
<div class="uf-password-section"><div class="settings-row"><label class="settings-label">Password${_hint('For Gmail, iCloud, and Yahoo: paste your App Password (NOT your normal account password). For Migadu and Fastmail, your mailbox password usually works. Outlook / Office 365 generally requires OAuth and will not work with this password form.')}</label><input id="uf-imap-pass" class="settings-input" type="password" placeholder="${placeholderPass}"></div></div>
|
||||
<div class="settings-row"><label class="settings-label">STARTTLS${_hint('Turn ON for port 143/587 to upgrade plain to TLS. Turn OFF for port 993 (IMAPS — already encrypted) or a local server with no TLS configured.')}</label><label class="admin-switch" style="margin-left:0"><input type="checkbox" id="uf-imap-starttls" checked><span class="admin-slider"></span></label></div>
|
||||
<div style="font-size:11px;font-weight:600;opacity:0.6;margin:8px 0 2px;display:flex;align-items:center;gap:5px;"><svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="color:var(--accent, var(--red));flex-shrink:0;" aria-hidden="true"><line x1="22" y1="2" x2="11" y2="13"/><polygon points="22 2 15 22 11 13 2 9 22 2"/></svg>SMTP (Sending) <span style="font-weight:normal;opacity:0.7">— optional, leave blank for read-only</span></div>
|
||||
<div class="settings-row"><label class="settings-label">Host${_hint('Your outgoing-mail server, e.g. smtp.gmail.com. Leave blank to make this account read-only.')}</label><input id="uf-smtp-host" class="settings-input" placeholder="smtp.example.com"></div>
|
||||
@@ -4491,6 +4545,16 @@ async function initUnifiedIntegrations() {
|
||||
</div>`;
|
||||
};
|
||||
|
||||
// Show/hide the OAuth section and password fields based on provider selection.
|
||||
function _syncOauthUI(providerKey) {
|
||||
const p = PROVIDERS[providerKey];
|
||||
const isOauth = !!(p && p.oauth);
|
||||
el('uf-oauth-section').style.display = isOauth ? '' : 'none';
|
||||
formEl.querySelectorAll('.uf-password-section').forEach(r => {
|
||||
r.style.display = isOauth ? 'none' : '';
|
||||
});
|
||||
}
|
||||
|
||||
// Custom dropdown wire-up — the native <select> stays in the DOM as the
|
||||
// data source and accessibility target, but the visible UI is a button +
|
||||
// popup so each provider row can render with its SVG logo. Selecting an
|
||||
@@ -4547,6 +4611,7 @@ async function initUnifiedIntegrations() {
|
||||
el('uf-email-provider').addEventListener('change', (e) => {
|
||||
const key = e.target.value;
|
||||
_renderProviderNote(key);
|
||||
_syncOauthUI(key);
|
||||
const p = PROVIDERS[key];
|
||||
if (!p) return;
|
||||
el('uf-imap-host').value = p.imap.host;
|
||||
@@ -4562,6 +4627,23 @@ async function initUnifiedIntegrations() {
|
||||
}
|
||||
});
|
||||
|
||||
// Init OAuth UI for accounts already connected via OAuth.
|
||||
if (existing && existing.oauth_provider === 'google') _syncOauthUI('google_workspace');
|
||||
|
||||
// "Connect with Google" — save the account first, then redirect to OAuth.
|
||||
el('uf-oauth-btn').addEventListener('click', async () => {
|
||||
const body = _collectBody();
|
||||
if (!body.name) body.name = body.from_address;
|
||||
if (!body.name) { el('uf-email-msg').textContent = 'Enter a Name or Email first'; el('uf-email-msg').style.color = 'var(--red)'; return; }
|
||||
const url = isEdit ? `/api/email/accounts/${editId}` : '/api/email/accounts';
|
||||
const method = isEdit ? 'PUT' : 'POST';
|
||||
const r = await fetch(url, { method, credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
|
||||
const d = await r.json();
|
||||
if (!(d.ok || d.id)) { el('uf-email-msg').textContent = d.error || 'Save failed'; el('uf-email-msg').style.color = 'var(--red)'; return; }
|
||||
const accId = isEdit ? editId : d.id;
|
||||
window.location.href = `/api/email/oauth/google/authorize?account_id=${encodeURIComponent(accId)}`;
|
||||
});
|
||||
|
||||
// "Same as IMAP" toggle — hide the SMTP creds rows when on.
|
||||
const _syncSmtpSame = () => {
|
||||
const same = el('uf-smtp-same').checked;
|
||||
@@ -4574,6 +4656,7 @@ async function initUnifiedIntegrations() {
|
||||
if (existing) {
|
||||
el('uf-email-name').value = existing.name || '';
|
||||
el('uf-email-from').value = existing.from_address || '';
|
||||
el('uf-display-name').value = existing.display_name || '';
|
||||
el('uf-imap-host').value = existing.imap_host || '';
|
||||
el('uf-imap-port').value = existing.imap_port || 993;
|
||||
el('uf-imap-user').value = existing.imap_user || '';
|
||||
@@ -4622,6 +4705,7 @@ async function initUnifiedIntegrations() {
|
||||
const body = {
|
||||
name: el('uf-email-name').value.trim(),
|
||||
from_address: el('uf-email-from').value.trim(),
|
||||
display_name: el('uf-display-name').value.trim(),
|
||||
imap_host: el('uf-imap-host').value.trim(),
|
||||
imap_port: parseInt(el('uf-imap-port').value) || 993,
|
||||
imap_user: el('uf-imap-user').value.trim(),
|
||||
@@ -5650,6 +5734,40 @@ export function close() {
|
||||
}
|
||||
}
|
||||
|
||||
// Handle redirect back from Google OAuth2 — open settings to integrations and show status.
|
||||
(function _handleOauthRedirect() {
|
||||
const sp = new URLSearchParams(window.location.search);
|
||||
if (!sp.has('email_oauth_success') && !sp.has('email_oauth_error')) return;
|
||||
// Strip params from URL without a page reload.
|
||||
const clean = window.location.pathname + window.location.hash;
|
||||
window.history.replaceState(null, '', clean);
|
||||
const success = sp.has('email_oauth_success');
|
||||
const errMsg = sp.get('email_oauth_error') || '';
|
||||
// Open settings → integrations after the app has initialised.
|
||||
function _tryOpen() {
|
||||
if (window.settingsModule && typeof window.settingsModule.open === 'function') {
|
||||
window.settingsModule.open('integrations');
|
||||
// Brief toast-style banner.
|
||||
const banner = document.createElement('div');
|
||||
banner.textContent = success
|
||||
? '✓ Google account connected — email is ready'
|
||||
: `Google OAuth failed: ${errMsg || 'unknown error'}`;
|
||||
Object.assign(banner.style, {
|
||||
position: 'fixed', bottom: '24px', left: '50%', transform: 'translateX(-50%)',
|
||||
background: success ? 'var(--accent, #50fa7b)' : 'var(--red, #ff5555)',
|
||||
color: '#000', padding: '8px 18px', borderRadius: '6px', fontSize: '12px',
|
||||
fontWeight: '600', zIndex: '99999', pointerEvents: 'none',
|
||||
boxShadow: '0 2px 12px rgba(0,0,0,0.3)',
|
||||
});
|
||||
document.body.appendChild(banner);
|
||||
setTimeout(() => banner.remove(), 4000);
|
||||
} else {
|
||||
setTimeout(_tryOpen, 100);
|
||||
}
|
||||
}
|
||||
_tryOpen();
|
||||
})();
|
||||
|
||||
const settingsModule = { open, close, initIntegrations, initUnifiedIntegrations, syncAdminVisibility, refreshAiModelEndpoints };
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
from src import ai_interaction
|
||||
|
||||
|
||||
class _GenerationResponse:
|
||||
status_code = 200
|
||||
text = ""
|
||||
|
||||
def __init__(self, image_url):
|
||||
self._image_url = image_url
|
||||
|
||||
def json(self):
|
||||
return {"data": [{"url": self._image_url}]}
|
||||
|
||||
|
||||
class _DownloadResponse:
|
||||
status_code = 503
|
||||
content = b""
|
||||
|
||||
|
||||
def _patch_generation(monkeypatch, image_url):
|
||||
async def _post(self, url, json, headers):
|
||||
return _GenerationResponse(image_url)
|
||||
|
||||
class _AsyncClient:
|
||||
def __init__(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
return False
|
||||
|
||||
post = _post
|
||||
|
||||
import httpx
|
||||
import src.settings as settings
|
||||
|
||||
monkeypatch.setattr(settings, "load_settings", lambda: {})
|
||||
monkeypatch.setattr(httpx, "AsyncClient", _AsyncClient)
|
||||
monkeypatch.setattr(
|
||||
ai_interaction,
|
||||
"_resolve_model",
|
||||
lambda model_spec, owner=None: (
|
||||
"https://api.openai.example/v1/chat/completions",
|
||||
"dall-e-3",
|
||||
{"Authorization": "Bearer test"},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
async def test_generate_image_validates_provider_url_before_download(monkeypatch):
|
||||
import httpx
|
||||
import src.url_safety as url_safety
|
||||
|
||||
provider_url = "https://images.example.com/generated.png?sig=abc"
|
||||
events = []
|
||||
_patch_generation(monkeypatch, provider_url)
|
||||
|
||||
def _check_outbound_url(url, *, block_private=False):
|
||||
events.append(("check", url, block_private))
|
||||
return True, "ok"
|
||||
|
||||
def _get(url, *, timeout):
|
||||
events.append(("get", url, timeout))
|
||||
return _DownloadResponse()
|
||||
|
||||
monkeypatch.setattr(url_safety, "check_outbound_url", _check_outbound_url)
|
||||
monkeypatch.setattr(httpx, "get", _get)
|
||||
|
||||
result = await ai_interaction.do_generate_image("draw a chair\ndall-e-3")
|
||||
|
||||
assert result["image_url"] == provider_url
|
||||
assert events == [
|
||||
("check", provider_url, False),
|
||||
("get", provider_url, 60),
|
||||
]
|
||||
|
||||
|
||||
async def test_generate_image_rejects_unsafe_provider_url_without_download(monkeypatch):
|
||||
import httpx
|
||||
import src.url_safety as url_safety
|
||||
|
||||
unsafe_url = "http://169.254.169.254/latest/meta-data"
|
||||
events = []
|
||||
_patch_generation(monkeypatch, unsafe_url)
|
||||
|
||||
def _check_outbound_url(url, *, block_private=False):
|
||||
events.append(("check", url, block_private))
|
||||
return False, "link-local address blocked (SSRF metadata risk): 169.254.169.254"
|
||||
|
||||
def _get(url, *, timeout):
|
||||
raise AssertionError("unsafe provider image URL must not be downloaded")
|
||||
|
||||
monkeypatch.setattr(url_safety, "check_outbound_url", _check_outbound_url)
|
||||
monkeypatch.setattr(httpx, "get", _get)
|
||||
|
||||
result = await ai_interaction.do_generate_image("draw a chair\ndall-e-3")
|
||||
|
||||
assert result["error"] == (
|
||||
"Image API returned unsafe image URL: "
|
||||
"link-local address blocked (SSRF metadata risk): 169.254.169.254"
|
||||
)
|
||||
assert events == [("check", unsafe_url, False)]
|
||||
@@ -219,6 +219,9 @@ class _WebhookManager:
|
||||
async def fire(self, event, payload):
|
||||
return None
|
||||
|
||||
def fire_and_forget(self, event, payload):
|
||||
return None
|
||||
|
||||
|
||||
def _install_sync_chat_stubs(monkeypatch):
|
||||
# FastAPI checks for python_multipart at import time when Form is used;
|
||||
|
||||
@@ -502,3 +502,77 @@ def test_delete_token_owner_check_skipped_when_auth_disabled(monkeypatch, token_
|
||||
resp = delete_token(request=req, token_id="tok123")
|
||||
assert resp == {"status": "deleted"}
|
||||
fake_session.delete.assert_called_once_with(fake_token)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 7. PATCH /api/tokens/{id} — non-object JSON bodies must not 500
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_update_token_with_array_body_does_not_500(monkeypatch, token_routes_mod):
|
||||
"""PATCH body of [] must be normalised to {} and not raise."""
|
||||
monkeypatch.setenv("AUTH_ENABLED", "true")
|
||||
mod = token_routes_mod
|
||||
|
||||
token = SimpleNamespace(
|
||||
id="tok123", name="original", owner="alice",
|
||||
token_prefix="ody_orig", scopes="email:read", is_active=True,
|
||||
)
|
||||
fake_session = MagicMock()
|
||||
fake_session.query.return_value.filter.return_value.first.return_value = token
|
||||
monkeypatch.setattr(mod, "get_db_session", lambda: _db_ctx(fake_session))
|
||||
|
||||
invalidator = MagicMock()
|
||||
req = _patch_request(invalidator, [])
|
||||
update_token = _get_handler(mod, "PATCH", "/tokens/{token_id}")
|
||||
resp = asyncio.run(update_token(request=req, token_id="tok123"))
|
||||
|
||||
# Name and scopes must be unchanged — payload was normalised to {}
|
||||
assert token.name == "original"
|
||||
assert token.scopes == "email:read"
|
||||
assert resp["name"] == "original"
|
||||
|
||||
|
||||
def test_update_token_with_null_body_does_not_500(monkeypatch, token_routes_mod):
|
||||
"""PATCH body of null must be normalised to {} and not raise."""
|
||||
monkeypatch.setenv("AUTH_ENABLED", "true")
|
||||
mod = token_routes_mod
|
||||
|
||||
token = SimpleNamespace(
|
||||
id="tok123", name="original", owner="alice",
|
||||
token_prefix="ody_orig", scopes="chat", is_active=True,
|
||||
)
|
||||
fake_session = MagicMock()
|
||||
fake_session.query.return_value.filter.return_value.first.return_value = token
|
||||
monkeypatch.setattr(mod, "get_db_session", lambda: _db_ctx(fake_session))
|
||||
|
||||
invalidator = MagicMock()
|
||||
req = _patch_request(invalidator, None)
|
||||
update_token = _get_handler(mod, "PATCH", "/tokens/{token_id}")
|
||||
resp = asyncio.run(update_token(request=req, token_id="tok123"))
|
||||
|
||||
assert token.name == "original"
|
||||
assert token.scopes == "chat"
|
||||
|
||||
|
||||
def test_update_token_normal_object_still_works(monkeypatch, token_routes_mod):
|
||||
"""Normal dict payload continues to update fields as before."""
|
||||
monkeypatch.setenv("AUTH_ENABLED", "true")
|
||||
mod = token_routes_mod
|
||||
|
||||
token = SimpleNamespace(
|
||||
id="tok123", name="original", owner="alice",
|
||||
token_prefix="ody_orig", scopes="email:read", is_active=True,
|
||||
)
|
||||
fake_session = MagicMock()
|
||||
fake_session.query.return_value.filter.return_value.first.return_value = token
|
||||
monkeypatch.setattr(mod, "get_db_session", lambda: _db_ctx(fake_session))
|
||||
|
||||
invalidator = MagicMock()
|
||||
req = _patch_request(invalidator, {"name": "updated"})
|
||||
update_token = _get_handler(mod, "PATCH", "/tokens/{token_id}")
|
||||
resp = asyncio.run(update_token(request=req, token_id="tok123"))
|
||||
|
||||
assert token.name == "updated"
|
||||
assert resp["name"] == "updated"
|
||||
invalidator.assert_called_once()
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
"""do_manage_calendar must honour abbreviated reminder phrasings like "mins"/"hrs".
|
||||
|
||||
`_reminder_minutes` parsed the reminder offset with regexes anchored on
|
||||
`(?:m|min|minute|minutes)\b` / `(?:h|hr|hour|hours)\b`. The trailing `\b`
|
||||
made the very common plural abbreviations "mins" and "hrs" fail to match
|
||||
(after "min" the next char "s" is a word char, so no boundary), so a request
|
||||
like ``reminder_minutes: "5 mins"`` silently produced no reminder at all —
|
||||
even though the sibling duration parser (no `\b`) already accepted them.
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.helpers.import_state import clear_fake_database_modules
|
||||
from tests.helpers.sqlite_db import make_temp_sqlite
|
||||
|
||||
clear_fake_database_modules()
|
||||
|
||||
import core.database as cdb
|
||||
from core.database import Note
|
||||
|
||||
_TS, _ENGINE, _TMPDB = make_temp_sqlite(cdb.Base.metadata)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _bind_temp_db(monkeypatch):
|
||||
monkeypatch.setitem(sys.modules, "core.database", cdb)
|
||||
parent = sys.modules.get("core")
|
||||
if parent is not None:
|
||||
monkeypatch.setattr(parent, "database", cdb, raising=False)
|
||||
monkeypatch.setattr(cdb, "SessionLocal", _TS)
|
||||
yield
|
||||
|
||||
|
||||
async def _create_with_reminder(reminder, owner):
|
||||
from src.tool_implementations import do_manage_calendar
|
||||
|
||||
payload = {
|
||||
"action": "create_event",
|
||||
"summary": "Dentist",
|
||||
# Far-future so the reminder is never "already passed".
|
||||
"dtstart": "2030-01-01T10:00:00",
|
||||
"reminder_minutes": reminder,
|
||||
}
|
||||
return await do_manage_calendar(json.dumps(payload), owner=owner)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("reminder,expected", [
|
||||
("5 mins", 5),
|
||||
("10 mins", 10),
|
||||
("2 hrs", 120),
|
||||
("1 hr", 60),
|
||||
("15 minutes", 15), # regression: long form still works
|
||||
("30m", 30), # regression: bare unit still works
|
||||
])
|
||||
async def test_reminder_minutes_accepts_abbreviations(reminder, expected):
|
||||
owner = "tester-" + uuid.uuid4().hex[:6]
|
||||
res = await _create_with_reminder(reminder, owner)
|
||||
assert res.get("exit_code") == 0, res
|
||||
assert f"reminder {expected} min before" in res.get("response", ""), res
|
||||
|
||||
db = _TS()
|
||||
try:
|
||||
note = (
|
||||
db.query(Note)
|
||||
.filter(Note.owner == owner, Note.title == "Reminder: Dentist")
|
||||
.first()
|
||||
)
|
||||
assert note is not None, "reminder note should have been created"
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
async def test_no_reminder_when_offset_absent():
|
||||
owner = "tester-" + uuid.uuid4().hex[:6]
|
||||
from src.tool_implementations import do_manage_calendar
|
||||
|
||||
payload = {
|
||||
"action": "create_event",
|
||||
"summary": "No Reminder Event",
|
||||
"dtstart": "2030-02-01T10:00:00",
|
||||
}
|
||||
res = await do_manage_calendar(json.dumps(payload), owner=owner)
|
||||
assert res.get("exit_code") == 0, res
|
||||
assert "reminder set" not in res.get("response", ""), res
|
||||
@@ -30,7 +30,7 @@ class _Session:
|
||||
|
||||
|
||||
def test_allowed_models_legacy_empty_list_remains_unrestricted(monkeypatch):
|
||||
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
|
||||
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
|
||||
|
||||
_enforce_chat_privileges(
|
||||
_Request({"allowed_models": [], "max_messages_per_day": 0}),
|
||||
@@ -39,7 +39,7 @@ def test_allowed_models_legacy_empty_list_remains_unrestricted(monkeypatch):
|
||||
|
||||
|
||||
def test_allowed_models_explicit_empty_restricted_list_blocks_all_models(monkeypatch):
|
||||
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
|
||||
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
|
||||
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
_enforce_chat_privileges(
|
||||
@@ -56,7 +56,7 @@ def test_allowed_models_explicit_empty_restricted_list_blocks_all_models(monkeyp
|
||||
|
||||
|
||||
def test_allowed_models_nonempty_list_still_restricts_without_new_flag(monkeypatch):
|
||||
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
|
||||
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
|
||||
|
||||
_enforce_chat_privileges(
|
||||
_Request({"allowed_models": ["provider/model-a"], "max_messages_per_day": 0}),
|
||||
@@ -70,7 +70,7 @@ def test_allowed_models_nonempty_list_still_restricts_without_new_flag(monkeypat
|
||||
|
||||
|
||||
def test_no_restriction_allows_any_model(monkeypatch):
|
||||
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
|
||||
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
|
||||
|
||||
privs = {"allowed_models": [], "block_all_models": False, "max_messages_per_day": 0}
|
||||
_enforce_chat_privileges(_Request(privs), _Session("provider/model-a"))
|
||||
@@ -78,7 +78,7 @@ def test_no_restriction_allows_any_model(monkeypatch):
|
||||
|
||||
|
||||
def test_specific_allowlist_blocks_models_outside_it(monkeypatch):
|
||||
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
|
||||
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
|
||||
|
||||
privs = {
|
||||
"allowed_models": ["gpt-4"],
|
||||
@@ -92,7 +92,7 @@ def test_specific_allowlist_blocks_models_outside_it(monkeypatch):
|
||||
|
||||
|
||||
def test_block_all_models_blocks_regardless_of_allowed_models_contents(monkeypatch):
|
||||
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
|
||||
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
|
||||
|
||||
# Even if allowed_models contains entries, block_all_models wins.
|
||||
privs = {
|
||||
@@ -111,7 +111,7 @@ def test_block_all_models_blocks_regardless_of_allowed_models_contents(monkeypat
|
||||
def test_admin_user_is_never_blocked(monkeypatch):
|
||||
from core.auth import ADMIN_PRIVILEGES
|
||||
|
||||
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "admin")
|
||||
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "admin")
|
||||
|
||||
class _AdminAuthManager:
|
||||
def get_privileges(self, username):
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Check-in calendar digest must be scoped to the task owner.
|
||||
|
||||
The digest query selected CalendarEvent with no owner scope, so a scheduled
|
||||
check-in for one user pulled EVERY user's calendar events (summaries,
|
||||
locations) into their digest — a cross-tenant leak. Ownership lives on
|
||||
CalendarCal.owner; the query must join it, like routes/calendar_routes.
|
||||
"""
|
||||
import tempfile
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from sqlalchemy.pool import NullPool
|
||||
|
||||
import core.database as cdb
|
||||
from core.database import CalendarEvent, CalendarCal
|
||||
from src.task_scheduler import _checkin_calendar_events
|
||||
|
||||
_TMPDB = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
|
||||
_ENGINE = create_engine(f"sqlite:///{_TMPDB.name}", connect_args={"check_same_thread": False}, poolclass=NullPool)
|
||||
cdb.Base.metadata.create_all(_ENGINE)
|
||||
_TS = sessionmaker(bind=_ENGINE, autoflush=False, autocommit=False)
|
||||
|
||||
|
||||
def _seed():
|
||||
db = _TS()
|
||||
try:
|
||||
db.query(CalendarEvent).delete(); db.query(CalendarCal).delete()
|
||||
db.add(CalendarCal(id="calA", owner="alice", name="A"))
|
||||
db.add(CalendarCal(id="calB", owner="bob", name="B"))
|
||||
db.add(CalendarEvent(uid="a1", calendar_id="calA", summary="Alice mtg",
|
||||
dtstart=datetime(2026, 6, 10, 9, 0),
|
||||
dtend=datetime(2026, 6, 10, 10, 0), status="confirmed"))
|
||||
db.add(CalendarEvent(uid="b1", calendar_id="calB", summary="Bob secret",
|
||||
dtstart=datetime(2026, 6, 10, 10, 0),
|
||||
dtend=datetime(2026, 6, 10, 11, 0), status="confirmed"))
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def test_digest_only_returns_owner_events():
|
||||
_seed()
|
||||
db = _TS()
|
||||
try:
|
||||
s, e = datetime(2026, 6, 1), datetime(2026, 6, 30)
|
||||
alice = _checkin_calendar_events(db, "alice", s, e)
|
||||
assert [ev.summary for ev in alice] == ["Alice mtg"] # not Bob's
|
||||
bob = _checkin_calendar_events(db, "bob", s, e)
|
||||
assert [ev.summary for ev in bob] == ["Bob secret"]
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def test_cancelled_excluded_and_window_respected():
|
||||
_seed()
|
||||
db = _TS()
|
||||
try:
|
||||
db2 = _TS()
|
||||
db2.add(CalendarEvent(uid="a2", calendar_id="calA", summary="cancelled",
|
||||
dtstart=datetime(2026, 6, 11),
|
||||
dtend=datetime(2026, 6, 11, 1, 0), status="cancelled"))
|
||||
db2.commit(); db2.close()
|
||||
s, e = datetime(2026, 6, 1), datetime(2026, 6, 30)
|
||||
out = _checkin_calendar_events(db, "alice", s, e)
|
||||
assert "cancelled" not in [ev.summary for ev in out]
|
||||
finally:
|
||||
db.close()
|
||||
@@ -7,12 +7,39 @@ in ``remoteHost`` would be injected into that command.
|
||||
These pin validation on the host/port before they reach the ssh string, matching
|
||||
the validators the rest of the cookbook routes already apply.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from starlette.requests import Request
|
||||
|
||||
import routes.codex_routes as codex_routes
|
||||
|
||||
|
||||
def _route_endpoint(path: str, method: str):
|
||||
router = codex_routes.setup_codex_routes()
|
||||
for route in router.routes:
|
||||
if route.path == path and method in route.methods:
|
||||
return route.endpoint
|
||||
raise AssertionError(f"{method} {path} route not found")
|
||||
|
||||
|
||||
def _launch_request() -> Request:
|
||||
request = Request(
|
||||
{
|
||||
"type": "http",
|
||||
"method": "POST",
|
||||
"path": "/api/codex/cookbook/adopt",
|
||||
"headers": [],
|
||||
"state": {},
|
||||
}
|
||||
)
|
||||
request.state.api_token = True
|
||||
request.state.api_token_owner = "alice"
|
||||
request.state.api_token_scopes = ["cookbook:launch"]
|
||||
return request
|
||||
|
||||
|
||||
def test_rejects_remote_host_with_shell_metacharacters():
|
||||
task = {"remoteHost": "box; rm -rf ~", "sshPort": ""}
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
@@ -47,3 +74,26 @@ def test_default_ssh_port_omits_flag():
|
||||
)
|
||||
assert host == "box"
|
||||
assert port_flag == ""
|
||||
|
||||
|
||||
def test_adopt_rejects_ssh_option_host_before_shell(monkeypatch):
|
||||
calls = []
|
||||
|
||||
async def fail_if_shell_runs(*args, **kwargs):
|
||||
calls.append((args, kwargs))
|
||||
raise RuntimeError("shell should not run for invalid host")
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_shell", fail_if_shell_runs)
|
||||
|
||||
endpoint = _route_endpoint("/api/codex/cookbook/adopt", "POST")
|
||||
body = {
|
||||
"tmux_session": "serve_abc123",
|
||||
"model": "org/model",
|
||||
"host": "-oProxyCommand=sh",
|
||||
}
|
||||
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
asyncio.run(endpoint(_launch_request(), body))
|
||||
|
||||
assert exc.value.status_code == 400
|
||||
assert calls == []
|
||||
|
||||
@@ -13,6 +13,9 @@ import json
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
# core.database instantiates SQLAlchemy declarative classes at import time, which
|
||||
@@ -225,12 +228,34 @@ def test_models_route_scopes_api_token_to_token_owner(monkeypatch):
|
||||
endpoints = _call_models_route(
|
||||
monkeypatch,
|
||||
rows,
|
||||
_request(api_token=True, api_token_owner="alice", current_user="api"),
|
||||
_request(
|
||||
api_token=True,
|
||||
api_token_owner="alice",
|
||||
api_token_scopes=["chat"],
|
||||
current_user="api",
|
||||
),
|
||||
)
|
||||
|
||||
assert _endpoint_names(endpoints) == ["alice-endpoint", "shared-endpoint"]
|
||||
|
||||
|
||||
def test_models_route_rejects_api_token_without_chat_scope(monkeypatch):
|
||||
monkeypatch.setattr(companion_routes, "get_current_user", lambda request: "api")
|
||||
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
_models_route()(
|
||||
_request(
|
||||
api_token=True,
|
||||
api_token_owner="alice",
|
||||
api_token_scopes=["todos:read"],
|
||||
current_user="api",
|
||||
)
|
||||
)
|
||||
|
||||
assert exc.value.status_code == 403
|
||||
assert "chat scope" in exc.value.detail
|
||||
|
||||
|
||||
def test_models_route_unresolved_owner_returns_only_shared_rows(monkeypatch):
|
||||
rows = [
|
||||
_ep(1, "alice-endpoint", "alice"),
|
||||
@@ -242,7 +267,12 @@ def test_models_route_unresolved_owner_returns_only_shared_rows(monkeypatch):
|
||||
endpoints = _call_models_route(
|
||||
monkeypatch,
|
||||
rows,
|
||||
_request(api_token=True, api_token_owner=None, current_user="api"),
|
||||
_request(
|
||||
api_token=True,
|
||||
api_token_owner=None,
|
||||
api_token_scopes=["chat"],
|
||||
current_user="api",
|
||||
),
|
||||
)
|
||||
|
||||
assert _endpoint_names(endpoints) == ["shared-endpoint"]
|
||||
|
||||
@@ -786,6 +786,50 @@ def test_cached_model_scan_reports_plain_dir_gguf(tmp_path):
|
||||
assert ggufs[3]["quant"] == "BF16"
|
||||
|
||||
|
||||
def test_cached_model_scan_uses_ollama_api_before_cli_and_windows_opt_in():
|
||||
script = _cached_model_scan_script()
|
||||
|
||||
assert "scan_ollama_api()\nscan_ollama()" in script
|
||||
assert "if any(m.get('is_ollama') for m in models): return" in script
|
||||
assert "os.name == 'nt'" in script
|
||||
assert "ODYSSEUS_ALLOW_OLLAMA_CLI_SCAN" in script
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != "nt", reason="Windows Ollama CLI startup guard")
|
||||
def test_cached_model_scan_does_not_launch_ollama_cli_on_windows(tmp_path):
|
||||
"""Official Ollama for Windows can auto-start the tray/server on `ollama list`.
|
||||
The read-only cache scanner must not invoke that CLI unless explicitly opted in.
|
||||
"""
|
||||
marker = tmp_path / "ollama-called.txt"
|
||||
fake_ollama = tmp_path / "ollama.cmd"
|
||||
fake_ollama.write_text(
|
||||
"@echo off\r\n"
|
||||
f'echo called>"{marker}"\r\n'
|
||||
"echo NAME ID SIZE MODIFIED\r\n"
|
||||
"echo local-model:latest abc 1 GB now\r\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
empty_home = tmp_path / "home"
|
||||
empty_home.mkdir()
|
||||
scan_py = tmp_path / "scan_cache.py"
|
||||
scan_py.write_text(_cached_model_scan_script(), encoding="utf-8")
|
||||
env = dict(os.environ)
|
||||
env["PATH"] = str(tmp_path) + os.pathsep + env.get("PATH", "")
|
||||
env["HOME"] = str(empty_home)
|
||||
env.pop("ODYSSEUS_ALLOW_OLLAMA_CLI_SCAN", None)
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(scan_py)],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
)
|
||||
|
||||
assert marker.exists() is False
|
||||
assert all(m.get("backend") != "ollama" for m in json.loads(proc.stdout))
|
||||
|
||||
|
||||
def test_cached_model_scan_uses_huggingface_cache_env(tmp_path):
|
||||
"""Docker recreates can leave the persisted HF cache outside HOME.
|
||||
The Serve scanner should honor the cache env path instead of only ~/.cache.
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from src import cookbook_serve_lifecycle as lifecycle
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_tick_persists_only_successfully_stopped_serves(tmp_path, monkeypatch):
|
||||
state_path = tmp_path / "cookbook_state.json"
|
||||
state_path.write_text(
|
||||
json.dumps({
|
||||
"tasks": [
|
||||
{
|
||||
"id": "stop-succeeds",
|
||||
"type": "serve",
|
||||
"status": "running",
|
||||
"_scheduledStopAtMs": 0,
|
||||
},
|
||||
{
|
||||
"id": "stop-fails",
|
||||
"type": "serve",
|
||||
"status": "running",
|
||||
"_scheduledStopAtMs": 0,
|
||||
},
|
||||
]
|
||||
}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
async def fake_stop_serve(session_id, remote_host="", ssh_port=""):
|
||||
return session_id == "stop-succeeds"
|
||||
|
||||
async def fake_delete_endpoint(task):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(lifecycle, "COOKBOOK_STATE_FILE", str(state_path))
|
||||
monkeypatch.setattr(lifecycle, "_stop_serve", fake_stop_serve)
|
||||
monkeypatch.setattr(lifecycle, "_delete_endpoint_for_task", fake_delete_endpoint)
|
||||
|
||||
await lifecycle._tick()
|
||||
|
||||
tasks = {
|
||||
task["id"]: task
|
||||
for task in json.loads(state_path.read_text(encoding="utf-8"))["tasks"]
|
||||
}
|
||||
assert tasks["stop-succeeds"]["status"] == "stopped"
|
||||
assert tasks["stop-succeeds"]["_scheduledStopAtMs"] is None
|
||||
assert tasks["stop-fails"]["status"] == "running"
|
||||
assert tasks["stop-fails"]["_scheduledStopAtMs"] == 0
|
||||
@@ -0,0 +1,580 @@
|
||||
"""Tests for the Google OAuth2 email helpers.
|
||||
|
||||
Covers the security-critical surface added for Google Workspace / .edu
|
||||
IMAP/SMTP support:
|
||||
|
||||
- `make_oauth_state` / `verify_oauth_state` — HMAC-signed OAuth state so the
|
||||
callback can't be CSRF'd or have its account_id/owner tampered with.
|
||||
- `_smtp_ready` — an OAuth account (no stored password) must still count as
|
||||
send-capable; a host+user-only account without password or OAuth must not.
|
||||
- `_xoauth2_raw` / `_xoauth2_bytes` — SASL XOAUTH2 framing for SMTP/IMAP.
|
||||
- `_refresh_google_token` — token refresh stores result encrypted; failure is
|
||||
silent (no token/secret in logs or return value).
|
||||
- `_get_valid_google_token` — uses cached token when fresh; calls refresh when
|
||||
expired.
|
||||
- `google_oauth_callback` (real route) — invalid/tampered/missing state and
|
||||
provider errors return generic redirects with no PII; owner mismatch refuses
|
||||
the token write; a valid owner writes encrypted tokens only to the intended
|
||||
account.
|
||||
- `list_email_accounts` (real route) — exposes OAuth status but never token
|
||||
values.
|
||||
- `_imap_connect` — password accounts use login(); OAuth accounts use XOAUTH2.
|
||||
|
||||
Route tests pull the live endpoint out of `setup_email_routes()` and call it
|
||||
directly — they pin the real handler, not a re-implementation. The ASGI app is
|
||||
not booted; outbound HTTP is mocked and the DB is an isolated in-memory SQLite.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import json
|
||||
import time
|
||||
import unittest.mock as mock
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
# ── OAuth state signing ──────────────────────────────────────────
|
||||
|
||||
def test_oauth_state_round_trips_account_and_owner():
|
||||
from routes.email_helpers import make_oauth_state, verify_oauth_state
|
||||
|
||||
state = make_oauth_state("acct-123", "user@example.com")
|
||||
payload = verify_oauth_state(state)
|
||||
|
||||
assert payload is not None
|
||||
assert payload["a"] == "acct-123"
|
||||
assert payload["o"] == "user@example.com"
|
||||
assert payload["n"] # nonce present
|
||||
|
||||
|
||||
def test_oauth_state_nonce_is_unique_per_call():
|
||||
from routes.email_helpers import make_oauth_state, verify_oauth_state
|
||||
|
||||
a = verify_oauth_state(make_oauth_state("acct", "o"))
|
||||
b = verify_oauth_state(make_oauth_state("acct", "o"))
|
||||
assert a["n"] != b["n"]
|
||||
|
||||
|
||||
def test_oauth_state_rejects_tampered_account_id():
|
||||
from routes.email_helpers import make_oauth_state, verify_oauth_state
|
||||
|
||||
state = make_oauth_state("acct-123", "user@example.com")
|
||||
decoded = base64.urlsafe_b64decode(state.encode()).decode()
|
||||
payload_str, sig = decoded.rsplit("|", 1)
|
||||
payload = json.loads(payload_str)
|
||||
payload["a"] = "evil-acct" # attacker swaps the target account
|
||||
forged = base64.urlsafe_b64encode(
|
||||
(json.dumps(payload, separators=(",", ":")) + "|" + sig).encode()
|
||||
).decode()
|
||||
|
||||
assert verify_oauth_state(forged) is None
|
||||
|
||||
|
||||
def test_oauth_state_rejects_forged_signature():
|
||||
from routes.email_helpers import make_oauth_state, verify_oauth_state
|
||||
|
||||
state = make_oauth_state("acct-123", "user@example.com")
|
||||
decoded = base64.urlsafe_b64decode(state.encode()).decode()
|
||||
payload_str, _ = decoded.rsplit("|", 1)
|
||||
forged = base64.urlsafe_b64encode((payload_str + "|" + "deadbeef" * 8).encode()).decode()
|
||||
|
||||
assert verify_oauth_state(forged) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("garbage", ["", "not-base64-at-all", "###", "a|b|c"])
|
||||
def test_oauth_state_rejects_garbage(garbage):
|
||||
from routes.email_helpers import verify_oauth_state
|
||||
|
||||
assert verify_oauth_state(garbage) is None
|
||||
|
||||
|
||||
# ── _smtp_ready: OAuth accounts have no password but can still send ──
|
||||
|
||||
def test_smtp_ready_true_for_oauth_account_without_password():
|
||||
from routes.email_routes import _smtp_ready
|
||||
|
||||
cfg = {
|
||||
"smtp_host": "smtp.gmail.com",
|
||||
"smtp_user": "me@nyu.edu",
|
||||
"smtp_password": "",
|
||||
"oauth_provider": "google",
|
||||
}
|
||||
assert _smtp_ready(cfg) is True
|
||||
|
||||
|
||||
def test_smtp_ready_true_for_password_account():
|
||||
from routes.email_routes import _smtp_ready
|
||||
|
||||
cfg = {
|
||||
"smtp_host": "smtp.example.com",
|
||||
"smtp_user": "me@example.com",
|
||||
"smtp_password": "app-password",
|
||||
"oauth_provider": "",
|
||||
}
|
||||
assert _smtp_ready(cfg) is True
|
||||
|
||||
|
||||
def test_smtp_ready_false_without_password_or_oauth():
|
||||
from routes.email_routes import _smtp_ready
|
||||
|
||||
cfg = {
|
||||
"smtp_host": "smtp.example.com",
|
||||
"smtp_user": "me@example.com",
|
||||
"smtp_password": "",
|
||||
"oauth_provider": "",
|
||||
}
|
||||
assert _smtp_ready(cfg) is False
|
||||
|
||||
|
||||
def test_smtp_ready_false_without_host():
|
||||
from routes.email_routes import _smtp_ready
|
||||
|
||||
cfg = {"smtp_host": "", "smtp_user": "me@x.com", "oauth_provider": "google"}
|
||||
assert _smtp_ready(cfg) is False
|
||||
|
||||
|
||||
# ── XOAUTH2 SASL framing ─────────────────────────────────────────
|
||||
|
||||
def test_xoauth2_raw_is_unencoded_sasl_frame():
|
||||
from routes.email_helpers import _xoauth2_raw
|
||||
|
||||
assert _xoauth2_raw("me@nyu.edu", "tok123") == "user=me@nyu.edu\x01auth=Bearer tok123\x01\x01"
|
||||
|
||||
|
||||
def test_xoauth2_bytes_is_raw_frame_encoded():
|
||||
from routes.email_helpers import _xoauth2_bytes
|
||||
|
||||
assert _xoauth2_bytes("me@nyu.edu", "tok123") == b"user=me@nyu.edu\x01auth=Bearer tok123\x01\x01"
|
||||
|
||||
|
||||
# ── Helpers for in-memory DB fixtures ────────────────────────────
|
||||
|
||||
def _make_db():
|
||||
"""Return (Session, SessionFactory) backed by an isolated in-memory SQLite DB.
|
||||
|
||||
Used to test DB-touching helpers without the real database.
|
||||
The factory lets tests open a fresh session after the helper closes its own.
|
||||
"""
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from core.database import Base
|
||||
engine = create_engine("sqlite:///:memory:", connect_args={"check_same_thread": False})
|
||||
Base.metadata.create_all(engine)
|
||||
Factory = sessionmaker(bind=engine)
|
||||
return Factory(), Factory
|
||||
|
||||
|
||||
def _make_account(session, account_id="acct-1", owner="alice", **kwargs):
|
||||
"""Insert a minimal EmailAccount row and return it."""
|
||||
from core.database import EmailAccount
|
||||
row = EmailAccount(
|
||||
id=account_id,
|
||||
owner=owner,
|
||||
name=kwargs.get("name", "Test"),
|
||||
from_address=kwargs.get("from_address", "test@example.com"),
|
||||
imap_host=kwargs.get("imap_host", "imap.gmail.com"),
|
||||
imap_port=kwargs.get("imap_port", 993),
|
||||
imap_user=kwargs.get("imap_user", "test@example.com"),
|
||||
smtp_host=kwargs.get("smtp_host", "smtp.gmail.com"),
|
||||
smtp_port=kwargs.get("smtp_port", 587),
|
||||
smtp_user=kwargs.get("smtp_user", "test@example.com"),
|
||||
)
|
||||
for k, v in kwargs.items():
|
||||
if hasattr(row, k):
|
||||
setattr(row, k, v)
|
||||
session.add(row)
|
||||
session.commit()
|
||||
return row
|
||||
|
||||
|
||||
# ── Token encryption at rest ─────────────────────────────────────
|
||||
|
||||
def test_refresh_token_stored_encrypted_not_raw():
|
||||
"""_refresh_google_token must encrypt the new access token before writing it
|
||||
to the DB — storing the raw token string would expose credentials at rest."""
|
||||
from src.secret_storage import encrypt as _enc, decrypt as _dec
|
||||
from core.database import EmailAccount
|
||||
|
||||
raw_token = "ya29.test_access_token_raw"
|
||||
|
||||
db, Factory = _make_db()
|
||||
_make_account(db, account_id="acct-r", owner="bob",
|
||||
oauth_refresh_token=_enc("refresh-tok-xyz"))
|
||||
db.close()
|
||||
|
||||
fake_resp = mock.MagicMock()
|
||||
fake_resp.raise_for_status = mock.MagicMock()
|
||||
fake_resp.json.return_value = {"access_token": raw_token, "expires_in": 3600}
|
||||
|
||||
with mock.patch("httpx.post", return_value=fake_resp), \
|
||||
mock.patch("core.database.SessionLocal", Factory), \
|
||||
mock.patch("routes.email_helpers.os.environ.get", side_effect=lambda k, d="": {
|
||||
"GOOGLE_OAUTH_CLIENT_ID": "cid", "GOOGLE_OAUTH_CLIENT_SECRET": "csec"
|
||||
}.get(k, d)):
|
||||
from routes.email_helpers import _refresh_google_token
|
||||
result = _refresh_google_token("acct-r")
|
||||
|
||||
verify_db = Factory()
|
||||
row = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-r").first()
|
||||
stored = row.oauth_access_token
|
||||
verify_db.close()
|
||||
|
||||
assert result == raw_token, "function should return the plain access token to callers"
|
||||
assert stored != raw_token, "raw token must not be stored directly in the DB"
|
||||
assert _dec(stored) == raw_token, "stored value must decrypt back to the raw token"
|
||||
|
||||
|
||||
def test_refresh_stores_encrypted_expiry_not_token():
|
||||
"""oauth_token_expiry stores only a timestamp, never the token value."""
|
||||
from src.secret_storage import encrypt as _enc
|
||||
from core.database import EmailAccount
|
||||
|
||||
db, Factory = _make_db()
|
||||
_make_account(db, account_id="acct-e", owner="bob",
|
||||
oauth_refresh_token=_enc("ref-tok"))
|
||||
db.close()
|
||||
|
||||
fake_resp = mock.MagicMock()
|
||||
fake_resp.raise_for_status = mock.MagicMock()
|
||||
fake_resp.json.return_value = {"access_token": "ya29.secret", "expires_in": 3600}
|
||||
|
||||
with mock.patch("httpx.post", return_value=fake_resp), \
|
||||
mock.patch("core.database.SessionLocal", Factory), \
|
||||
mock.patch("routes.email_helpers.os.environ.get", side_effect=lambda k, d="": {
|
||||
"GOOGLE_OAUTH_CLIENT_ID": "cid", "GOOGLE_OAUTH_CLIENT_SECRET": "csec"
|
||||
}.get(k, d)):
|
||||
from routes.email_helpers import _refresh_google_token
|
||||
_refresh_google_token("acct-e")
|
||||
|
||||
verify_db = Factory()
|
||||
row = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-e").first()
|
||||
expiry = row.oauth_token_expiry
|
||||
verify_db.close()
|
||||
|
||||
assert "ya29" not in (expiry or ""), \
|
||||
"token_expiry must be a timestamp, not the token string"
|
||||
|
||||
|
||||
# ── Real OAuth callback route ─────────────────────────────────────
|
||||
#
|
||||
# These pull the actual google_oauth_callback endpoint out of the router and
|
||||
# invoke it — they pin the real route's behaviour, not a re-implementation, so
|
||||
# they fail if the ownership/state guards are ever removed or weakened.
|
||||
|
||||
def _callback_endpoint():
|
||||
"""Return the live google_oauth_callback endpoint from the email router."""
|
||||
from routes.email_routes import setup_email_routes
|
||||
router = setup_email_routes()
|
||||
for route in router.routes:
|
||||
if route.path == "/api/email/oauth/google/callback" and "GET" in getattr(route, "methods", set()):
|
||||
return route.endpoint
|
||||
raise AssertionError("google_oauth_callback route not found")
|
||||
|
||||
|
||||
class _FakeRequest:
|
||||
"""Minimal stand-in for starlette Request — the callback only reads headers."""
|
||||
headers = {"host": "localhost:7000"}
|
||||
|
||||
|
||||
def _location(resp):
|
||||
"""Pull the redirect target out of a RedirectResponse."""
|
||||
return resp.headers["location"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_callback_missing_code_returns_generic_error():
|
||||
"""No `code` query param → generic error redirect, with no account id, owner,
|
||||
or state echoed back into the URL."""
|
||||
from routes.email_helpers import make_oauth_state
|
||||
|
||||
callback = _callback_endpoint()
|
||||
state = make_oauth_state("acct-1", "alice")
|
||||
resp = await callback(code=None, state=state, error=None, request=_FakeRequest())
|
||||
|
||||
loc = _location(resp)
|
||||
assert "email_oauth_error=missing_code" in loc
|
||||
assert "acct-1" not in loc, "account id must not appear in redirect URL"
|
||||
assert "alice" not in loc, "owner must not appear in redirect URL"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_callback_provider_error_returns_generic_error():
|
||||
"""An `error` from Google → generic error redirect, no raw provider text."""
|
||||
callback = _callback_endpoint()
|
||||
resp = await callback(code=None, state=None, error="access_denied", request=_FakeRequest())
|
||||
|
||||
loc = _location(resp)
|
||||
assert "email_oauth_error=google_error" in loc
|
||||
assert "access_denied" not in loc, "raw provider error must not leak into redirect"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_callback_tampered_state_returns_generic_error_no_leak():
|
||||
"""Tampered/invalid state → invalid_state redirect; the auth code and any
|
||||
token must never appear in the redirect URL."""
|
||||
callback = _callback_endpoint()
|
||||
resp = await callback(code="4/secret-auth-code", state="not-a-valid-state",
|
||||
error=None, request=_FakeRequest())
|
||||
|
||||
loc = _location(resp)
|
||||
assert "email_oauth_error=invalid_state" in loc
|
||||
assert "4/secret-auth-code" not in loc, "auth code must not leak into redirect"
|
||||
assert "token" not in loc
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_callback_owner_mismatch_does_not_write_tokens():
|
||||
"""A signed, valid state whose owner does not match the target account's
|
||||
owner must NOT write tokens — this blocks one authenticated user from
|
||||
binding their Google account onto another user's mailbox row.
|
||||
"""
|
||||
from routes.email_helpers import make_oauth_state
|
||||
from core.database import EmailAccount
|
||||
|
||||
db, Factory = _make_db()
|
||||
_make_account(db, account_id="acct-x", owner="alice")
|
||||
db.close()
|
||||
|
||||
# Token-exchange + userinfo would succeed — the point is the ownership gate
|
||||
# rejects the write *before* trusting them.
|
||||
token_resp = mock.MagicMock()
|
||||
token_resp.raise_for_status = mock.MagicMock()
|
||||
token_resp.json.return_value = {"access_token": "ya29.attacker", "refresh_token": "r", "expires_in": 3600}
|
||||
userinfo_resp = mock.MagicMock()
|
||||
userinfo_resp.is_success = True
|
||||
userinfo_resp.json.return_value = {"email": "bob@evil.com", "name": "Bob"}
|
||||
|
||||
# State is genuinely signed, but for owner "bob" — not the row owner "alice".
|
||||
state = make_oauth_state("acct-x", "bob")
|
||||
|
||||
with mock.patch("httpx.post", return_value=token_resp), \
|
||||
mock.patch("httpx.get", return_value=userinfo_resp), \
|
||||
mock.patch("core.database.SessionLocal", Factory):
|
||||
callback = _callback_endpoint()
|
||||
resp = await callback(code="4/code", state=state, error=None, request=_FakeRequest())
|
||||
|
||||
loc = _location(resp)
|
||||
assert "email_oauth_error=ownership_error" in loc
|
||||
|
||||
verify_db = Factory()
|
||||
row = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-x").first()
|
||||
token_after = row.oauth_access_token
|
||||
verify_db.close()
|
||||
assert token_after is None, "no token may be written when ownership check fails"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_callback_valid_owner_writes_encrypted_tokens_to_intended_account():
|
||||
"""A signed state whose owner matches the target account writes the tokens —
|
||||
and only to that account, stored encrypted (raw token never persisted)."""
|
||||
from routes.email_helpers import make_oauth_state
|
||||
from src.secret_storage import decrypt as _dec
|
||||
from core.database import EmailAccount
|
||||
|
||||
db, Factory = _make_db()
|
||||
_make_account(db, account_id="acct-v", owner="alice", imap_host="", smtp_host="")
|
||||
_make_account(db, account_id="acct-other", owner="alice") # must stay untouched
|
||||
db.close()
|
||||
|
||||
raw_access = "ya29.legit_access_token"
|
||||
raw_refresh = "1//legit_refresh_token"
|
||||
token_resp = mock.MagicMock()
|
||||
token_resp.raise_for_status = mock.MagicMock()
|
||||
token_resp.json.return_value = {"access_token": raw_access, "refresh_token": raw_refresh, "expires_in": 3600}
|
||||
userinfo_resp = mock.MagicMock()
|
||||
userinfo_resp.is_success = True
|
||||
userinfo_resp.json.return_value = {"email": "alice@nyu.edu", "name": "Alice"}
|
||||
|
||||
state = make_oauth_state("acct-v", "alice")
|
||||
|
||||
with mock.patch("httpx.post", return_value=token_resp), \
|
||||
mock.patch("httpx.get", return_value=userinfo_resp), \
|
||||
mock.patch("core.database.SessionLocal", Factory):
|
||||
callback = _callback_endpoint()
|
||||
resp = await callback(code="4/code", state=state, error=None, request=_FakeRequest())
|
||||
|
||||
assert "email_oauth_success=1" in _location(resp)
|
||||
|
||||
verify_db = Factory()
|
||||
target = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-v").first()
|
||||
other = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-other").first()
|
||||
verify_db.close()
|
||||
|
||||
assert target.oauth_provider == "google"
|
||||
assert target.oauth_access_token != raw_access, "access token must be stored encrypted"
|
||||
assert _dec(target.oauth_access_token) == raw_access
|
||||
assert _dec(target.oauth_refresh_token) == raw_refresh
|
||||
assert other.oauth_access_token is None, "tokens must only touch the intended account"
|
||||
|
||||
|
||||
# ── Token refresh scenarios ───────────────────────────────────────
|
||||
|
||||
def test_get_valid_google_token_uses_cached_when_fresh():
|
||||
"""_get_valid_google_token must NOT call refresh when the stored token is
|
||||
still valid (expiry - 60s buffer > now). Refresh is an outbound HTTP call
|
||||
that should only happen when genuinely needed."""
|
||||
from src.secret_storage import encrypt as _enc
|
||||
from routes.email_helpers import _get_valid_google_token
|
||||
|
||||
future_expiry = str(int(time.time()) + 7200) # 2 hours from now
|
||||
cfg = {
|
||||
"account_id": "acct-fresh",
|
||||
"oauth_access_token": _enc("ya29.fresh_token"),
|
||||
"oauth_token_expiry": future_expiry,
|
||||
}
|
||||
|
||||
with mock.patch("routes.email_helpers._refresh_google_token") as mock_refresh:
|
||||
result = _get_valid_google_token("acct-fresh", cfg)
|
||||
|
||||
assert result == "ya29.fresh_token"
|
||||
mock_refresh.assert_not_called()
|
||||
|
||||
|
||||
def test_get_valid_google_token_refreshes_when_expired():
|
||||
"""_get_valid_google_token must call refresh when the token is expired."""
|
||||
from src.secret_storage import encrypt as _enc
|
||||
from routes.email_helpers import _get_valid_google_token
|
||||
|
||||
past_expiry = str(int(time.time()) - 10) # already expired
|
||||
cfg = {
|
||||
"account_id": "acct-exp",
|
||||
"oauth_access_token": _enc("ya29.old_token"),
|
||||
"oauth_token_expiry": past_expiry,
|
||||
}
|
||||
|
||||
with mock.patch("routes.email_helpers._refresh_google_token", return_value="ya29.new_token") as mock_refresh:
|
||||
result = _get_valid_google_token("acct-exp", cfg)
|
||||
|
||||
mock_refresh.assert_called_once_with("acct-exp")
|
||||
assert result == "ya29.new_token"
|
||||
|
||||
|
||||
def test_refresh_failure_returns_none_no_secret_raised():
|
||||
"""When the refresh HTTP call fails, _refresh_google_token must return None
|
||||
silently. It must not raise an exception or surface token/secret details."""
|
||||
from src.secret_storage import encrypt as _enc
|
||||
|
||||
db, Factory = _make_db()
|
||||
_make_account(db, account_id="acct-fail", owner="dave",
|
||||
oauth_refresh_token=_enc("ref-tok"))
|
||||
db.close()
|
||||
|
||||
failing_resp = mock.MagicMock()
|
||||
failing_resp.raise_for_status.side_effect = Exception("401 Unauthorized")
|
||||
|
||||
with mock.patch("httpx.post", return_value=failing_resp), \
|
||||
mock.patch("core.database.SessionLocal", Factory), \
|
||||
mock.patch("routes.email_helpers.os.environ.get", side_effect=lambda k, d="": {
|
||||
"GOOGLE_OAUTH_CLIENT_ID": "cid", "GOOGLE_OAUTH_CLIENT_SECRET": "csec"
|
||||
}.get(k, d)):
|
||||
from routes.email_helpers import _refresh_google_token
|
||||
result = _refresh_google_token("acct-fail")
|
||||
|
||||
assert result is None, "failed refresh must return None, not raise"
|
||||
|
||||
|
||||
def test_refresh_without_credentials_returns_none():
|
||||
"""_refresh_google_token must return None immediately when the OAuth client
|
||||
credentials are not configured — no DB query, no HTTP call."""
|
||||
with mock.patch("routes.email_helpers.os.environ.get", return_value=""):
|
||||
from routes.email_helpers import _refresh_google_token
|
||||
result = _refresh_google_token("acct-any")
|
||||
|
||||
assert result is None
|
||||
|
||||
|
||||
# ── Password-account regression ───────────────────────────────────
|
||||
|
||||
def test_imap_connect_uses_login_for_password_accounts():
|
||||
"""Existing password-auth IMAP accounts must still call conn.login() and
|
||||
must NOT trigger the XOAUTH2 authenticate path."""
|
||||
from routes.email_helpers import _imap_connect
|
||||
|
||||
mock_conn = mock.MagicMock()
|
||||
# _imap_connect calls _get_email_config internally — mock it to return our cfg.
|
||||
cfg = {
|
||||
"imap_host": "imap.gmail.com",
|
||||
"imap_port": 993,
|
||||
"imap_starttls": False,
|
||||
"imap_user": "me@gmail.com",
|
||||
"imap_password": "app-password-xyz",
|
||||
"oauth_provider": "",
|
||||
"account_id": "acct-pw",
|
||||
}
|
||||
|
||||
with mock.patch("routes.email_helpers._open_imap_connection", return_value=mock_conn), \
|
||||
mock.patch("routes.email_helpers._get_email_config", return_value=cfg):
|
||||
_imap_connect("acct-pw", owner="alice")
|
||||
|
||||
mock_conn.login.assert_called_once_with("me@gmail.com", "app-password-xyz")
|
||||
mock_conn.authenticate.assert_not_called()
|
||||
|
||||
|
||||
def test_imap_connect_uses_xoauth2_for_oauth_accounts():
|
||||
"""OAuth accounts must call conn.authenticate('XOAUTH2', ...) and must NOT
|
||||
call conn.login() — which would fail without a password."""
|
||||
from routes.email_helpers import _imap_connect
|
||||
from src.secret_storage import encrypt as _enc
|
||||
|
||||
mock_conn = mock.MagicMock()
|
||||
future_expiry = str(int(time.time()) + 7200)
|
||||
cfg = {
|
||||
"imap_host": "imap.gmail.com",
|
||||
"imap_port": 993,
|
||||
"imap_starttls": False,
|
||||
"imap_user": "me@nyu.edu",
|
||||
"imap_password": "",
|
||||
"oauth_provider": "google",
|
||||
"account_id": "acct-oauth",
|
||||
"oauth_access_token": _enc("ya29.live_token"),
|
||||
"oauth_token_expiry": future_expiry,
|
||||
}
|
||||
|
||||
with mock.patch("routes.email_helpers._open_imap_connection", return_value=mock_conn), \
|
||||
mock.patch("routes.email_helpers._get_email_config", return_value=cfg):
|
||||
_imap_connect("acct-oauth", owner="alice")
|
||||
|
||||
mock_conn.authenticate.assert_called_once()
|
||||
assert mock_conn.authenticate.call_args[0][0] == "XOAUTH2"
|
||||
mock_conn.login.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_account_list_response_does_not_expose_token_values():
|
||||
"""The /accounts list route is the client-facing account inventory. It must
|
||||
expose `oauth_provider` (so the UI can show OAuth status) but never the
|
||||
access/refresh token values, encrypted or otherwise — only boolean
|
||||
has_*_password flags and the provider name."""
|
||||
from routes.email_routes import setup_email_routes
|
||||
from src.secret_storage import encrypt as _enc
|
||||
|
||||
raw_access = "ya29.super_secret_access_token"
|
||||
raw_refresh = "1//super_secret_refresh_token"
|
||||
|
||||
db, Factory = _make_db()
|
||||
_make_account(db, account_id="acct-list", owner="alice",
|
||||
oauth_provider="google",
|
||||
oauth_access_token=_enc(raw_access),
|
||||
oauth_refresh_token=_enc(raw_refresh))
|
||||
db.close()
|
||||
|
||||
router = setup_email_routes()
|
||||
list_accounts = None
|
||||
for route in router.routes:
|
||||
if route.path == "/api/email/accounts" and "GET" in getattr(route, "methods", set()):
|
||||
list_accounts = route.endpoint
|
||||
break
|
||||
assert list_accounts is not None, "accounts list route not found"
|
||||
|
||||
with mock.patch("core.database.SessionLocal", Factory):
|
||||
result = await list_accounts(owner="alice")
|
||||
|
||||
blob = json.dumps(result)
|
||||
assert raw_access not in blob, "raw access token must not appear in list response"
|
||||
assert raw_refresh not in blob, "raw refresh token must not appear in list response"
|
||||
assert _enc(raw_access) not in blob, "encrypted token must not be sent to the client either"
|
||||
|
||||
acct = result["accounts"][0]
|
||||
assert acct["oauth_provider"] == "google" # status is exposed
|
||||
assert "oauth_access_token" not in acct # token value is not
|
||||
assert "oauth_refresh_token" not in acct
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Regression: FASTEMBED_CACHE_DIR must tolerate a PRESENT-but-EMPTY
|
||||
FASTEMBED_CACHE_PATH.
|
||||
|
||||
docker-compose.yml injects ``FASTEMBED_CACHE_PATH=${FASTEMBED_CACHE_PATH:-}``,
|
||||
which sets the variable to ``""`` when the host has not defined it. The old
|
||||
``os.getenv("FASTEMBED_CACHE_PATH", default)`` only used the default when the
|
||||
variable was ABSENT, so an empty value made ``FASTEMBED_CACHE_DIR == ""`` →
|
||||
``os.makedirs("")`` raised ``[Errno 2] No such file or directory: ''`` →
|
||||
FastEmbed failed to initialise and every vector feature (RAG, semantic memory,
|
||||
tool index) silently degraded on the default Docker stack.
|
||||
|
||||
These tests pin the fix: empty is treated like absent → use the DATA_DIR
|
||||
default, while an explicit non-empty override is still honoured.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import os
|
||||
|
||||
import src.constants as constants
|
||||
|
||||
|
||||
def _reload_with(monkeypatch, value):
|
||||
"""Reload src.constants with FASTEMBED_CACHE_PATH set to ``value`` (or
|
||||
removed when ``value`` is None) and return the reloaded module."""
|
||||
if value is None:
|
||||
monkeypatch.delenv("FASTEMBED_CACHE_PATH", raising=False)
|
||||
else:
|
||||
monkeypatch.setenv("FASTEMBED_CACHE_PATH", value)
|
||||
return importlib.reload(constants)
|
||||
|
||||
|
||||
def _restore(monkeypatch):
|
||||
"""Return the module to its env-default state so reloading it here does
|
||||
not leak a test-specific FASTEMBED_CACHE_DIR into other tests."""
|
||||
monkeypatch.delenv("FASTEMBED_CACHE_PATH", raising=False)
|
||||
importlib.reload(constants)
|
||||
|
||||
|
||||
def test_empty_fastembed_cache_path_falls_back_to_default(monkeypatch):
|
||||
"""The bug: an empty FASTEMBED_CACHE_PATH (exactly what Docker injects)
|
||||
must fall back to the DATA_DIR default, never the empty string."""
|
||||
try:
|
||||
mod = _reload_with(monkeypatch, "")
|
||||
assert mod.FASTEMBED_CACHE_DIR, "empty env must not yield an empty path"
|
||||
assert mod.FASTEMBED_CACHE_DIR == os.path.join(mod.DATA_DIR, "fastembed_cache")
|
||||
finally:
|
||||
_restore(monkeypatch)
|
||||
|
||||
|
||||
def test_unset_fastembed_cache_path_uses_default(monkeypatch):
|
||||
"""Sanity: an absent variable also resolves to the default."""
|
||||
try:
|
||||
mod = _reload_with(monkeypatch, None)
|
||||
assert mod.FASTEMBED_CACHE_DIR == os.path.join(mod.DATA_DIR, "fastembed_cache")
|
||||
finally:
|
||||
_restore(monkeypatch)
|
||||
|
||||
|
||||
def test_explicit_fastembed_cache_path_is_respected(monkeypatch):
|
||||
"""A real explicit override must still win — the fix only changes the
|
||||
empty-value handling, not the documented FASTEMBED_CACHE_PATH override."""
|
||||
custom = os.path.join("custom", "fastembed-cache")
|
||||
try:
|
||||
mod = _reload_with(monkeypatch, custom)
|
||||
assert mod.FASTEMBED_CACHE_DIR == custom
|
||||
finally:
|
||||
_restore(monkeypatch)
|
||||
@@ -2,7 +2,14 @@ import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi import HTTPException
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from sqlalchemy.pool import NullPool
|
||||
|
||||
from core.database import Base, GalleryImage
|
||||
|
||||
|
||||
def _gallery_module():
|
||||
@@ -53,6 +60,57 @@ def test_gallery_image_path_rejects_symlink_escape(tmp_path, monkeypatch):
|
||||
assert exc.value.status_code == 400
|
||||
|
||||
|
||||
def test_gallery_replace_rejects_symlink_escape(tmp_path, monkeypatch):
|
||||
gallery_routes = _gallery_module()
|
||||
image_dir = tmp_path / "generated_images"
|
||||
image_dir.mkdir()
|
||||
outside = tmp_path / "outside.png"
|
||||
outside.write_bytes(b"outside image root")
|
||||
link = image_dir / "escape.png"
|
||||
try:
|
||||
os.symlink(outside, link)
|
||||
except (AttributeError, NotImplementedError, OSError) as exc:
|
||||
pytest.skip(f"symlinks unavailable: {exc}")
|
||||
|
||||
engine = create_engine(
|
||||
f"sqlite:///{tmp_path / 'gallery.db'}",
|
||||
connect_args={"check_same_thread": False},
|
||||
poolclass=NullPool,
|
||||
)
|
||||
Base.metadata.create_all(engine)
|
||||
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
db.add(
|
||||
GalleryImage(
|
||||
id="img-1",
|
||||
filename="escape.png",
|
||||
prompt="escape",
|
||||
owner="alice",
|
||||
is_active=True,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
monkeypatch.setattr(gallery_routes, "GALLERY_IMAGE_DIR", image_dir)
|
||||
monkeypatch.setattr(gallery_routes, "SessionLocal", SessionLocal)
|
||||
monkeypatch.setattr(gallery_routes, "get_current_user", lambda request: "alice")
|
||||
|
||||
app = FastAPI()
|
||||
app.include_router(gallery_routes.setup_gallery_routes())
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/gallery/img-1/replace",
|
||||
files={"image": ("replacement.png", b"replacement bytes", "image/png")},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert outside.read_bytes() == b"outside image root"
|
||||
|
||||
|
||||
def test_gallery_file_operations_use_confining_resolver():
|
||||
source = Path("routes/gallery_routes.py").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
@@ -79,7 +79,7 @@ def _build_context_harness(monkeypatch, chat_helpers, history):
|
||||
monkeypatch.setattr(chat_helpers, "extract_preset", fake_extract_preset)
|
||||
monkeypatch.setattr(chat_helpers, "add_user_message", fake_add_user_message)
|
||||
monkeypatch.setattr(chat_helpers, "load_prefs_for_user", lambda user: {})
|
||||
monkeypatch.setattr(chat_helpers, "get_current_user", lambda request: "tester")
|
||||
monkeypatch.setattr(chat_helpers, "effective_user", lambda request: "tester")
|
||||
monkeypatch.setattr(chat_helpers, "normalize_model_id", lambda endpoint_url, model, **kwargs: None)
|
||||
monkeypatch.setattr(chat_helpers, "maybe_compact", fake_maybe_compact)
|
||||
monkeypatch.setattr(chat_helpers, "trim_for_context", lambda messages, context_length: messages)
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
import asyncio
|
||||
|
||||
import mcp_servers.memory_server as memory_server
|
||||
from src.memory import MemoryManager
|
||||
|
||||
|
||||
class FakeVector:
|
||||
healthy = True
|
||||
|
||||
def __init__(self):
|
||||
self.added = []
|
||||
self.removed = []
|
||||
|
||||
def add(self, memory_id, text):
|
||||
self.added.append((memory_id, text))
|
||||
|
||||
def remove(self, memory_id):
|
||||
self.removed.append(memory_id)
|
||||
|
||||
|
||||
def _tool_text(arguments):
|
||||
result = asyncio.run(memory_server.call_tool("manage_memory", arguments))
|
||||
return result[0].text
|
||||
|
||||
|
||||
def _entry(manager, text, owner=None, memory_id=None, category="fact"):
|
||||
entry = manager.add_entry(text, owner=owner, category=category)
|
||||
if memory_id:
|
||||
entry["id"] = memory_id
|
||||
return entry
|
||||
|
||||
|
||||
def _configure_server(monkeypatch, manager, vector=None):
|
||||
monkeypatch.setattr(memory_server, "_memory_manager", manager)
|
||||
monkeypatch.setattr(memory_server, "_memory_vector", vector)
|
||||
monkeypatch.setattr(memory_server, "_initialized", True)
|
||||
for key in memory_server._OWNER_ENV_KEYS:
|
||||
monkeypatch.delenv(key, raising=False)
|
||||
|
||||
|
||||
def test_mcp_memory_uses_configured_owner_for_all_operations(monkeypatch, tmp_path):
|
||||
manager = MemoryManager(str(tmp_path))
|
||||
vector = FakeVector()
|
||||
alice = _entry(
|
||||
manager,
|
||||
"Alice likes green tea",
|
||||
owner="alice",
|
||||
memory_id="aaaaaaaa-0000-0000-0000-000000000000",
|
||||
)
|
||||
bob = _entry(
|
||||
manager,
|
||||
"Bob likes espresso",
|
||||
owner="bob",
|
||||
memory_id="bbbbbbbb-0000-0000-0000-000000000000",
|
||||
)
|
||||
manager.save([alice, bob])
|
||||
_configure_server(monkeypatch, manager, vector)
|
||||
monkeypatch.setenv("ODYSSEUS_MCP_MEMORY_OWNER", "alice")
|
||||
|
||||
list_text = _tool_text({"action": "list"})
|
||||
assert "Alice likes green tea" in list_text
|
||||
assert "Bob likes espresso" not in list_text
|
||||
|
||||
search_text = _tool_text({"action": "search", "text": "likes"})
|
||||
assert "Alice likes green tea" in search_text
|
||||
assert "Bob likes espresso" not in search_text
|
||||
|
||||
add_text = _tool_text({
|
||||
"action": "add",
|
||||
"text": "Alice prefers concise notes",
|
||||
"category": "preference",
|
||||
})
|
||||
assert "Memory added" in add_text
|
||||
added = next(
|
||||
entry for entry in manager.load_all()
|
||||
if entry["text"] == "Alice prefers concise notes"
|
||||
)
|
||||
assert added["owner"] == "alice"
|
||||
assert vector.added == [(added["id"], "Alice prefers concise notes")]
|
||||
|
||||
edit_text = _tool_text({
|
||||
"action": "edit",
|
||||
"memory_id": bob["id"][:8],
|
||||
"text": "Bob changed",
|
||||
})
|
||||
assert edit_text == "Error: Memory 'bbbbbbbb' not found"
|
||||
bob_after_edit = next(
|
||||
entry for entry in manager.load_all()
|
||||
if entry["id"] == bob["id"]
|
||||
)
|
||||
assert bob_after_edit["text"] == "Bob likes espresso"
|
||||
|
||||
delete_text = _tool_text({"action": "delete", "memory_id": bob["id"][:8]})
|
||||
assert delete_text == "Error: Memory 'bbbbbbbb' not found"
|
||||
assert any(entry["id"] == bob["id"] for entry in manager.load_all())
|
||||
|
||||
|
||||
def test_mcp_memory_fails_closed_without_owner_for_owner_scoped_store(monkeypatch, tmp_path):
|
||||
manager = MemoryManager(str(tmp_path))
|
||||
alice = _entry(manager, "Alice private memory", owner="alice", memory_id="aaaaaaaa-0000")
|
||||
bob = _entry(manager, "Bob private memory", owner="bob", memory_id="bbbbbbbb-0000")
|
||||
manager.save([alice, bob])
|
||||
_configure_server(monkeypatch, manager, FakeVector())
|
||||
before = manager.load_all()
|
||||
|
||||
actions = [
|
||||
{"action": "list"},
|
||||
{"action": "search", "text": "private"},
|
||||
{"action": "add", "text": "new ownerless memory"},
|
||||
{"action": "edit", "memory_id": alice["id"][:8], "text": "changed"},
|
||||
{"action": "delete", "memory_id": alice["id"][:8]},
|
||||
]
|
||||
|
||||
for arguments in actions:
|
||||
assert _tool_text(arguments).startswith("Error: Memory MCP owner is not configured")
|
||||
|
||||
assert manager.load_all() == before
|
||||
|
||||
|
||||
def test_mcp_memory_preserves_ownerless_local_behavior(monkeypatch, tmp_path):
|
||||
manager = MemoryManager(str(tmp_path))
|
||||
legacy = _entry(
|
||||
manager,
|
||||
"Legacy local memory",
|
||||
memory_id="llllllll-0000-0000-0000-000000000000",
|
||||
)
|
||||
manager.save([legacy])
|
||||
_configure_server(monkeypatch, manager, FakeVector())
|
||||
|
||||
assert "Legacy local memory" in _tool_text({"action": "list"})
|
||||
assert "Legacy local memory" in _tool_text({"action": "search", "text": "legacy"})
|
||||
|
||||
add_text = _tool_text({"action": "add", "text": "Another local memory"})
|
||||
assert "Memory added" in add_text
|
||||
added = next(
|
||||
entry for entry in manager.load_all()
|
||||
if entry["text"] == "Another local memory"
|
||||
)
|
||||
assert "owner" not in added
|
||||
|
||||
assert _tool_text({
|
||||
"action": "edit",
|
||||
"memory_id": legacy["id"][:8],
|
||||
"text": "Updated local memory",
|
||||
}) == "Memory updated: Updated local memory"
|
||||
assert any(entry["text"] == "Updated local memory" for entry in manager.load_all())
|
||||
|
||||
delete_text = _tool_text({"action": "delete", "memory_id": legacy["id"][:8]})
|
||||
assert delete_text.startswith("Memory deleted:")
|
||||
assert all(entry["id"] != legacy["id"] for entry in manager.load_all())
|
||||
@@ -1286,6 +1286,14 @@ class _ImmediateThread:
|
||||
self.target()
|
||||
|
||||
|
||||
class _NoopThread:
|
||||
def __init__(self, target, daemon=None):
|
||||
self.target = target
|
||||
|
||||
def start(self):
|
||||
return None
|
||||
|
||||
|
||||
def _wait_for(predicate, timeout=2.0):
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
@@ -1313,6 +1321,7 @@ def _route_ep(
|
||||
pinned_models=None,
|
||||
refresh_mode="auto",
|
||||
refresh_timeout=None,
|
||||
owner=None,
|
||||
):
|
||||
return SimpleNamespace(
|
||||
id=id,
|
||||
@@ -1329,7 +1338,7 @@ def _route_ep(
|
||||
model_refresh_interval=None,
|
||||
model_refresh_timeout=refresh_timeout,
|
||||
supports_tools=None,
|
||||
owner=None,
|
||||
owner=owner,
|
||||
created_at=None,
|
||||
updated_at=None,
|
||||
)
|
||||
@@ -1342,6 +1351,72 @@ def _route_request():
|
||||
)
|
||||
|
||||
|
||||
def test_api_models_rejects_api_token_without_chat_scope(monkeypatch):
|
||||
router = model_routes.setup_model_routes(model_discovery=None)
|
||||
|
||||
def fail_session():
|
||||
raise AssertionError("model DB should not be queried without chat scope")
|
||||
|
||||
monkeypatch.setattr(model_routes, "SessionLocal", fail_session)
|
||||
|
||||
request = SimpleNamespace(
|
||||
state=SimpleNamespace(
|
||||
current_user="api",
|
||||
api_token=True,
|
||||
api_token_owner="alice",
|
||||
api_token_scopes=["documents:read"],
|
||||
),
|
||||
app=SimpleNamespace(
|
||||
state=SimpleNamespace(
|
||||
auth_manager=SimpleNamespace(is_configured=True, is_admin=lambda user: False),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
_route_endpoint(router, "/api/models")(request)
|
||||
|
||||
assert exc.value.status_code == 403
|
||||
assert "chat" in str(exc.value.detail)
|
||||
|
||||
|
||||
def test_api_models_scopes_api_token_to_token_owner(monkeypatch):
|
||||
rows = [
|
||||
_route_ep("alice", "http://alice.example/v1", cached_models=["alice-model"], owner="alice"),
|
||||
_route_ep("shared", "http://shared.example/v1", cached_models=["shared-model"], owner=None),
|
||||
_route_ep("bob", "http://bob.example/v1", cached_models=["bob-model"], owner="bob"),
|
||||
]
|
||||
db = _RouteDb(rows)
|
||||
router = model_routes.setup_model_routes(model_discovery=None)
|
||||
admin_checks = []
|
||||
|
||||
monkeypatch.setattr(model_routes, "ModelEndpoint", _RouteModelEndpoint)
|
||||
monkeypatch.setattr(model_routes, "SessionLocal", lambda: db)
|
||||
monkeypatch.setattr(threading, "Thread", _NoopThread)
|
||||
|
||||
request = SimpleNamespace(
|
||||
state=SimpleNamespace(
|
||||
current_user="api",
|
||||
api_token=True,
|
||||
api_token_owner="alice",
|
||||
api_token_scopes=["chat"],
|
||||
),
|
||||
app=SimpleNamespace(
|
||||
state=SimpleNamespace(
|
||||
auth_manager=SimpleNamespace(
|
||||
is_configured=True,
|
||||
is_admin=lambda user: admin_checks.append(user) or False,
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
result = _route_endpoint(router, "/api/models")(request)
|
||||
|
||||
assert [item["endpoint_name"] for item in result["items"]] == ["alice", "shared"]
|
||||
assert admin_checks == ["alice"]
|
||||
|
||||
|
||||
def test_api_models_returns_cached_proxy_models_without_refresh_probe(monkeypatch):
|
||||
row = _route_ep(
|
||||
"proxy",
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
"""Owner-scoped note routes must fail closed when the request has no identity.
|
||||
|
||||
The notes CRUD routes resolved the acting user with bare get_current_user().
|
||||
A request that reached them carrying no identity (auth-middleware regression,
|
||||
SSRF from a sibling service) therefore came through as user=None — and the
|
||||
queries treat None as the single-user mode, i.e. blanket access to every
|
||||
account's notes: list everything, read/update/delete/pin/archive any row,
|
||||
reorder globally.
|
||||
|
||||
require_user() already encodes the correct policy — 401 when auth is
|
||||
configured, while the documented anonymous modes (AUTH_ENABLED=false,
|
||||
LOCALHOST_BYPASS on loopback, unconfigured first-run) still pass — and
|
||||
fire-reminder in the same file already used it. The CRUD routes now resolve
|
||||
the owner through it too.
|
||||
|
||||
Test transport note: these drive the ASGI app through ``httpx.ASGITransport``
|
||||
+ ``httpx.AsyncClient`` rather than ``starlette.testclient.TestClient``.
|
||||
TestClient runs the app inside a background event-loop thread spun up by
|
||||
``anyio.from_thread.start_blocking_portal`` and then dispatches each sync
|
||||
endpoint onto *another* worker thread; on some anyio/httpx/platform
|
||||
combinations that two-thread handshake deadlocks and ``TestClient(app).get(...)``
|
||||
simply hangs. ASGITransport runs the whole request on the test's own event
|
||||
loop — no portal thread, no BaseHTTPMiddleware — so the suite is portable.
|
||||
Identity is injected by a pure-ASGI shim that writes the same
|
||||
``request.state`` fields the real auth middleware sets.
|
||||
"""
|
||||
import uuid
|
||||
from types import SimpleNamespace
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from sqlalchemy.pool import NullPool
|
||||
|
||||
import core.database as cdb
|
||||
from core.database import Note
|
||||
import routes.note_routes as nr
|
||||
|
||||
|
||||
# A deliberately NON-loopback peer. require_user has loopback fall-throughs
|
||||
# (unconfigured first-run, LOCALHOST_BYPASS); pinning a public-looking client
|
||||
# keeps every assertion below about the *configured-auth* path and not an
|
||||
# accidental loopback bypass — the same reason the old fixture leaned on
|
||||
# TestClient's non-loopback "testclient" host.
|
||||
_PEER = ("203.0.113.7", 54321)
|
||||
|
||||
|
||||
class _Identity:
|
||||
"""Pure-ASGI shim mirroring what the auth middleware writes onto
|
||||
request.state. Pure-ASGI on purpose — it stays off Starlette's
|
||||
BaseHTTPMiddleware + sync-TestClient path, the source of the
|
||||
``TestClient(app).get(...)`` hang. No x-test-user header => no identity,
|
||||
the exact state an auth-middleware regression would produce."""
|
||||
|
||||
def __init__(self, app):
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] == "http":
|
||||
headers = dict(scope.get("headers") or [])
|
||||
state = scope.setdefault("state", {})
|
||||
user = headers.get(b"x-test-user")
|
||||
if user:
|
||||
state["current_user"] = user.decode()
|
||||
if headers.get(b"x-test-api-token"):
|
||||
state["current_user"] = "api"
|
||||
state["api_token"] = True
|
||||
await self.app(scope, receive, send)
|
||||
|
||||
|
||||
def _temp_db(tmp_path):
|
||||
"""Note routes over a fresh temp DB; returns the session factory."""
|
||||
engine = create_engine(
|
||||
f"sqlite:///{tmp_path / 'notes.db'}",
|
||||
connect_args={"check_same_thread": False},
|
||||
poolclass=NullPool,
|
||||
)
|
||||
cdb.Base.metadata.create_all(engine)
|
||||
return sessionmaker(bind=engine)
|
||||
|
||||
|
||||
def _build_app(factory, *, configured=True):
|
||||
app = FastAPI()
|
||||
app.state.auth_manager = SimpleNamespace(is_configured=configured)
|
||||
app.include_router(nr.setup_note_routes())
|
||||
return _Identity(app)
|
||||
|
||||
|
||||
def _client(app):
|
||||
"""AsyncClient over the ASGI app with a non-loopback peer. Caller drives
|
||||
it inside ``async with``."""
|
||||
transport = httpx.ASGITransport(app=app, client=_PEER)
|
||||
return httpx.AsyncClient(transport=transport, base_url="http://notes.test")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def env(monkeypatch, tmp_path):
|
||||
"""Configured-auth world: AUTH_ENABLED=true, auth_manager.is_configured,
|
||||
no LOCALHOST_BYPASS. Identity comes only from the x-test-user header
|
||||
(mirroring the auth middleware); no header => no identity, the exact state
|
||||
an auth-middleware regression leaves behind. Seeds one note each for alice
|
||||
and bob. Returns (app, factory)."""
|
||||
factory = _temp_db(tmp_path)
|
||||
monkeypatch.setattr(nr, "SessionLocal", factory)
|
||||
monkeypatch.setenv("AUTH_ENABLED", "true")
|
||||
monkeypatch.delenv("LOCALHOST_BYPASS", raising=False)
|
||||
|
||||
app = _build_app(factory)
|
||||
|
||||
db = factory()
|
||||
db.add(Note(id="note-alice", owner="alice", title="a", content="x",
|
||||
items='[{"text": "t", "done": false}]'))
|
||||
db.add(Note(id="note-bob", owner="bob", title="b", content="y"))
|
||||
db.commit()
|
||||
db.close()
|
||||
return app, factory
|
||||
|
||||
|
||||
async def test_no_identity_fails_closed_on_every_owner_scoped_route(env):
|
||||
app, _ = env
|
||||
async with _client(app) as c:
|
||||
assert (await c.get("/api/notes")).status_code == 401
|
||||
assert (await c.get("/api/notes/note-alice")).status_code == 401
|
||||
assert (await c.put("/api/notes/note-alice", json={"title": "pwn"})).status_code == 401
|
||||
assert (await c.delete("/api/notes/note-alice")).status_code == 401
|
||||
assert (await c.post("/api/notes/note-alice/pin")).status_code == 401
|
||||
assert (await c.post("/api/notes/note-alice/archive")).status_code == 401
|
||||
assert (await c.post("/api/notes/note-alice/items/0/toggle")).status_code == 401
|
||||
assert (await c.post("/api/notes/reorder", json={"ids": ["note-bob", "note-alice"]})).status_code == 401
|
||||
assert (await c.post("/api/notes", json={"title": "ghost"})).status_code == 401
|
||||
|
||||
|
||||
async def test_no_identity_did_not_mutate_anything(env):
|
||||
app, factory = env
|
||||
async with _client(app) as c:
|
||||
await c.put("/api/notes/note-alice", json={"title": "pwn"})
|
||||
await c.post("/api/notes/note-alice/pin")
|
||||
await c.delete("/api/notes/note-bob")
|
||||
db = factory()
|
||||
rows = {n.id: n for n in db.query(Note).all()}
|
||||
db.close()
|
||||
assert set(rows) == {"note-alice", "note-bob"}
|
||||
assert rows["note-alice"].title == "a"
|
||||
assert not rows["note-alice"].pinned
|
||||
|
||||
|
||||
async def test_authenticated_user_still_scoped_to_own_notes(env):
|
||||
app, _ = env
|
||||
alice = {"x-test-user": "alice"}
|
||||
async with _client(app) as c:
|
||||
listed = (await c.get("/api/notes", headers=alice)).json()["notes"]
|
||||
assert [n["id"] for n in listed] == ["note-alice"]
|
||||
assert (await c.get("/api/notes/note-alice", headers=alice)).status_code == 200
|
||||
# Someone else's note stays a 404 (don't reveal it exists).
|
||||
assert (await c.get("/api/notes/note-bob", headers=alice)).status_code == 404
|
||||
assert (await c.put("/api/notes/note-alice", json={"title": "mine"}, headers=alice)).status_code == 200
|
||||
|
||||
|
||||
async def test_api_token_pseudo_user_is_rejected(env):
|
||||
"""Bearer tokens must use the scope-aware API routes (require_user's
|
||||
existing contract), not slip into cookie-session routes as user 'api'."""
|
||||
app, _ = env
|
||||
async with _client(app) as c:
|
||||
r = await c.get("/api/notes", headers={"x-test-api-token": "1"})
|
||||
assert r.status_code == 403
|
||||
|
||||
|
||||
async def test_auth_disabled_keeps_single_user_mode_working(monkeypatch, tmp_path):
|
||||
"""AUTH_ENABLED=false is the operator's explicit anonymous mode: no
|
||||
identity must still mean full single-user access (issue #622 contract),
|
||||
even with a stale configured auth.json on disk."""
|
||||
factory = _temp_db(tmp_path)
|
||||
monkeypatch.setattr(nr, "SessionLocal", factory)
|
||||
monkeypatch.setenv("AUTH_ENABLED", "false")
|
||||
|
||||
app = _build_app(factory)
|
||||
|
||||
db = factory()
|
||||
db.add(Note(id="n1", owner=None, title="solo", content="x"))
|
||||
db.commit()
|
||||
db.close()
|
||||
|
||||
async with _client(app) as c:
|
||||
assert [n["id"] for n in (await c.get("/api/notes")).json()["notes"]] == ["n1"]
|
||||
assert (await c.put("/api/notes/n1", json={"title": "still mine"})).status_code == 200
|
||||
assert (await c.post("/api/notes/n1/pin")).status_code == 200
|
||||
@@ -0,0 +1,74 @@
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from routes import personal_routes
|
||||
|
||||
|
||||
class _FakePersonalDocs:
|
||||
def __init__(self):
|
||||
self.excluded = []
|
||||
|
||||
def exclude_file(self, filepath):
|
||||
self.excluded.append(filepath)
|
||||
|
||||
|
||||
class _FakeRAG:
|
||||
def __init__(self):
|
||||
self.deleted_sources = []
|
||||
|
||||
def delete_by_source(self, filepath):
|
||||
self.deleted_sources.append(filepath)
|
||||
return 1
|
||||
|
||||
|
||||
def _delete_endpoint(personal_docs):
|
||||
router = personal_routes.setup_personal_routes(personal_docs, None, True)
|
||||
for route in router.routes:
|
||||
if getattr(route, "path", "") == "/api/personal/file" and "DELETE" in getattr(route, "methods", set()):
|
||||
return route.endpoint
|
||||
raise AssertionError("DELETE /api/personal/file endpoint not found")
|
||||
|
||||
|
||||
def test_delete_file_refuses_symlink_directory_escape(tmp_path, monkeypatch):
|
||||
uploads = tmp_path / "uploads"
|
||||
uploads.mkdir()
|
||||
outside = tmp_path / "outside"
|
||||
outside.mkdir()
|
||||
victim = outside / "victim.txt"
|
||||
victim.write_text("keep me", encoding="utf-8")
|
||||
os.symlink(outside, uploads / "linked")
|
||||
|
||||
docs = _FakePersonalDocs()
|
||||
rag = _FakeRAG()
|
||||
monkeypatch.setattr(personal_routes, "UPLOADS_DIR", str(uploads))
|
||||
monkeypatch.setattr(personal_routes, "get_rag_manager", lambda: rag)
|
||||
|
||||
filepath = str(uploads / "linked" / "victim.txt")
|
||||
result = asyncio.run(_delete_endpoint(docs)(filepath=filepath, owner="alice", _admin=None))
|
||||
|
||||
assert result["deleted_from_disk"] is False
|
||||
assert victim.read_text(encoding="utf-8") == "keep me"
|
||||
assert docs.excluded == [filepath]
|
||||
assert rag.deleted_sources == [filepath]
|
||||
|
||||
|
||||
def test_delete_file_removes_regular_file_inside_upload_root(tmp_path, monkeypatch):
|
||||
uploads = tmp_path / "uploads"
|
||||
uploads.mkdir()
|
||||
uploaded_file = uploads / "alice" / "notes.txt"
|
||||
uploaded_file.parent.mkdir()
|
||||
uploaded_file.write_text("delete me", encoding="utf-8")
|
||||
|
||||
docs = _FakePersonalDocs()
|
||||
rag = _FakeRAG()
|
||||
monkeypatch.setattr(personal_routes, "UPLOADS_DIR", str(uploads))
|
||||
monkeypatch.setattr(personal_routes, "get_rag_manager", lambda: rag)
|
||||
|
||||
filepath = str(uploaded_file)
|
||||
result = asyncio.run(_delete_endpoint(docs)(filepath=filepath, owner="alice", _admin=None))
|
||||
|
||||
assert result["deleted_from_disk"] is True
|
||||
assert not uploaded_file.exists()
|
||||
assert docs.excluded == [filepath]
|
||||
assert rag.deleted_sources == [filepath]
|
||||
@@ -1,16 +1,18 @@
|
||||
"""Regression guard for issue #1390 — the README banner / ASCII art was not in a
|
||||
fenced code block, so GitHub's markdown collapsed its leading whitespace and the
|
||||
box-drawing rules, rendering it misaligned instead of monospace-as-typed.
|
||||
"""Regression guard for the README title presentation.
|
||||
|
||||
This pins that the decorative banner stays inside a ``` code fence.
|
||||
Originally (#1390) the README opened with an ASCII-art banner that had to live
|
||||
inside a ``` code fence, otherwise GitHub's markdown collapsed its leading
|
||||
whitespace and box-drawing rules and rendered it misaligned. The README refresh
|
||||
(#4306) dropped that banner in favour of a centered wordmark image, so the guard
|
||||
now pins the wordmark identity instead, while still catching the original failure
|
||||
mode if an un-fenced ASCII banner is ever reintroduced.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
README = Path(__file__).resolve().parent.parent / "README.md"
|
||||
|
||||
# Distinctive bits of the banner (box-drawing rule + the kaomoji version line).
|
||||
# Box-drawing rule from the legacy ASCII banner (the #1390 failure mode).
|
||||
_RULE = "─" * 10
|
||||
_BANNER_LINE = "Odysseus vers. 1.0"
|
||||
|
||||
|
||||
def _fenced_segments(text: str):
|
||||
@@ -20,15 +22,18 @@ def _fenced_segments(text: str):
|
||||
return parts[1::2]
|
||||
|
||||
|
||||
def test_readme_banner_is_inside_a_code_fence():
|
||||
def test_readme_opens_with_wordmark_title():
|
||||
# The README must still open with a recognizable Odysseus title: now the
|
||||
# centered wordmark image rather than an H1 / ASCII banner.
|
||||
head = "\n".join(README.read_text(encoding="utf-8").splitlines()[:15])
|
||||
assert 'alt="Odysseus"' in head, "README must open with the Odysseus wordmark image"
|
||||
|
||||
|
||||
def test_reintroduced_ascii_banner_stays_fenced():
|
||||
# Defensive: if a box-drawing banner is ever added back, it must be fenced so
|
||||
# GitHub renders it monospace-as-typed (the original #1390 regression).
|
||||
text = README.read_text(encoding="utf-8")
|
||||
assert _BANNER_LINE in text, "banner line missing from README"
|
||||
if _RULE not in text:
|
||||
return
|
||||
inside = "\n".join(_fenced_segments(text))
|
||||
assert _BANNER_LINE in inside, "banner version line must be inside a ``` code fence"
|
||||
assert _RULE in inside, "banner rule line must be inside a ``` code fence"
|
||||
|
||||
|
||||
def test_readme_title_stays_a_heading():
|
||||
# The H1 must remain a real heading, not get swallowed into the fence.
|
||||
first = README.read_text(encoding="utf-8").splitlines()[0]
|
||||
assert first.strip() == "# Odysseus"
|
||||
assert _RULE in inside, "ASCII banner rule must be inside a ``` code fence"
|
||||
|
||||
@@ -385,7 +385,7 @@ async def test_build_chat_context_incognito_does_not_duplicate_current_user_mess
|
||||
monkeypatch.setattr(chat_helpers, "extract_preset", fake_extract_preset)
|
||||
monkeypatch.setattr(chat_helpers, "add_user_message", fake_add_user_message)
|
||||
monkeypatch.setattr(chat_helpers, "load_prefs_for_user", lambda user: {})
|
||||
monkeypatch.setattr(chat_helpers, "get_current_user", lambda request: "tester")
|
||||
monkeypatch.setattr(chat_helpers, "effective_user", lambda request: "tester")
|
||||
monkeypatch.setattr(chat_helpers, "normalize_model_id", lambda endpoint_url, model, **kwargs: None)
|
||||
monkeypatch.setattr(chat_helpers, "maybe_compact", fake_maybe_compact)
|
||||
monkeypatch.setattr(chat_helpers, "trim_for_context", lambda messages, context_length: messages)
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
import os
|
||||
import sys
|
||||
from unittest import mock
|
||||
import pytest
|
||||
from src.runtime_paths import get_app_root, get_default_data_dir
|
||||
|
||||
|
||||
def test_get_app_root_normal_run():
|
||||
"""Verify that get_app_root returns the repository root parent of src/ when not frozen."""
|
||||
with mock.patch.object(sys, "frozen", False, create=True):
|
||||
app_root = get_app_root()
|
||||
# Verify it is a valid directory path and matches expected parent structure
|
||||
assert os.path.isdir(app_root)
|
||||
assert os.path.exists(os.path.join(app_root, "src"))
|
||||
|
||||
|
||||
def test_get_app_root_frozen_with_meipass():
|
||||
"""Verify that get_app_root returns the sys._MEIPASS directory when frozen by PyInstaller."""
|
||||
mock_meipass = os.path.abspath("mock_meipass_dir")
|
||||
with mock.patch.object(sys, "frozen", True, create=True), \
|
||||
mock.patch.object(sys, "_MEIPASS", mock_meipass, create=True):
|
||||
app_root = get_app_root()
|
||||
assert app_root == mock_meipass
|
||||
|
||||
|
||||
def test_get_app_root_frozen_without_meipass():
|
||||
"""Verify that get_app_root falls back to the sys.executable parent directory when frozen but _MEIPASS is absent."""
|
||||
mock_exe_path = os.path.join(os.path.abspath("mock_exe_dir"), "Odysseus.exe")
|
||||
with mock.patch.object(sys, "frozen", True, create=True), \
|
||||
mock.patch.object(sys, "executable", mock_exe_path, create=True):
|
||||
# Remove sys._MEIPASS if it exists in the test process environment
|
||||
if hasattr(sys, "_MEIPASS"):
|
||||
delattr(sys, "_MEIPASS")
|
||||
app_root = get_app_root()
|
||||
assert app_root == os.path.abspath("mock_exe_dir")
|
||||
|
||||
|
||||
def test_get_default_data_dir_normal():
|
||||
"""Verify that get_default_data_dir resolves to get_app_root() / 'data' when not frozen."""
|
||||
with mock.patch.object(sys, "frozen", False, create=True):
|
||||
res = get_default_data_dir()
|
||||
assert res == os.path.join(get_app_root(), "data")
|
||||
|
||||
|
||||
def test_get_default_data_dir_frozen():
|
||||
"""Verify that get_default_data_dir resolves to a persistent user path under ~ when frozen."""
|
||||
with mock.patch.object(sys, "frozen", True, create=True):
|
||||
res = get_default_data_dir()
|
||||
expected = os.path.join(os.path.expanduser("~"), ".odysseus", "data")
|
||||
assert res == expected
|
||||
@@ -58,7 +58,7 @@ def test_content_fetcher_extracts_og_image_and_body_fallback(module, tmp_path, m
|
||||
|
||||
monkeypatch.setattr(module, "CONTENT_CACHE_DIR", tmp_path)
|
||||
module.content_cache_index.clear()
|
||||
monkeypatch.setattr(module, "_get_public_url", lambda url, headers, timeout: _FakeResponse(html))
|
||||
monkeypatch.setattr(module, "_get_public_url", lambda url, headers, timeout, **kwargs: _FakeResponse(html))
|
||||
|
||||
result = module.fetch_webpage_content("https://example.com/parity-test")
|
||||
|
||||
@@ -82,7 +82,7 @@ def test_fetch_webpage_content_returns_empty_result_on_http_status_error(status_
|
||||
monkeypatch.setattr(
|
||||
service_content,
|
||||
"_get_public_url",
|
||||
lambda url, headers, timeout: _FakeErrorResponse(status_code),
|
||||
lambda url, headers, timeout, **kwargs: _FakeErrorResponse(status_code),
|
||||
)
|
||||
|
||||
result = service_content.fetch_webpage_content(f"https://example.com/status-{status_code}")
|
||||
@@ -119,7 +119,7 @@ def test_fetch_webpage_content_429_takes_distinct_rate_limit_path(tmp_path, monk
|
||||
monkeypatch.setattr(
|
||||
service_content,
|
||||
"_get_public_url",
|
||||
lambda url, headers, timeout: _FakeRateLimitResponse(),
|
||||
lambda url, headers, timeout, **kwargs: _FakeRateLimitResponse(),
|
||||
)
|
||||
|
||||
result = service_content.fetch_webpage_content("https://example.com/rate-limited")
|
||||
|
||||
@@ -121,9 +121,12 @@ def test_docker_compose_binds_web_ui_to_loopback_by_default():
|
||||
|
||||
|
||||
def test_readme_native_quickstart_uses_loopback():
|
||||
readme = Path("README.md").read_text(encoding="utf-8")
|
||||
assert "python -m uvicorn app:app --host 127.0.0.1 --port 7000" in readme
|
||||
assert "0.0.0.0` only when you intentionally want" in readme
|
||||
# The README refresh (#4306) moved the native quickstart into docs/setup.md,
|
||||
# so accept the loopback guidance from either the README or the setup guide.
|
||||
docs = Path("README.md").read_text(encoding="utf-8")
|
||||
docs += "\n" + Path("docs/setup.md").read_text(encoding="utf-8")
|
||||
assert "python -m uvicorn app:app --host 127.0.0.1 --port 7000" in docs
|
||||
assert "0.0.0.0` only when you intentionally want" in docs
|
||||
|
||||
|
||||
def test_ollama_cookbook_runner_does_not_force_public_bind():
|
||||
@@ -901,7 +904,13 @@ def test_web_fetch_guard_blocks_redirect_into_private(monkeypatch):
|
||||
url = "http://public.example/start"
|
||||
headers = {"location": "http://169.254.169.254/latest/meta-data/"}
|
||||
|
||||
monkeypatch.setattr(httpx, "get", lambda url, **kwargs: _Resp())
|
||||
from contextlib import contextmanager
|
||||
|
||||
@contextmanager
|
||||
def _fake_stream(method, url, **kwargs):
|
||||
yield _Resp()
|
||||
|
||||
monkeypatch.setattr(httpx, "stream", _fake_stream)
|
||||
|
||||
with _pytest.raises(httpx.RequestError) as exc:
|
||||
content._get_public_url("http://public.example/start", headers={}, timeout=5)
|
||||
|
||||
@@ -52,6 +52,6 @@ def test_chat_endpoint_recovery_paths_are_owner_scoped():
|
||||
assert "def _clear_orphaned_session_endpoint(sess, owner:" in chat_routes
|
||||
assert "def _recover_empty_session_model(sess, session_id: str, owner:" in chat_routes
|
||||
assert "q = owner_filter(q, ModelEndpoint, owner)" in chat_routes
|
||||
assert "resolve_session_auth(sess, session, owner=get_current_user(request))" in chat_routes
|
||||
assert "resolve_session_auth(sess, session, owner=effective_user(request))" in chat_routes
|
||||
assert "def resolve_session_auth(sess, session_id: str, owner:" in chat_helpers
|
||||
assert "update_q = update_q.filter(DBSession.owner == owner)" in chat_helpers
|
||||
|
||||
@@ -35,7 +35,7 @@ def _patch_fetch(monkeypatch, text, content_type):
|
||||
monkeypatch.setattr(
|
||||
content_mod,
|
||||
"_get_public_url",
|
||||
lambda url, headers=None, timeout=5: _FakeResponse(text, content_type),
|
||||
lambda url, headers=None, timeout=5, **kwargs: _FakeResponse(text, content_type),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,206 @@
|
||||
"""web_fetch download budgets (#3812).
|
||||
|
||||
MAX_OUTPUT_CHARS only trims what the agent sees; these caps bound what the
|
||||
server downloads, parses, and caches. Soft cap by default with a truncation
|
||||
notice, per-call override clamped to the hard cap, and a pre-buffer refusal
|
||||
when Content-Length already exceeds the hard ceiling.
|
||||
"""
|
||||
import json
|
||||
from contextlib import contextmanager
|
||||
|
||||
import pytest
|
||||
|
||||
from src.constants import WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES
|
||||
from services.search import content as content_mod
|
||||
|
||||
|
||||
class _FakeStream:
|
||||
"""Stands in for the httpx.stream(...) context manager."""
|
||||
|
||||
def __init__(self, body: bytes, content_type="text/plain", content_length=None,
|
||||
status_code=200, chunk=8192):
|
||||
self._body = body
|
||||
self._chunk = chunk
|
||||
self.status_code = status_code
|
||||
self.encoding = "utf-8"
|
||||
self.url = "https://example.com/x"
|
||||
self.headers = {"Content-Type": content_type}
|
||||
if content_length is not None:
|
||||
self.headers["content-length"] = str(content_length)
|
||||
self.body_reads = 0
|
||||
|
||||
def iter_bytes(self):
|
||||
for i in range(0, len(self._body), self._chunk):
|
||||
self.body_reads += 1
|
||||
yield self._body[i:i + self._chunk]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def no_cache(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(content_mod, "CONTENT_CACHE_DIR", tmp_path)
|
||||
monkeypatch.setattr(content_mod, "_cache_result", lambda *a, **k: None)
|
||||
monkeypatch.setattr(content_mod, "_public_http_url", lambda u: True)
|
||||
|
||||
|
||||
def _patch_stream(monkeypatch, fake):
|
||||
@contextmanager
|
||||
def fake_stream(method, url, **kwargs):
|
||||
yield fake
|
||||
monkeypatch.setattr(content_mod.httpx, "stream", fake_stream)
|
||||
return fake
|
||||
|
||||
|
||||
def test_body_under_cap_is_untouched(monkeypatch, no_cache):
|
||||
_patch_stream(monkeypatch, _FakeStream(b"hello world"))
|
||||
r = content_mod.fetch_webpage_content("https://example.com/a.txt")
|
||||
assert r["success"] is True
|
||||
assert r["content"] == "hello world"
|
||||
assert r["truncated"] is False
|
||||
assert r["fetched_bytes"] == len(b"hello world")
|
||||
|
||||
|
||||
def test_body_over_soft_cap_truncates_with_flags(monkeypatch, no_cache):
|
||||
body = b"x" * (WEB_FETCH_SOFT_MAX_BYTES + 50_000)
|
||||
_patch_stream(monkeypatch, _FakeStream(body, content_length=len(body)))
|
||||
r = content_mod.fetch_webpage_content("https://example.com/big.txt")
|
||||
assert r["truncated"] is True
|
||||
assert r["fetched_bytes"] == WEB_FETCH_SOFT_MAX_BYTES
|
||||
assert r["total_bytes"] == len(body)
|
||||
assert len(r["content"]) == WEB_FETCH_SOFT_MAX_BYTES
|
||||
|
||||
|
||||
def test_max_bytes_override_raises_budget(monkeypatch, no_cache):
|
||||
body = b"y" * (WEB_FETCH_SOFT_MAX_BYTES + 50_000)
|
||||
_patch_stream(monkeypatch, _FakeStream(body))
|
||||
r = content_mod.fetch_webpage_content(
|
||||
"https://example.com/big.txt", max_bytes=len(body) + 1
|
||||
)
|
||||
assert r["truncated"] is False
|
||||
assert r["fetched_bytes"] == len(body)
|
||||
|
||||
|
||||
def test_override_is_clamped_to_hard_cap(monkeypatch, no_cache):
|
||||
# Ask for more than the ceiling; the effective budget must be the ceiling.
|
||||
fake = _patch_stream(monkeypatch, _FakeStream(b"z" * 10, chunk=4))
|
||||
r = content_mod.fetch_webpage_content(
|
||||
"https://example.com/a.txt", max_bytes=WEB_FETCH_HARD_MAX_BYTES * 10
|
||||
)
|
||||
assert r["success"] is True
|
||||
# The clamp itself: effective cap recorded in the cache key path is the
|
||||
# hard cap, and a declared body over the ceiling is refused regardless.
|
||||
big = _FakeStream(b"", content_length=WEB_FETCH_HARD_MAX_BYTES + 1)
|
||||
_patch_stream(monkeypatch, big)
|
||||
r = content_mod.fetch_webpage_content(
|
||||
"https://example.com/huge.bin", max_bytes=WEB_FETCH_HARD_MAX_BYTES * 10
|
||||
)
|
||||
assert r["success"] is False
|
||||
assert "TooLarge" in r["error"]
|
||||
assert big.body_reads == 0 # refused before buffering
|
||||
|
||||
|
||||
def test_declared_over_hard_cap_refused_before_buffering(monkeypatch, no_cache):
|
||||
fake = _FakeStream(b"irrelevant", content_length=WEB_FETCH_HARD_MAX_BYTES + 1)
|
||||
_patch_stream(monkeypatch, fake)
|
||||
r = content_mod.fetch_webpage_content("https://example.com/huge.iso")
|
||||
assert r["success"] is False
|
||||
assert "TooLarge" in r["error"]
|
||||
assert fake.body_reads == 0
|
||||
|
||||
|
||||
def test_truncated_pdf_is_an_error_not_garbage(monkeypatch, no_cache):
|
||||
body = b"%PDF-1.4 " + b"p" * (WEB_FETCH_SOFT_MAX_BYTES + 10)
|
||||
_patch_stream(monkeypatch, _FakeStream(body, content_type="application/pdf"))
|
||||
r = content_mod.fetch_webpage_content("https://example.com/big.pdf")
|
||||
assert r["success"] is False
|
||||
assert "TooLarge" in r["error"]
|
||||
|
||||
|
||||
def test_fetch_requests_identity_encoding(monkeypatch, no_cache):
|
||||
# Compressed responses can decode to far more than Content-Length, so the
|
||||
# streamed cap and the hard-cap preflight are only honest when we refuse
|
||||
# transfer compression. Pin that the fetch advertises identity, not gzip.
|
||||
seen = {}
|
||||
|
||||
@contextmanager
|
||||
def fake_stream(method, url, **kwargs):
|
||||
seen["headers"] = kwargs.get("headers") or {}
|
||||
yield _FakeStream(b"hello")
|
||||
monkeypatch.setattr(content_mod.httpx, "stream", fake_stream)
|
||||
|
||||
content_mod.fetch_webpage_content("https://example.com/a.txt")
|
||||
assert seen["headers"].get("Accept-Encoding") == "identity"
|
||||
|
||||
|
||||
def test_rejects_compressed_response_that_ignored_identity(monkeypatch, no_cache):
|
||||
# We request Accept-Encoding: identity, but a server can ignore it and send
|
||||
# gzip anyway. httpx would decode it, so a tiny compressed body could balloon
|
||||
# past the cap in one decoded chunk. Refuse before reading the body.
|
||||
fake = _FakeStream(b"x" * 5000, content_length=40)
|
||||
fake.headers["content-encoding"] = "gzip"
|
||||
_patch_stream(monkeypatch, fake)
|
||||
r = content_mod.fetch_webpage_content("https://example.com/a.txt")
|
||||
assert r["success"] is False
|
||||
assert "Content-Encoding" in r["error"] or "compressed" in r["error"]
|
||||
assert fake.body_reads == 0 # refused before decoding any body
|
||||
|
||||
|
||||
def test_oversized_title_does_not_hide_partial_notice(monkeypatch):
|
||||
# The partial-content notice is the PR's core contract; an untrusted,
|
||||
# oversized page title must not push it past MAX_OUTPUT_CHARS.
|
||||
import asyncio
|
||||
from src.agent_tools.web_tools import WebFetchTool
|
||||
from src.constants import MAX_OUTPUT_CHARS
|
||||
|
||||
def fake_fetch(url, timeout=10, max_bytes=None):
|
||||
return {
|
||||
"content": "partial body",
|
||||
"title": "T" * (MAX_OUTPUT_CHARS + 5_000),
|
||||
"error": "",
|
||||
"truncated": True,
|
||||
"fetched_bytes": WEB_FETCH_SOFT_MAX_BYTES,
|
||||
"total_bytes": 9_000_000,
|
||||
}
|
||||
|
||||
import src.search.content as alias_mod
|
||||
monkeypatch.setattr(alias_mod, "fetch_webpage_content", fake_fetch)
|
||||
|
||||
out = asyncio.run(WebFetchTool().execute(
|
||||
json.dumps({"url": "https://example.com/big.txt"}), ctx={}
|
||||
))
|
||||
assert out["exit_code"] == 0
|
||||
assert out["output"].startswith("[partial content:")
|
||||
assert '"full": true' in out["output"]
|
||||
|
||||
|
||||
def test_tool_layer_emits_partial_notice_and_parses_full(monkeypatch):
|
||||
import asyncio
|
||||
from src.agent_tools.web_tools import WebFetchTool
|
||||
|
||||
calls = {}
|
||||
|
||||
def fake_fetch(url, timeout=10, max_bytes=None):
|
||||
calls["max_bytes"] = max_bytes
|
||||
return {
|
||||
"content": "partial body",
|
||||
"title": "Big File",
|
||||
"error": "",
|
||||
"truncated": True,
|
||||
"fetched_bytes": WEB_FETCH_SOFT_MAX_BYTES,
|
||||
"total_bytes": 5_000_000,
|
||||
}
|
||||
|
||||
import src.search.content as alias_mod
|
||||
monkeypatch.setattr(alias_mod, "fetch_webpage_content", fake_fetch)
|
||||
|
||||
out = asyncio.run(WebFetchTool().execute(
|
||||
json.dumps({"url": "https://example.com/big.txt"}), ctx={}
|
||||
))
|
||||
assert out["exit_code"] == 0
|
||||
assert "[partial content:" in out["output"]
|
||||
assert '"full": true' in out["output"]
|
||||
assert calls["max_bytes"] is None
|
||||
|
||||
asyncio.run(WebFetchTool().execute(
|
||||
json.dumps({"url": "https://example.com/big.txt", "full": True}), ctx={}
|
||||
))
|
||||
assert calls["max_bytes"] == WEB_FETCH_HARD_MAX_BYTES
|
||||
@@ -0,0 +1,18 @@
|
||||
"""The web scraping path routes its User-Agent through one constant.
|
||||
|
||||
Guards the dedup: web_fetch / web_search outbound UAs go through
|
||||
WEB_FETCH_USER_AGENT, so a stale or bare Mozilla string cannot be re-inlined in
|
||||
the search sources.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
_SEARCH = Path(__file__).resolve().parent.parent / "services" / "search"
|
||||
|
||||
|
||||
def test_search_sources_have_no_inline_mozilla_ua():
|
||||
offenders = [
|
||||
str(py.relative_to(_SEARCH.parent.parent))
|
||||
for py in _SEARCH.rglob("*.py")
|
||||
if "Mozilla/" in py.read_text(encoding="utf-8")
|
||||
]
|
||||
assert not offenders, f"inline Mozilla UA found; use WEB_FETCH_USER_AGENT: {offenders}"
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Guard: every public webhook emitter goes through the manager.
|
||||
|
||||
Public emitters in `routes/` must schedule their fire through
|
||||
`webhook_manager.fire_and_forget(...)` (or `_spawn_tracked`). A bare
|
||||
`asyncio.create_task(webhook_manager.fire(...))` escapes
|
||||
`WebhookManager._bg_tasks`, so asyncio only holds a weak reference to the
|
||||
delivery task and the GC can collect it before it sends — silently dropping
|
||||
the webhook. Catching this with a scan stops a regression from sneaking
|
||||
back in via a copy-paste.
|
||||
"""
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
ROUTES_DIR = Path(__file__).resolve().parent.parent / "routes"
|
||||
|
||||
|
||||
def _untracked_fire_calls(tree: ast.AST) -> list[tuple[int, str]]:
|
||||
"""Return (lineno, snippet) for any asyncio.create_task(webhook_manager.fire(...))."""
|
||||
hits: list[tuple[int, str]] = []
|
||||
for node in ast.walk(tree):
|
||||
if not isinstance(node, ast.Call):
|
||||
continue
|
||||
func = node.func
|
||||
if not (isinstance(func, ast.Attribute) and func.attr == "create_task"):
|
||||
continue
|
||||
if not (isinstance(func.value, ast.Name) and func.value.id == "asyncio"):
|
||||
continue
|
||||
if not node.args:
|
||||
continue
|
||||
inner = node.args[0]
|
||||
if not isinstance(inner, ast.Call):
|
||||
continue
|
||||
inner_func = inner.func
|
||||
if (
|
||||
isinstance(inner_func, ast.Attribute)
|
||||
and inner_func.attr == "fire"
|
||||
and isinstance(inner_func.value, ast.Name)
|
||||
and inner_func.value.id == "webhook_manager"
|
||||
):
|
||||
hits.append((node.lineno, ast.unparse(node)))
|
||||
return hits
|
||||
|
||||
|
||||
def test_no_untracked_webhook_fire_in_routes():
|
||||
offenders: list[str] = []
|
||||
for path in ROUTES_DIR.rglob("*.py"):
|
||||
tree = ast.parse(path.read_text(), filename=str(path))
|
||||
for lineno, snippet in _untracked_fire_calls(tree):
|
||||
offenders.append(f"{path.relative_to(ROUTES_DIR.parent)}:{lineno}: {snippet}")
|
||||
assert not offenders, (
|
||||
"Public webhook emitters must use webhook_manager.fire_and_forget(...) "
|
||||
"so the delivery task is tracked in WebhookManager._bg_tasks. Found "
|
||||
"untracked emitter(s):\n " + "\n ".join(offenders)
|
||||
)
|
||||