29 Commits

Author SHA1 Message Date
Alexandre Teixeira 68f19a889a test: add fire_and_forget to API chat webhook stub 2026-06-16 03:24:48 +01:00
RaresKeY 422f23fb12 fix(mcp): scope memory server by owner (#4315) 2026-06-16 03:18:17 +01:00
TheDragonTail 0f966d6b9f fix(embeddings): fall back to default cache dir when FASTEMBED_CACHE_PATH is empty (#3434)
docker-compose.yml injects FASTEMBED_CACHE_PATH=${FASTEMBED_CACHE_PATH:-},
which sets the variable to an empty string when the host has not defined it.
FASTEMBED_CACHE_DIR used os.getenv("FASTEMBED_CACHE_PATH", default), and
os.getenv only returns the default when the variable is ABSENT -- so the empty
value won and FASTEMBED_CACHE_DIR became "". os.makedirs("") then raised
[Errno 2] No such file or directory: '', FastEmbed failed to initialise, and
every vector feature (RAG, semantic memory, tool index) silently degraded on
the default Docker stack.

Treat an empty value like an absent one via `os.getenv(...) or default`.
Add a regression test covering the empty, unset, and explicit cases.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 03:11:48 +01:00
Afonso Coutinho 7b09491557 fix: check-in calendar digest leaks every user's events (missing owner scope) (#1925)
* fix: check-in calendar digest leaks every user's events (no owner scope)

* Seed dtend on calendar events in digest test so the NOT NULL column is satisfied
2026-06-16 02:42:41 +01:00
Kenny Van de Maele fafaf089c5 refactor(search): centralize the web-scraping User-Agent into one constant (#4325)
The outbound UA for web_fetch / web_search was inlined in four places with
two different values and nothing keeping them current: content.py pinned a
mid-2021 Chrome 91 build, and providers.py sent a bare Mozilla/5.0 in three
spots. Some sites serve a degraded or blocked page to a UA that old.

Add WEB_FETCH_USER_AGENT to src/constants.py (env-overridable, matching the
existing Copilot/Kimi UA-constant pattern) and import it in content.py and
providers.py. Default to a current, common desktop UA so pages return their
normal HTML: the market-leading desktop OS (Windows; NT 10.0 covers Windows
10 and 11) and browser (Chrome) on a current stable build. The version is now
bumped in one place.

Service-specific self-identifying agents (Copilot, Kimi, webhooks, cookbook)
are intentionally left separate. Adds a regression pinning the constant shape,
the env override, and a guard against a new inline Mozilla literal in the
search sources.

Closes #4324
2026-06-16 01:33:47 +00:00
RaresKeY b58af4267b fix(companion): require chat scope for model inventory (#4319) 2026-06-16 01:15:05 +02:00
AkioKoneko 8ff76f083c fix(cookbook): avoid launching Ollama during Windows cache scan (#4368) 2026-06-16 01:00:40 +02:00
Wei Hong 2196869c86 fix(webhooks): route public emitters through fire_and_forget (#3964) (#4336)
The three public webhook emitters in chat_helpers and webhook_routes
schedule deliveries via asyncio.create_task(webhook_manager.fire(...)),
which bypasses WebhookManager._bg_tasks. asyncio only holds a weak
reference to the outer task, so the GC can collect it mid-delivery and
the webhook is silently dropped.

Route all three through webhook_manager.fire_and_forget() so the task
is tracked by _spawn_tracked() and the manager owns the full lifecycle.

Adds an AST-level guard test that scans routes/ for direct
asyncio.create_task wrapping webhook_manager.fire(...) to prevent
regressions.
2026-06-16 00:41:45 +02:00
holden093 dd2e23c9af fix(agent): report phone numbers from resolve_contact when a matched contact has no email (#4327)
When a CardDAV contact matched the search query but had no email
address (only phone numbers), the tool silently dropped it and
returned 'No contacts found'.  Fall back to the contact's phone
number(s) so the caller still receives usable information.

Refs: #4178 (the contacts-domain classifier fix that made the model
actually call resolve_contact for contacts queries, surfacing this
pre-existing gap)
2026-06-16 00:03:33 +02:00
Fahim facc50cb0f fix(api): attribute bearer-token actions to the token owner on owner-scoped routes (#4054)
* fix(api): attribute bearer-token actions to the token owner on owner-scoped routes

Owner-scoped chat, session, and upload routes called
get_current_user(), which resolves a bearer ody_ API token to the
sandboxed "api" pseudo-user. A paired API-token client (companion, CLI,
IDE extension) therefore saw and created a separate "api"-owned silo
instead of the owner's data.

effective_user() already exists for exactly this: it attributes a token's
actions to request.state.api_token_owner, is identical to
get_current_user() for cookie sessions, and falls back safely when a
token has no owner. session_routes.py was already migrated; this
completes the migration for the remaining owner-scoped routes:

- chat_helpers.py: chat-privilege enforcement, message attribution, prefs/context
- chat_routes.py: orphaned-endpoint owner, session-auth owner, message search
- upload_routes.py: upload owner attribution + access checks

The /api/models swap is intentionally omitted: #4292 already migrated it
to effective_user (plus the chat-scope gate and ownerless-token 403), so
this PR keeps dev's version of routes/model_routes.py unchanged.

chat_routes.py keeps importing get_current_user for the workspace owner
gate; session_routes.py drops the now-unused import.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: target effective_user in auth monkeypatches and owner-scope assertion

The owner-scoped routes now call effective_user() instead of
get_current_user(), so the tests that stubbed get_current_user (or
asserted on it) follow suit:

- test_chat_helpers.py, test_review_regressions.py,
  test_kv_cache_invalidation_2927.py: monkeypatch effective_user
- test_session_endpoint_owner_scope.py: assert the owner-scope guard uses
  effective_user(request)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:56:22 +02:00
Kenny Van de Maele 074a1e6eff fix(search): add download budgets to web_fetch with truncation notice and hard ceiling (#3955)
* fix(search): add download budgets to web_fetch with truncation notice and hard ceiling

MAX_OUTPUT_CHARS only trims what the agent sees; fetch_webpage_content
buffered and cached the entire response body first, so a large or hostile
URL could pull arbitrarily many bytes into memory and the content cache.

The fetch is now a capped streaming GET (SSRF redirect guard unchanged):
a soft default budget (WEB_FETCH_SOFT_MAX_BYTES, 2 MB), a per-call
override via full/max_bytes on the web_fetch tool, and a hard ceiling
(WEB_FETCH_HARD_MAX_BYTES, 20 MB) that the override can never exceed.
When Content-Length already declares a body over the ceiling the fetch
is refused before any body bytes are buffered. Truncated results carry
truncated/fetched_bytes/total_bytes, the tool output leads with a
partial-content notice telling the model how to re-fetch with full=true,
and the tool schema documents the flag. A truncated PDF is reported as
a budget error since a cut PDF is unparseable. The effective cap is part
of the content-cache key so a truncated fetch is never served to a
full-budget request.

Existing tests that faked httpx.get or the old _get_public_url signature
are adapted to the streaming interface; behavior pins are unchanged.

Fixes #3812

* fix(search): close compressed-body cap bypass and protect the partial notice

Addresses RaresKeY's review on #3955:

- Force Accept-Encoding: identity for the capped fetch. With gzip/deflate the
  wire bytes (and Content-Length) can be a fraction of the decoded body, so a
  tiny compressed response could pass the hard-cap preflight and then expand
  past the ceiling in a single decoded chunk before the streamed cap could
  slice it. Identity makes Content-Length the true body size and keeps each
  streamed chunk bounded by the network read, so the hard ceiling actually
  bounds memory.
- Lead web_fetch output with the partial-content notice and cap the page
  title. The notice is the user-facing contract for partial fetches, but the
  title is untrusted, uncapped page content; placed ahead of the notice a giant
  title could push it past MAX_OUTPUT_CHARS and drop it. The notice now leads
  and the title is capped as a second guard.

Adds regressions: the fetch advertises identity encoding, and a truncated
result with an oversized title still surfaces the partial notice.

* fix(search): reject compressed responses that ignore the identity request

Requesting Accept-Encoding: identity is not enough on its own: a server can
ignore it and still return Content-Encoding: gzip, and httpx.iter_bytes would
decode that, so a tiny compressed body could balloon into one decoded chunk
far past the hard cap before the streamed loop slices it (and Content-Length,
the compressed wire length, makes the preflight and size metadata unreliable).

Refuse a non-identity Content-Encoding before reading the body. Adds a
regression where the server ignores the identity request and returns gzip;
the fetch is refused before any body is decoded.
2026-06-15 17:38:09 +00:00
Kenny Van de Maele 2fab378c6a refactor(search): import REQUEST_TIMEOUT from constants in providers.py (#4331)
providers.py redefined REQUEST_TIMEOUT = 20 locally, shadowing the same
value in src/constants.py and risking drift if the constant is bumped.
Import it from src.constants and drop the local copy; same value, one
source of truth.

Closes #4329
2026-06-15 17:22:08 +00:00
Michael 5bafc30622 fix(api): normalize non-object JSON bodies to empty dict in token PATCH (#3976)
* fix(api): normalize non-object JSON bodies to empty dict in token PATCH

Valid non-dict JSON (e.g. [] or null) reaches payload.get(...) and
raises AttributeError. Normalize to {} so the route returns a controlled
response instead of an unhandled 500.

Fixes #3966

* test(api): add regression tests for PATCH with non-object JSON bodies

Covers array body ([]), null body, and normal object body as requested
in alteixeira20's review of #3976.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
2026-06-15 18:05:15 +01:00
darius-f96 d6d2e17214 fix(hwfit): add GB10 unified-memory bandwidth so speed scores are real (#4270)
NVIDIA Grace Blackwell GB10 / DGX Spark was missing from GPU_BANDWIDTH, so
_lookup_bandwidth() returned None for it and _estimate_speed() fell through
to the crude FALLBACK_K path (k/active-params). That over-stated tok/s and
let speed scores saturate regardless of the box's real ~273 GB/s LPDDR5X
pool — distorting model ranking on these 128GB unified-memory rigs.

Add "gb10": 273 (GB/s). nvidia-smi reports the device name as "NVIDIA GB10",
which substring-matches the new key, so detected GB10 boxes now estimate
speed from the real bandwidth instead of the fallback.
2026-06-15 18:55:15 +02:00
Lucas Daniel f4e8990635 chore: add warnings to silent except Exception blocks (#3212)
* log(app): add warnings to silent except Exception blocks

- Internal tool auth header failure now logs a warning instead of
  silently passing, making auth bypass easier to spot in logs.
- Token last_used_at update failure now logs at DEBUG (fire-and-forget,
  non-critical, but useful when debugging token tracking issues).
- Image ownership verification failure now logs a warning so unexpected
  access-check errors surface instead of silently allowing the request.

* log(chat_routes): add warnings to silent except Exception blocks

- clear_orphaned_session_endpoint: log before rollback so failures
  appear in traces when users see stale/deleted model options.
- _endpoint_has_model (JSON parse): log malformed cached_models instead
  of silently treating endpoint as valid.
- _has_any_visible_model (JSON parse): log malformed cached_models
  instead of silently returning empty list.
- timezone header parse: log failure so time-zone-related tool bugs
  (wrong scheduled times, calendar events) are traceable.
- attachments JSON parse: log failure so silently-dropped attachments
  are visible in server logs.

* log(email_routes): add warnings to silent except Exception blocks

- Email alias resolution failure now logs a warning instead of silently
  returning an empty list, making broken account configs diagnosable.

* log(document_routes): add warnings to silent except Exception blocks

- Export ZIP request body parse failure now logs a warning so empty
  exports caused by malformed requests are diagnosable.
- clear_active_document failure on detach now logs a warning to help
  trace doc re-injection bugs like #1160.

* log(agent_loop): add warnings to silent except Exception blocks

- builtin tool overrides load failure now logs a warning so misconfigured
  settings don't silently fall back to defaults without a trace.
- Timezone context injection failure now logs a warning to help debug
  incorrect scheduled times in agent-created tasks.
- PDF form-backed document detection failure now logs a warning so
  broken form-doc UI is traceable to the root cause.

* log(llm_core): add warnings to silent except Exception blocks

- Malformed URL in _is_ollama_native_url now logs a warning so bad
  endpoint configs are traceable instead of silently returning False.
- Model list fetch failure now logs a warning with the endpoint URL so
  endpoints that silently vanish from the model picker are diagnosable.

* log: pass exception via exc_info instead of string interpolation

* fix(logging): avoid logging raw URLs in llm_core error paths

Drop the raw url/base_chat_url from the Ollama-detection and
model-list-fetch warning logs added by this sweep, since these values
can contain private hostnames, internal IPs, credentials, or other
deployment details.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 17:49:27 +01:00
Kfir Sadeh fc3a5e555e feat(paths): abstract runtime path logic for frozen distribution packages (#969)
* feat(core): abstract runtime path logic for frozen distribution packages

* Address review feedback: revert browser MCP check, persistent data dir default when frozen, and add path tests
2026-06-15 17:44:10 +01:00
Hriday Ranka 270b8570fc feat(email): add Google OAuth2 for Google Workspace / .edu IMAP & SMTP (#237)
* feat(email): add Google OAuth2 for Google Workspace / .edu IMAP & SMTP

Google deprecated basic-auth (password) access for Google Workspace
accounts in May 2025. This means any .edu or org Google email account
could no longer connect via IMAP/SMTP with a username + password —
the email feature was silently broken for a large class of users.

This PR adds full OAuth2 (XOAUTH2) support for Google accounts so
Workspace / .edu emails work out of the box.

## What changed

### Backend
- `core/database.py`: add `oauth_provider`, `oauth_access_token`,
  `oauth_refresh_token`, `oauth_token_expiry`, and `display_name`
  columns to `EmailAccount` + idempotent migration
- `routes/email_helpers.py`: XOAUTH2 auth in `_imap_connect()` and
  `_send_smtp_message()`, automatic token refresh, OAuth fields in
  `_get_email_config()`
- `routes/email_routes.py`: OAuth authorize + callback routes,
  `_smtp_ready()` fix, OAuth fields through `_deliver()` closure,
  `display_name` in `From:` header

### Frontend
- `static/js/settings.js`: "Google Workspace / .edu" provider preset,
  "Connect with Google" button, success/error banner, display name field
- `static/js/document.js`: `_accountCanSend()` recognises OAuth accounts
  as SMTP-capable

* security: sign OAuth state, scope callback by owner, fix quotes & logs

Addresses reviewer feedback on the email OAuth2 PR:

- OAuth state is now HMAC-SHA256 signed (keyed with the app secret from
  secret_storage) encoding account_id + owner + a random nonce, and is
  verified with constant-time comparison in the callback before any
  token write. Replaces the bare account_id state, closing the CSRF /
  state-guessing gap.
- Callback extracts the owner from the verified state and re-checks it
  against EmailAccount.owner before writing tokens, matching the
  ownership guards used elsewhere in the email routes. Single-user mode
  (owner == "") still accepts any account, consistent with
  _assert_owns_account.
- Replaced curly/smart quotes in the Name/Email/Display Name input rows
  with plain ASCII so getElementById lookups and event wiring work.
- Stripped account name, SMTP host/user, owner, and raw provider error
  text from send-config and OAuth logs; failures now surface as generic
  error codes in the redirect instead of raw exception strings.

* test(email): add OAuth2 state, _smtp_ready, and XOAUTH2 tests

Move the OAuth state sign/verify helpers out of the setup_email_routes
closure into module-level make_oauth_state/verify_oauth_state in
email_helpers.py so they can be unit-tested, then add tests/test_email_oauth.py:

- signed state round-trips account_id + owner, nonce is unique per call
- tampered account_id, forged signature, and garbage states are rejected
- _smtp_ready treats an OAuth account (no password) as send-capable, and
  still rejects host+user-only accounts with neither password nor OAuth
- _xoauth2_string / _xoauth2_bytes produce the correct SASL XOAUTH2 framing

14 new tests; existing test_security_regressions.py still passes (28).

* refactor(email): single XOAUTH2 frame helper, use RuntimeError

Polish from self-review before merge:

- Collapse the XOAUTH2 framing to one source of truth: _xoauth2_raw()
  returns the unencoded SASL string used by both the SMTP and IMAP auth
  callbacks (each library base64-encodes it), and _xoauth2_bytes() is
  just its .encode(). Removes the unused base64 _xoauth2_string helper
  and the duplicated inline frame in _send_smtp_message.
- Raise RuntimeError (not bare Exception) for the "OAuth token
  unavailable" path, matching the convention used across src/.
- Update tests accordingly.

All 14 OAuth tests + 28 security regressions pass; SMTP/IMAP XOAUTH2
verified live against a real Workspace account.

* tests(email-oauth): cover the security-sensitive OAuth paths before merge

The previous tests only exercised pure helpers (state signing, _smtp_ready,
XOAUTH2 framing). This adds coverage for the actual token-custody and
ownership behaviour, pinning the real route handlers rather than
re-implementations of their logic.

Real OAuth callback route (pulled live from setup_email_routes()):
- missing code -> generic missing_code redirect, no account id / owner in URL
- provider error -> generic google_error redirect, raw error not echoed
- tampered/invalid state -> invalid_state redirect, auth code never leaked
- signed state with owner mismatch -> token write refused (ownership_error),
  DB row left untouched
- signed state with matching owner -> tokens written encrypted, and only to
  the intended account (a second account stays untouched)

Real accounts-list route:
- exposes oauth_provider status but never the access/refresh token values,
  encrypted or otherwise

Token storage / refresh helpers (isolated in-memory SQLite, mocked HTTP):
- refreshed access token stored encrypted; expiry is a timestamp, not a token
- fresh token uses cache (no refresh call); expired token triggers refresh
- refresh HTTP failure returns None silently, no exception or secret surfaced
- missing client credentials short-circuits to None

Password-account regression:
- password IMAP accounts call conn.login(); OAuth accounts call XOAUTH2
  authenticate() and never login()

28 tests pass (14 prior + 14 new).

* fix(email-oauth): drop raw exception text from token-refresh log

Google token refresh failures now log the account id only, matching
the conservative logging used elsewhere on the OAuth path — no raw
provider/exception details surfacing in logs.

* fix(email-oauth): bring OAuth UI parity to the Integrations email form

The Google Workspace / .edu provider preset, Display Name field, and
Connect-with-Google flow were only wired into the Email-tab account
form. The Integrations-tab form (a separate code path for the same
account type) was missing all three, so the OAuth option was invisible
from that entry point. Mirrors the same PROVIDERS entry, OAuth section,
and connect handler so both forms behave identically.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Alexandre Teixeira <111787685+alteixeira20@users.noreply.github.com>
2026-06-15 17:02:58 +01:00
Léo 0750486654 fix(notes): fail closed when an unauthenticated request reaches owner-scoped routes (#4062)
* fix(notes): fail closed when an unauthenticated request reaches owner-scoped routes

The notes CRUD routes resolved the acting user with bare get_current_user().
A request that reached them with no identity (auth-middleware regression,
SSRF from a sibling service) came through as user=None — which every query
treats as the single-user mode: list all accounts' notes, read/update/
delete/pin/archive any row, reorder globally.

Resolve the owner through require_user() instead, which already encodes the
right policy: 401 when auth is configured, while the documented anonymous
modes (AUTH_ENABLED=false, LOCALHOST_BYPASS on loopback, unconfigured
first-run) still resolve to the single-user path. fire-reminder in the same
file already gated this way; the CRUD routes now match, and the inline
require_user import there is folded into the module import.

Extracted from #2940 (stabilization slice).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(notes): drive fail-closed test via ASGITransport, not sync TestClient

The focused fail-closed test hung at `TestClient(app).get(...)` on some
environments. Starlette's sync TestClient runs the app in a background
event-loop thread (anyio blocking portal) and then dispatches each sync
endpoint onto a second worker thread; that handshake deadlocks on certain
anyio/httpx/platform combos. The identity injection also used
BaseHTTPMiddleware (@app.middleware("http")), the other known TestClient
deadlock source.

Switch to the repo's existing httpx.ASGITransport + AsyncClient idiom so the
whole request runs on the test's own event loop (no portal thread, no
BaseHTTPMiddleware). Identity now comes from a pure-ASGI shim that writes the
same request.state fields the real auth middleware sets, and a non-loopback
client peer keeps require_user's loopback fall-throughs out of the picture.
Same assertions and coverage; production code unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 17:43:28 +02:00
RaresKeY d38e2cbc07 fix(ci): avoid duplicate CodeQL setup (#4297) 2026-06-15 16:39:13 +01:00
Ashvin 7fd937fa57 fix(calendar): parse "mins"/"hrs" reminder offsets in manage_calendar (#4266)
_reminder_minutes matched the offset with (?:m|min|minute|minutes)\b and
(?:h|hr|hour|hours)\b. The trailing \b makes the common plural
abbreviations "mins"/"hrs" fail to match (after "min" the "s" is a word
char, so no boundary), so reminder_minutes "5 mins" or "2 hrs" returned
None and the event was created with no reminder, silently.

Widen the two unit regexes and the matching reminder_only description
regex to a strict superset that also accepts mins/hrs. The sibling
duration parser already accepts these forms (it has no \b), so this only
brings the reminder parser in line.
2026-06-15 17:37:28 +02:00
Catalin Iliescu c41caac438 fix(cookbook): only persist successfully stopped scheduled serves (#4267)
Co-authored-by: Cata <cata@bigjohn.local>
2026-06-15 17:30:18 +02:00
Kenny Van de Maele 1747c13133 test: align README presentation guards with the #4306 refresh (#4311)
* test: align README presentation guards with the #4306 refresh

The 'Refresh README presentation' change (#4306) swapped the ASCII banner
for a centered wordmark image and moved the native quickstart into
docs/setup.md, which left four base tests failing on dev and froze the
merge gate:

- test_security_regressions::test_readme_native_quickstart_uses_loopback
  now also accepts the loopback guidance from docs/setup.md, where the
  quickstart moved (no behaviour change; the guidance is intact there).
- test_readme_ascii_fenced guards the new wordmark title instead of the
  removed ASCII banner, and keeps a defensive check that any reintroduced
  box-drawing banner stays inside a code fence (the original #1390 mode).
- The five unreferenced demo gifs under docs/ (chat, compare, document,
  notes, research) are removed so test_docs_no_orphan_images passes; they
  were de-referenced by the refresh. Recoverable from history if a docs
  page wants to embed them again.

* chore: refresh PR checks

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>
2026-06-15 16:25:38 +01:00
RaresKeY ffd0aaf69b fix(cookbook): validate adopt host (#4282) 2026-06-15 16:44:24 +02:00
RaresKeY 81e7074d93 fix(gallery): confine replacement image path (#4285) 2026-06-15 16:42:41 +02:00
RaresKeY f66a23d19d fix(ai): validate generated image result URLs (#4289) 2026-06-15 16:40:49 +02:00
RaresKeY f602819523 fix(models): scope API-token model listing (#4292) 2026-06-15 16:38:41 +02:00
RaresKeY 85a773ea02 fix(personal): resolve upload delete path (#4291) 2026-06-15 16:38:37 +02:00
PewDiePie fb0a64fe4f Merge pull request #4306 from pewdiepie-archdaemon/readme-refresh-default-branch
docs: refresh README presentation
2026-06-15 23:28:20 +09:00
pewdiepie-archdaemon bcf46dafb9 Refresh README presentation 2026-06-15 23:26:10 +09:00
78 changed files with 3544 additions and 689 deletions
+38 -463
View File
@@ -1,476 +1,65 @@
# Odysseus
<p align="center">
<img src="docs/odysseus-wordmark.png" alt="Odysseus" width="280">
</p>
> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main).
<p align="center">
A self-hosted AI workspace for chat, agents, research, documents, email, notes, calendar, and local model workflows.
</p>
```
───────────────────────────────────────────────
⊹ ࣪ ˖ ૮( ˶ᵔ ᵕ ᵔ˶ )っ Odysseus vers. 1.0
───────────────────────────────────────────────
```
<p align="center">
<a href="#quick-start">Quick Start</a> ·
<a href="docs/setup.md">Setup Guide</a> ·
<a href="CONTRIBUTING.md">Contributing</a> ·
<a href="ROADMAP.md">Roadmap</a>
</p>
![Odysseus](docs/odysseus.jpg)
<p align="center">
<a href="https://repology.org/project/odysseus-ai/versions"><img src="https://repology.org/badge/vertical-allrepos/odysseus-ai.svg" alt="Packaging status"></a>
</p>
A self-hosted AI workspace -- meant to be the self-hosted version of the UI experience you get from ChatGPT and Claude. But with more jank and fun. Running on your own hardware, with your own data -- local-first, privacy-first, and no trojan.
<p align="center">
<img src="docs/odysseus.jpg" alt="Odysseus interface">
</p>
[![Packaging status](https://repology.org/badge/vertical-allrepos/odysseus-ai.svg)](https://repology.org/project/odysseus-ai/versions)
## Features
- **Chat** -- chat with any local model or API; adding them is super simple.<br> <sub>vLLM · llama.cpp · Ollama · OpenRouter · OpenAI · GitHub Copilot</sub>
- **Agent** -- hand it tools and let it run the whole task itself.<br> <sub>built on [opencode](https://github.com/anomalyco/opencode) · MCP · web · files · shell · skills · memory</sub>
- **Cookbook** -- Scans your hardware, recommends models, click to download and serve.. easy!<br> <sub>built on [llmfit](https://github.com/AlexsJones/llmfit) · VRAM-aware · GGUF / FP8 / AWQ · fit scoring · vLLM / llama.cpp serving</sub>
- **Deep Research** -- multi-step runs that gather, read, and synthesize sources into a nice visual report.<br> <sub>adapted from [Tongyi DeepResearch](https://github.com/Alibaba-NLP/DeepResearch)</sub>
- **Compare** -- a fun tool to compare models side by side. Test completely blind, no bias!<br> <sub>multi-model · blind test · synthesis</sub>
- **Documents** -- YOU write the text, AI is there to assist, not the opposite.<br> <sub>multi-tab editor · markdown · HTML · CSV · syntax highlighting · AI edits · suggestions</sub>
- **Memory / Skills** -- Persistent memory and skills, your agent evolves over time as it better understands you and your tasks!<br> <sub>ChromaDB · fastembed (ONNX) · vector + keyword retrieval · import/export</sub>
- **Email** -- IMAP/SMTP inbox with AI triage built in: urgency reminders, auto-tag, auto-summary, auto-reply drafts, auto-spam.<br> <sub>IMAP · SMTP · per-account routing · CalDAV-aware</sub>
- **Notes & Tasks** -- Quick notes with reminders, a todo list, and scheduled tasks the agent can act on.<br> <sub>note pings · checklist · cron-style tasks · ntfy / browser / email channels</sub>
- **Calendar** -- Local-first calendar with CalDAV sync to Radicale / Nextcloud / Apple / Fastmail.<br> <sub>CalDAV pull · .ics import/export · per-calendar colors · agent-aware</sub>
- **Works on mobile** -- looks and runs great on your phone, not just desktop.<br> <sub>responsive · installable (PWA) · touch gestures</sub>
- **Extras** -- more to explore, happy if you give it a go!<br> <sub>image editor · theme editor · file uploads (vision + PDF) · web search · presets · sessions · 2FA</sub>
## Demo
A full, hover-to-play tour lives on the landing page (`docs/index.html`).
<details>
<summary>Screenshots / clips</summary>
### Chat & Agents
![Chat & Agents](docs/chat.gif)
### Deep Research
![Deep Research](docs/research.gif)
### Compare
![Compare](docs/compare.gif)
### Documents
![Documents](docs/document.gif)
### Notes & Tasks
![Notes & Tasks](docs/notes.gif)
</details>
---
## Quick Start
Defaults work out of the box: clone, run, then configure models/search/email
inside **Settings**. Only edit `.env` for deployment-level overrides like
`APP_BIND`, `APP_PORT`, `AUTH_ENABLED`, `DATABASE_URL`, or a pre-seeded admin password.
> `dev` is the default branch and gets the newest changes first. Use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main) if you want the more curated branch.
On first setup, Odysseus creates an admin account (`admin` unless
`ODYSSEUS_ADMIN_USER` is set) and prints a temporary password in the terminal.
For Docker installs, the same line is in `docker compose logs odysseus`.
Use that for the first login, then change it in **Settings**.
Contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, testing, and
pull request guidelines.
### Docker (recommended)
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
cp .env.example .env # optional, but recommended for explicit defaults
cp .env.example .env
docker compose up -d --build
```
To include optional extras in the image (PDF viewer, Office extraction; includes AGPL PyMuPDF), build with `docker compose build --build-arg INSTALL_OPTIONAL=true` before `up`.
Open `http://localhost:7000` when the containers are healthy. Docker Compose
binds the web UI to `127.0.0.1` by default. If the port is taken, set
`APP_PORT=7001` in `.env` and recreate the container. Set `APP_BIND=0.0.0.0`
only when you intentionally want LAN/reverse-proxy access.
Open `http://localhost:7000` when the containers are healthy. The first admin password is printed in `docker compose logs odysseus`.
> **On Apple Silicon (M-series) Macs:** Docker can't reach the Metal GPU, so
> Cookbook serves local models on CPU only. For GPU-accelerated model serving,
> run natively instead — see [Apple Silicon](#apple-silicon) below.
Native installs, GPU notes, Windows/macOS instructions, HTTPS, and configuration live in the [setup guide](docs/setup.md).
### Native Linux / macOS
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
Requirements: Python 3.11+. Cookbook also needs `tmux` for background model
downloads and serves. The app itself is lightweight; local model serving is the
heavy part and depends on the model, runtime, GPU, and VRAM, so small hosts can
connect to API or remote model servers instead. Use `--host 0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
## Features
### Apple Silicon
Docker on macOS cannot use the Metal GPU. For GPU-accelerated Cookbook on an
M-series Mac, run Odysseus natively:
- **Chat + Agents** — local/API models, tools, MCP, files, shell, skills, and memory.
- **Cookbook** — hardware-aware model recommendations, downloads, and serving.
- **Deep Research** — multi-step web research with source reading and report generation.
- **Compare** — blind side-by-side model testing and synthesis.
- **Documents** — writing-first editor with AI edits, suggestions, Markdown, HTML, CSV, and syntax highlighting.
- **Email** — IMAP/SMTP inbox with triage, tags, summaries, reminders, and reply drafts.
- **Notes, Tasks + Calendar** — reminders, todos, scheduled agent tasks, and CalDAV sync.
- **Extras** — gallery/image editor, themes, uploads, web search, presets, sessions, and 2FA.
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
./start-macos.sh
```
## Demo
It launches at `http://127.0.0.1:7860`. To expose it to your phone over a trusted LAN/VPN such as Tailscale, bind all interfaces:
```bash
ODYSSEUS_HOST=0.0.0.0 ./start-macos.sh
# then open http://<tailscale-ip>:7860
```
The script also reads `.env` at startup, so `APP_BIND=0.0.0.0` and `APP_PORT`
set there are picked up automatically without a command-line override each run.
Keep `AUTH_ENABLED=true` (the default) before binding outside loopback. Do not
expose this port directly to the public internet. To build a clickable app wrapper:
```bash
./build-macos-app.sh
```
<details>
<summary>Cookbook, GPU, Ollama, and troubleshooting notes</summary>
**Docker bundled services.** Compose starts Odysseus, ChromaDB, SearXNG, and
ntfy. Odysseus and the bundled service ports bind to `127.0.0.1` by default, so
they are reachable from the host but not exposed to your LAN/public internet
unless you opt in.
**Cookbook storage in Docker.** Downloads live in `./data/huggingface`
(`~/.cache/huggingface` in the container). Cookbook-installed Python CLIs and
serve engines live in `./data/local` (`~/.local` in the container), so they
survive container recreation.
**Remote servers.** In **Cookbook -> Settings -> Servers**, generate the
Odysseus SSH key and add the public key to the remote server's
`~/.ssh/authorized_keys`. From the host you can also run:
```bash
ssh-copy-id -i data/ssh/id_ed25519.pub user@server
```
**Docker GPU overlays.** CPU-only users can skip this section. Cookbook can
only detect GPUs that Docker exposes to the container — if the host runtime or
device passthrough is not configured, Cookbook sees the iGPU, another card, or
CPU instead of your intended GPU.
For NVIDIA, `scripts/check-docker-gpu.sh` diagnoses GPU passthrough and can
optionally install the host runtime or update `.env`.
```bash
# Read-only diagnostic (default — installs nothing, never edits .env):
scripts/check-docker-gpu.sh
# Print OS-specific install commands without running them:
scripts/check-docker-gpu.sh --print-install-commands
# Install NVIDIA Container Toolkit on Ubuntu/Debian (requires sudo):
scripts/check-docker-gpu.sh --install-nvidia-toolkit
# Write COMPOSE_FILE to .env (only when GPU passthrough is confirmed working):
scripts/check-docker-gpu.sh --enable-nvidia-overlay
# Full assisted setup — install toolkit, then enable overlay if passthrough works:
scripts/check-docker-gpu.sh --install-nvidia-toolkit --enable-nvidia-overlay
```
Safety notes:
- The app never installs host GPU runtime automatically.
- The app never edits `.env` automatically.
- `.env` is only modified when `--enable-nvidia-overlay` is explicitly passed,
and only after GPU passthrough succeeds. `--yes` skips prompts but does not
bypass the passthrough gate.
- `.env.bak.*` backups created by `--enable-nvidia-overlay` are ignored by
Git and the Docker build context.
To enable manually without the script, add this to `.env`:
```bash
COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
```
**AMD / ROCm.** AMD setup is read-only diagnostic plus manual `.env` edit. Run:
```bash
scripts/check-docker-amd-gpu.sh
```
Then add the reported values to `.env`, replacing `RENDER_GID` with your host's
numeric render group id:
```bash
COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
RENDER_GID=989
```
For NVIDIA/AMD GPU support, also read the comments in the selected overlay file: docker/gpu.nvidia.yml or docker/gpu.amd.yml.
**Stack-management UIs (Portainer, Coolify, Dockhand, etc.).** These tools
often accept only a single Compose file and do not reliably honor `COMPOSE_FILE`
or multiple `-f` overlays. CLI users should keep using the `COMPOSE_FILE`
overlay workflow above. For stack UIs, point the stack at one of the standalone
files instead, which bundle the base stack plus the GPU settings:
- `docker-compose.gpu-nvidia.yml` — still requires the NVIDIA Container Toolkit
on the host.
- `docker-compose.gpu-amd.yml` — still requires host ROCm/kfd/DRI setup, the
`video`/`render` group membership, and `RENDER_GID` when needed.
The base `docker-compose.yml` plus the `docker/gpu.*.yml` overlays remain the
source of truth; the standalone files mirror them for single-file deployments.
Verify after enabling either overlay:
```bash
docker compose exec odysseus nvidia-smi -L # NVIDIA
docker compose exec odysseus sh -lc 'test -e /dev/kfd && test -d /dev/dri && ls -l /dev/kfd /dev/dri/renderD*' # AMD
```
> **GPU passthrough ≠ llama.cpp CUDA.** `nvidia-smi` passing inside the
> container confirms Docker GPU access, but llama.cpp also needs `cudart` and
> the CUDA Toolkit at runtime. If Cookbook logs show `Unable to find cudart
> library`, `Could NOT find CUDAToolkit`, `CUDA Toolkit not found`, or
> tensors/layers assigned to CPU, that is a Cookbook/llama.cpp build issue —
> not a Docker passthrough failure. Reinstall the serve engine via
> **Cookbook → Dependencies** to get a CUDA-enabled build.
>
> The same split applies to AMD/ROCm: seeing `/dev/kfd` and `/dev/dri` inside
> the container confirms device passthrough, not ROCm userspace or a
> ROCm-enabled vLLM/llama.cpp build. `rocm-smi` and `rocminfo` are not expected
> inside the slim Odysseus image.
**Ollama with Docker.** If Ollama runs on the host, add this endpoint in
Settings:
```text
http://host.docker.internal:11434/v1
```
Ollama must listen outside its own loopback interface:
```bash
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
This connects Odysseus in Docker to an Ollama server that is already running on
your host machine; it does not start Ollama inside the container.
`host.docker.internal` is Docker's hostname for the host machine from inside the
container. Cookbook **Serve** is a separate workflow for serving downloaded
models through Odysseus/llama.cpp, so Windows users with an existing Ollama
install usually only need to add the endpoint in Settings.
**Useful checks.**
```bash
docker compose ps
docker compose logs --tail=120 odysseus
docker compose logs odysseus | grep -E 'ChromaDB|MemoryVectorStore|DEGRADED'
```
**macOS details.** `start-macos.sh` installs Homebrew deps, creates the venv,
runs setup, and starts uvicorn on port `7860` because AirPlay often holds
`7000`. It uses llama.cpp/Ollama for Metal. vLLM/SGLang are CUDA/ROCm-only and
do not run on macOS. MLX-only models are not served by Odysseus.
</details>
### Native Windows
**One-command launcher** (creates the venv, installs deps, runs setup, starts the
server; safe to re-run):
```powershell
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
```
Or do it by hand:
```powershell
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
py -3.11 -m venv venv
venv\Scripts\Activate.ps1
pip install -r requirements.txt
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
If `python` points at an older interpreter, use `py -3.12` (or another installed
3.11+ version) for the venv step.
**Requirements:** Python 3.11+. The core app (chat, agent, memory, documents,
email, calendar, deep research) runs fully native. For full **Cookbook** background
model downloads and the agent shell tool, also install
[Git for Windows](https://git-scm.com/download/win) (provides `bash.exe`).
Local GPU *serving* of vLLM/SGLang needs Linux/WSL2; for a local model on Windows,
[Ollama](https://ollama.com/download) is the easiest path — point Odysseus at
`http://localhost:11434/v1` in Settings.
Open `http://localhost:7000`, log in with the generated admin password,
and configure everything else inside **Settings**.
## Troubleshooting & Advanced Setup
### `chromadb-client` conflicts with embedded ChromaDB
If `chromadb-client` (the lightweight HTTP-only package) is installed alongside the full `chromadb` package, Odysseus starts but ChromaDB silently falls back to HTTP-only mode and fails.
**Fix:** uninstall `chromadb-client` and force-reinstall the full package:
```bash
./venv/bin/pip uninstall chromadb-client -y
./venv/bin/pip install --force-reinstall chromadb
```
### HTTPS + LAN/Tailscale exposure
To expose Odysseus on a local network or Tailscale with HTTPS:
1. Change the bind address to `0.0.0.0` in `.env` (`APP_BIND=0.0.0.0` or `ODYSSEUS_HOST=0.0.0.0`).
2. Generate a locally-trusted cert for your LAN/Tailscale IPs using [mkcert](https://github.com/FiloSottile/mkcert):
```bash
mkcert -install
mkcert -cert-file cert.pem -key-file key.pem 192.168.1.100 tailscale-ip
```
3. Run `uvicorn` with the generated certs:
```bash
python -m uvicorn app:app --host 0.0.0.0 --port 7000 --ssl-certfile=cert.pem --ssl-keyfile=key.pem
```
4. Install the `mkcert` CA on any other device you want to access Odysseus from (e.g., for iOS, email the `rootCA.pem` to yourself, install the profile, and trust it in Certificate Trust Settings).
### Optional Dependencies
`requirements-optional.txt` contains packages that unlock extra features. It is not installed by default.
| Package | Feature unlocked |
|---------|-----------------|
| `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
| `ddgs` | DuckDuckGo as a search provider option. |
| `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
| `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
### Faster, reproducible installs with uv (optional)
[uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
```bash
uv venv venv --python 3.13
uv pip install -r requirements.txt
# then continue as usual: python setup.py, uvicorn, ...
```
`requirements.txt` is intentionally unpinned, so two installs at different times can produce different package versions. If you want a reproducible environment (e.g. across your own machines, or to roll back after a bad upgrade), snapshot and restore exact versions with:
```bash
uv pip compile requirements.txt -o requirements.lock # snapshot current resolution
uv pip sync requirements.lock # reproduce it exactly later
```
`requirements.lock` is gitignored and platform-specific (compile it on the OS you deploy to). Regenerate it deliberately when you want to take upgrades. The plain `uv pip install -r requirements.txt` keeps following the unpinned requirements like pip does.
### Outlook / Office 365 email
Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
current limitation and the planned integration direction.
## Security Notes
Odysseus is a self-hosted workspace with powerful local tools: shell access, file uploads, model downloads, web research, email/calendar integrations, and API tokens. Treat it like an admin console.
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
- Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
- Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
- Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
- Non-admin users do not get shell/Python/file read/write by default, and admin-only routes/tools such as MCP management, API tokens, webhooks, model/cookbook serving, backup/vault, and app settings are admin-gated. Other features are controlled by per-user privileges, so review each user's privileges before exposing a deployment.
- Rotate any API keys or tokens that were ever pasted into a shared chat, demo, screenshot, or log.
- If you enable API tokens or webhooks, create separate tokens per integration and delete unused ones.
- Prefer binding manual development runs to `127.0.0.1`; bind to `0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
- Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
### Private or proxied deployments
Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
1. Keep Odysseus on localhost, for example `127.0.0.1:7000`.
2. Terminate HTTPS at a trusted reverse proxy or private access gateway.
3. Put the authenticated Odysseus web/API entrypoint behind that layer.
4. Keep raw service and model ports internal-only.
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
`ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
Common internal-only ports from the default docs/compose setup:
| Port | Service |
|---|---|
| `7000` | Odysseus raw app port |
| `8080` | SearXNG |
| `8091` | ntfy |
| `8100` | ChromaDB host port for manual/compose access |
| `11434` | Ollama |
| `8000-8020` | Common local model/provider APIs |
A full hover-to-play tour lives on the landing page: [`docs/index.html`](docs/index.html).
## Contributing
Help is welcome. The best entry points are fresh-install testing, provider setup
bugs, mobile/editor polish, docs, and small focused refactors. See
[ROADMAP.md](ROADMAP.md) for the current help-wanted list.
## Configuration
Most setup is done inside the app with `/setup` or **Settings**. Use `.env`
for deployment-level defaults and secrets you want present before first boot.
Key settings:
Help is welcome. The best entry points are fresh-install testing, provider setup bugs, mobile/editor polish, docs, and small focused refactors. See [CONTRIBUTING.md](CONTRIBUTING.md) and [ROADMAP.md](ROADMAP.md).
| Variable | Default | Description |
|---|---|---|
| `LLM_HOST` | `localhost` | Your LLM server (e.g. `llm-host.local:8000`) |
| `LLM_HOSTS` | -- | Comma-separated list for model discovery |
| `OPENAI_API_KEY` | -- | Optional OpenAI key. Prefer adding providers in the app unless pre-seeding. |
| `SEARXNG_INSTANCE` | `http://localhost:8080` | SearXNG URL. Docker overrides this to `http://searxng:8080`. |
| `SEARXNG_SECRET` | generated on first Docker boot | Optional SearXNG cookie/CSRF secret. Leave blank unless you need to pin it. |
| `APP_BIND` | `127.0.0.1` | Docker Compose host bind address for the web UI. Use `0.0.0.0` only for intentional LAN/reverse-proxy access. |
| `APP_PORT` | `7000` | Docker Compose host port for the web UI. |
| `APP_DATA_DIR` | `./data` | Docker Compose host directory for application data volumes. |
| `APP_LOGS_DIR` | `./logs` | Docker Compose host directory for application logs. |
| `AUTH_ENABLED` | `true` | Enable/disable login |
| `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
| `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
| `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
| `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
| `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |
| `EMBEDDING_URL` | -- | OpenAI-compatible embeddings endpoint |
| `ODYSSEUS_CHAT_UPLOAD_MAX_BYTES` | `10485760` | Chat/agent attachment cap in bytes. Raise for larger local PDFs or text documents. |
| `ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES` | `104857600` | Gallery image upload cap in bytes (100 MB). |
| `ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES` | `26214400` | Gallery transform input cap in bytes (25 MB). |
| `ODYSSEUS_MEMORY_IMPORT_MAX_BYTES` | `10485760` | Memory import file cap in bytes (10 MB). |
| `ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES` | `26214400` | Personal document upload cap in bytes (25 MB). |
| `ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES` | `26214400` | Email compose attachment cap in bytes (25 MB). |
| `ODYSSEUS_STT_MAX_AUDIO_BYTES` | `26214400` | Speech-to-text audio cap in bytes (25 MB). |
| `ODYSSEUS_ICS_MAX_BYTES` | `10485760` | Calendar `.ics` import cap in bytes (10 MB). |
## Security
All upload-limit vars are validated (must be a positive integer) and optional; an invalid value fails fast at startup.
### Built-in MCP servers (optional setup)
Odysseus auto-registers a few built-in MCP servers at startup. The npx-based ones (currently the browser server, `@playwright/mcp`) only start when their npm package is already in the local npx cache. If a package isn't cached, that server is skipped with a startup log message explaining what to do, so a fresh install does not block on a multi-minute npm download or hang if Playwright system deps are missing.
To enable the browser MCP (page navigation, screenshots, vision), run once:
```bash
npx -y @playwright/mcp@latest --version
```
That installs `@playwright/mcp` plus Playwright (~300MB total). Restart Odysseus and the server will register at startup.
## Architecture
```
app.py # FastAPI entry point
core/ auth, database, middleware, constants
src/ llm_core, agent_loop, agent_tools, chat_processor, search/
routes/ chat, session, document, memory, model … endpoints
services/ docs, memory, search, hwfit (Cookbook) …
static/ index.html + app.js + style.css + js/ (modular front-end)
docs/ landing page (index.html) + preview clips
```
## Data
All user data lives in `data/` (gitignored): `app.db` (sessions, messages, documents),
`memory.json`, `presets.json`, `uploads/`, `personal_docs/`, `chroma/`, `settings.json`.
To back up or restore everything in `data/`, see the
[Backup & Restore guide](docs/backup-restore.md).
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly. Deployment details are in the [setup guide](docs/setup.md#security-notes).
## Star History
@@ -483,19 +72,5 @@ To back up or restore everything in `data/`, see the
</a>
## License
AGPL-3.0-or-later -- see [LICENSE](LICENSE) and [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md).
```
|
|||
|||||
| | | |||||||
)_) )_) )_) ~|~
)___))___))___)\ |
)____)____)_____)\\|
_____|____|____|_____\\\__
\ /
~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~
~^~ all aboard! ~^~
~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~
```
AGPL-3.0-or-later -- see [LICENSE](LICENSE) and [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md).
+6 -7
View File
@@ -331,8 +331,8 @@ if AUTH_ENABLED:
request.state.current_user = "internal-tool"
request.state.api_token = False
return await call_next(request)
except Exception:
pass
except Exception as _e:
logger.warning("Internal tool auth header check failed", exc_info=_e)
# Allow DIRECT localhost requests (internal service calls from
# heartbeats etc.). Tunnel/proxy-forwarded requests are excluded by
# _is_trusted_loopback so LOCALHOST_BYPASS can't be abused over a
@@ -385,11 +385,10 @@ if AUTH_ENABLED:
_db.close()
try:
await _asyncio.to_thread(_do)
except Exception:
pass
except Exception as _e:
logger.debug("Failed to update token last_used_at", exc_info=_e)
_asyncio.create_task(_touch_last_used(matched_id))
# Keep bearer-token callers out of normal cookie/user
# routes. API-aware routes can read api_token_owner.
request.state.current_user = "api"
request.state.api_token = True
request.state.api_token_id = matched_id
@@ -464,8 +463,8 @@ async def serve_generated_image(filename: str, request: Request):
_db.close()
except HTTPException:
raise
except Exception:
pass
except Exception as _e:
logger.warning("Image ownership verification failed for %r", filename, exc_info=_e)
ext = filename.rsplit('.', 1)[-1].lower()
mime = {
"png": "image/png", "jpg": "image/jpeg", "jpeg": "image/jpeg",
+17 -3
View File
@@ -5,8 +5,9 @@ offers and pair to it, without duplicating any LLM logic.
Auth is enforced globally by AuthMiddleware (app.py), so reaching a handler here
means the caller is authenticated by either a cookie session or a Bearer `ody_`
API token. The read endpoints (ping/info/models) accept either; the pairing
endpoints are admin-cookie only.
API token. Ping/info accept either credential type, models requires a chat-
scoped API token for bearer callers, and the pairing endpoints are admin-cookie
only.
Pairing CSRF posture: minting happens ONLY on POST. The session cookie is
SameSite=Lax (routes/auth_routes.py), which a browser does not send on a
@@ -18,7 +19,7 @@ on a GET would be unsafe (Lax cookies ride top-level GET navigations), so GET
import html
from fastapi import APIRouter, Request
from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import HTMLResponse
from core.middleware import require_admin
@@ -52,6 +53,18 @@ def owner_can_see(row_owner, owner) -> bool:
return row_owner is None or row_owner == owner
def require_models_scope(request: Request) -> None:
"""Require the companion chat scope for bearer-token model inventory."""
if not getattr(request.state, "api_token", False):
return
scopes = getattr(request.state, "api_token_scopes", None) or []
if isinstance(scopes, str):
scopes = [scope.strip() for scope in scopes.split(",")]
scope_set = {str(scope).strip() for scope in scopes if str(scope).strip()}
if _pairing.COMPANION_SCOPE not in scope_set:
raise HTTPException(403, "API token requires chat scope")
def mint_pairing_token(owner: str, invalidate=None) -> tuple[str, str]:
"""Mint a pairing token AND invalidate the auth middleware's in-memory token
cache, so the new token is accepted on the very next request without a server
@@ -103,6 +116,7 @@ def setup_companion_routes() -> APIRouter:
rows -- the same rule as owner_filter. Read-only; never returns api_key
material.
"""
require_models_scope(request)
import json as _json
from core.database import SessionLocal, ModelEndpoint
+49 -2
View File
@@ -2,12 +2,15 @@ import os
import logging
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
from sqlalchemy import event, create_engine, Column, String, Text, Boolean, DateTime, Integer, ForeignKey, JSON, Index, func, text
from sqlalchemy.engine import Engine
from sqlalchemy.types import TypeDecorator
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy.orm import relationship, sessionmaker, backref
from src.runtime_paths import get_app_root
logger = logging.getLogger(__name__)
# Create base class for declarative models
@@ -29,9 +32,26 @@ class TimestampMixin:
def updated_at(cls):
return Column(DateTime, default=utcnow_naive, onupdate=utcnow_naive, nullable=False)
# Get database URL from environment, default to SQLite in DATA_DIR
# Ensure the writable data directory exists before SQLite connects.
from src.constants import DATA_DIR, AUTH_FILE, MEMORY_FILE, USER_PREFS_FILE, SETTINGS_FILE
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{DATA_DIR}/app.db")
Path(DATA_DIR).mkdir(parents=True, exist_ok=True)
def _default_database_url() -> str:
return f"sqlite:///{Path(DATA_DIR) / 'app.db'}"
def _normalize_sqlite_url(url: str) -> str:
if not url.startswith("sqlite:///"):
return url
db_path = url.replace("sqlite:///", "", 1)
if db_path == ":memory:" or os.path.isabs(db_path):
return url
return f"sqlite:///{(Path(get_app_root()) / db_path).resolve().as_posix()}"
# Get database URL from environment, default to SQLite in DATA_DIR
DATABASE_URL = _normalize_sqlite_url(os.getenv("DATABASE_URL", _default_database_url()))
# Create engine
engine = create_engine(
@@ -324,6 +344,13 @@ class EmailAccount(TimestampMixin, Base):
smtp_password = Column(String, default="")
from_address = Column(String, default="")
display_name = Column(String, nullable=True) # "Hriday Ranka" — used in From: header
# OAuth2 (Google / Google Workspace). Tokens stored encrypted via secret_storage.
oauth_provider = Column(String, nullable=True) # "google" or None
oauth_access_token = Column(String, nullable=True) # encrypted
oauth_refresh_token = Column(String, nullable=True) # encrypted
oauth_token_expiry = Column(String, nullable=True) # unix timestamp string
__table_args__ = (
Index('ix_email_accounts_owner_default', 'owner', 'is_default'),
@@ -1427,6 +1454,25 @@ def _migrate_add_task_automation_columns():
except Exception as e:
logging.getLogger(__name__).warning(f"task automation migration: {e}")
def _migrate_add_email_oauth_columns():
"""Add Google OAuth and display_name columns to email_accounts if missing."""
try:
with engine.connect() as conn:
cols = [r[1] for r in conn.execute(text("PRAGMA table_info(email_accounts)"))]
for col, typedef in [
("oauth_provider", "TEXT"),
("oauth_access_token", "TEXT"),
("oauth_refresh_token", "TEXT"),
("oauth_token_expiry", "TEXT"),
("display_name", "TEXT"),
]:
if col not in cols:
conn.execute(text(f"ALTER TABLE email_accounts ADD COLUMN {col} {typedef}"))
conn.commit()
except Exception as e:
logging.getLogger(__name__).warning(f"email oauth columns migration: {e}")
def _migrate_add_oauth_config():
"""Add oauth_config column to mcp_servers table if missing."""
try:
@@ -1771,6 +1817,7 @@ def init_db():
_migrate_add_tidy_verdict()
_migrate_add_doc_source_email_cols()
_migrate_add_oauth_config()
_migrate_add_email_oauth_columns()
_migrate_add_task_automation_columns()
_migrate_add_disabled_tools()
_migrate_add_mcp_oauth_tokens_column()
BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.0 MiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.4 MiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 MiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1003 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 52 KiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.5 MiB

+14 -9
View File
@@ -1,14 +1,16 @@
# Security CI guide
This project runs a set of automated security checks on every pull request and
on every push to `main`. This page explains what each one does, whether it can
This project runs a set of automated security checks on pull requests and
selected branch pushes. This page explains what each one does, whether it can
block a merge, and the few one-time settings you should turn on to get the full
benefit.
## What runs, and why
Each check lives in its own file under `.github/workflows/`. They run
automatically; you do not start them.
Most checks live in files under `.github/workflows/`. CodeQL is configured
through GitHub's code scanning default setup, so it appears as a dynamic GitHub
workflow instead of a checked-in workflow file. They run automatically; you do
not start them.
| Check | What it protects against | Blocks a merge? |
|---|---|---|
@@ -88,11 +90,14 @@ let the workflows run on one pull request first, then add them here.
2. Turn on **Dependency graph** (usually on by default for public repos) -- this
powers Dependency review and Dependabot.
3. Turn on **Dependabot alerts** and **Dependabot security updates**.
4. Under **Code scanning**, you have two ways to scan the app code with CodeQL:
- The included `codeql.yml` workflow already scans `main` and runs weekly.
- To also scan **pull requests** (recommended, since most contributions come
from forks), click **Set up -> Default** under Code scanning. GitHub then
runs CodeQL on pull requests for you, with no token limitations.
4. Under **Code scanning**, use **Set up -> Default** for CodeQL. GitHub then
runs CodeQL as a dynamic workflow without the fork-token limitations that
affect checked-in advanced workflows.
Do not also add a checked-in CodeQL workflow while default setup is enabled:
GitHub rejects advanced CodeQL uploads when default setup is active. If the
project later needs an advanced CodeQL workflow, disable default setup first
and keep only one CodeQL publishing path active.
## Keeping it current
+425
View File
@@ -0,0 +1,425 @@
# Odysseus Setup Guide
This page keeps the detailed install, deployment, troubleshooting, and configuration notes out of the front README.
## Quick Start
> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main).
Defaults work out of the box: clone, run, then configure models/search/email
inside **Settings**. Only edit `.env` for deployment-level overrides like
`APP_BIND`, `APP_PORT`, `AUTH_ENABLED`, `DATABASE_URL`, or a pre-seeded admin password.
On first setup, Odysseus creates an admin account (`admin` unless
`ODYSSEUS_ADMIN_USER` is set) and prints a temporary password in the terminal.
For Docker installs, the same line is in `docker compose logs odysseus`.
Use that for the first login, then change it in **Settings**.
Contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, testing, and
pull request guidelines.
### Docker (recommended)
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
cp .env.example .env # optional, but recommended for explicit defaults
docker compose up -d --build
```
To include optional extras in the image (PDF viewer, Office extraction; includes AGPL PyMuPDF), build with `docker compose build --build-arg INSTALL_OPTIONAL=true` before `up`.
Open `http://localhost:7000` when the containers are healthy. Docker Compose
binds the web UI to `127.0.0.1` by default. If the port is taken, set
`APP_PORT=7001` in `.env` and recreate the container. Set `APP_BIND=0.0.0.0`
only when you intentionally want LAN/reverse-proxy access.
> **On Apple Silicon (M-series) Macs:** Docker can't reach the Metal GPU, so
> Cookbook serves local models on CPU only. For GPU-accelerated model serving,
> run natively instead — see [Apple Silicon](#apple-silicon) below.
### Native Linux / macOS
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
Requirements: Python 3.11+. Cookbook also needs `tmux` for background model
downloads and serves. The app itself is lightweight; local model serving is the
heavy part and depends on the model, runtime, GPU, and VRAM, so small hosts can
connect to API or remote model servers instead. Use `--host 0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
### Apple Silicon
Docker on macOS cannot use the Metal GPU. For GPU-accelerated Cookbook on an
M-series Mac, run Odysseus natively:
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
./start-macos.sh
```
It launches at `http://127.0.0.1:7860`. To expose it to your phone over a trusted LAN/VPN such as Tailscale, bind all interfaces:
```bash
ODYSSEUS_HOST=0.0.0.0 ./start-macos.sh
# then open http://<tailscale-ip>:7860
```
The script also reads `.env` at startup, so `APP_BIND=0.0.0.0` and `APP_PORT`
set there are picked up automatically without a command-line override each run.
Keep `AUTH_ENABLED=true` (the default) before binding outside loopback. Do not
expose this port directly to the public internet. To build a clickable app wrapper:
```bash
./build-macos-app.sh
```
<details>
<summary>Cookbook, GPU, Ollama, and troubleshooting notes</summary>
**Docker bundled services.** Compose starts Odysseus, ChromaDB, SearXNG, and
ntfy. Odysseus and the bundled service ports bind to `127.0.0.1` by default, so
they are reachable from the host but not exposed to your LAN/public internet
unless you opt in.
**Cookbook storage in Docker.** Downloads live in `./data/huggingface`
(`~/.cache/huggingface` in the container). Cookbook-installed Python CLIs and
serve engines live in `./data/local` (`~/.local` in the container), so they
survive container recreation.
**Remote servers.** In **Cookbook -> Settings -> Servers**, generate the
Odysseus SSH key and add the public key to the remote server's
`~/.ssh/authorized_keys`. From the host you can also run:
```bash
ssh-copy-id -i data/ssh/id_ed25519.pub user@server
```
**Docker GPU overlays.** CPU-only users can skip this section. Cookbook can
only detect GPUs that Docker exposes to the container — if the host runtime or
device passthrough is not configured, Cookbook sees the iGPU, another card, or
CPU instead of your intended GPU.
For NVIDIA, `scripts/check-docker-gpu.sh` diagnoses GPU passthrough and can
optionally install the host runtime or update `.env`.
```bash
# Read-only diagnostic (default — installs nothing, never edits .env):
scripts/check-docker-gpu.sh
# Print OS-specific install commands without running them:
scripts/check-docker-gpu.sh --print-install-commands
# Install NVIDIA Container Toolkit on Ubuntu/Debian (requires sudo):
scripts/check-docker-gpu.sh --install-nvidia-toolkit
# Write COMPOSE_FILE to .env (only when GPU passthrough is confirmed working):
scripts/check-docker-gpu.sh --enable-nvidia-overlay
# Full assisted setup — install toolkit, then enable overlay if passthrough works:
scripts/check-docker-gpu.sh --install-nvidia-toolkit --enable-nvidia-overlay
```
Safety notes:
- The app never installs host GPU runtime automatically.
- The app never edits `.env` automatically.
- `.env` is only modified when `--enable-nvidia-overlay` is explicitly passed,
and only after GPU passthrough succeeds. `--yes` skips prompts but does not
bypass the passthrough gate.
- `.env.bak.*` backups created by `--enable-nvidia-overlay` are ignored by
Git and the Docker build context.
To enable manually without the script, add this to `.env`:
```bash
COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
```
**AMD / ROCm.** AMD setup is read-only diagnostic plus manual `.env` edit. Run:
```bash
scripts/check-docker-amd-gpu.sh
```
Then add the reported values to `.env`, replacing `RENDER_GID` with your host's
numeric render group id:
```bash
COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
RENDER_GID=989
```
For NVIDIA/AMD GPU support, also read the comments in the selected overlay file: docker/gpu.nvidia.yml or docker/gpu.amd.yml.
**Stack-management UIs (Portainer, Coolify, Dockhand, etc.).** These tools
often accept only a single Compose file and do not reliably honor `COMPOSE_FILE`
or multiple `-f` overlays. CLI users should keep using the `COMPOSE_FILE`
overlay workflow above. For stack UIs, point the stack at one of the standalone
files instead, which bundle the base stack plus the GPU settings:
- `docker-compose.gpu-nvidia.yml` — still requires the NVIDIA Container Toolkit
on the host.
- `docker-compose.gpu-amd.yml` — still requires host ROCm/kfd/DRI setup, the
`video`/`render` group membership, and `RENDER_GID` when needed.
The base `docker-compose.yml` plus the `docker/gpu.*.yml` overlays remain the
source of truth; the standalone files mirror them for single-file deployments.
Verify after enabling either overlay:
```bash
docker compose exec odysseus nvidia-smi -L # NVIDIA
docker compose exec odysseus sh -lc 'test -e /dev/kfd && test -d /dev/dri && ls -l /dev/kfd /dev/dri/renderD*' # AMD
```
> **GPU passthrough ≠ llama.cpp CUDA.** `nvidia-smi` passing inside the
> container confirms Docker GPU access, but llama.cpp also needs `cudart` and
> the CUDA Toolkit at runtime. If Cookbook logs show `Unable to find cudart
> library`, `Could NOT find CUDAToolkit`, `CUDA Toolkit not found`, or
> tensors/layers assigned to CPU, that is a Cookbook/llama.cpp build issue —
> not a Docker passthrough failure. Reinstall the serve engine via
> **Cookbook → Dependencies** to get a CUDA-enabled build.
>
> The same split applies to AMD/ROCm: seeing `/dev/kfd` and `/dev/dri` inside
> the container confirms device passthrough, not ROCm userspace or a
> ROCm-enabled vLLM/llama.cpp build. `rocm-smi` and `rocminfo` are not expected
> inside the slim Odysseus image.
**Ollama with Docker.** If Ollama runs on the host, add this endpoint in
Settings:
```text
http://host.docker.internal:11434/v1
```
Ollama must listen outside its own loopback interface:
```bash
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
This connects Odysseus in Docker to an Ollama server that is already running on
your host machine; it does not start Ollama inside the container.
`host.docker.internal` is Docker's hostname for the host machine from inside the
container. Cookbook **Serve** is a separate workflow for serving downloaded
models through Odysseus/llama.cpp, so Windows users with an existing Ollama
install usually only need to add the endpoint in Settings.
**Useful checks.**
```bash
docker compose ps
docker compose logs --tail=120 odysseus
docker compose logs odysseus | grep -E 'ChromaDB|MemoryVectorStore|DEGRADED'
```
**macOS details.** `start-macos.sh` installs Homebrew deps, creates the venv,
runs setup, and starts uvicorn on port `7860` because AirPlay often holds
`7000`. It uses llama.cpp/Ollama for Metal. vLLM/SGLang are CUDA/ROCm-only and
do not run on macOS. MLX-only models are not served by Odysseus.
</details>
### Native Windows
**One-command launcher** (creates the venv, installs deps, runs setup, starts the
server; safe to re-run):
```powershell
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
```
Or do it by hand:
```powershell
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
py -3.11 -m venv venv
venv\Scripts\Activate.ps1
pip install -r requirements.txt
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
If `python` points at an older interpreter, use `py -3.12` (or another installed
3.11+ version) for the venv step.
**Requirements:** Python 3.11+. The core app (chat, agent, memory, documents,
email, calendar, deep research) runs fully native. For full **Cookbook** background
model downloads and the agent shell tool, also install
[Git for Windows](https://git-scm.com/download/win) (provides `bash.exe`).
Local GPU *serving* of vLLM/SGLang needs Linux/WSL2; for a local model on Windows,
[Ollama](https://ollama.com/download) is the easiest path — point Odysseus at
`http://localhost:11434/v1` in Settings.
Open `http://localhost:7000`, log in with the generated admin password,
and configure everything else inside **Settings**.
## Troubleshooting & Advanced Setup
### `chromadb-client` conflicts with embedded ChromaDB
If `chromadb-client` (the lightweight HTTP-only package) is installed alongside the full `chromadb` package, Odysseus starts but ChromaDB silently falls back to HTTP-only mode and fails.
**Fix:** uninstall `chromadb-client` and force-reinstall the full package:
```bash
./venv/bin/pip uninstall chromadb-client -y
./venv/bin/pip install --force-reinstall chromadb
```
### HTTPS + LAN/Tailscale exposure
To expose Odysseus on a local network or Tailscale with HTTPS:
1. Change the bind address to `0.0.0.0` in `.env` (`APP_BIND=0.0.0.0` or `ODYSSEUS_HOST=0.0.0.0`).
2. Generate a locally-trusted cert for your LAN/Tailscale IPs using [mkcert](https://github.com/FiloSottile/mkcert):
```bash
mkcert -install
mkcert -cert-file cert.pem -key-file key.pem 192.168.1.100 tailscale-ip
```
3. Run `uvicorn` with the generated certs:
```bash
python -m uvicorn app:app --host 0.0.0.0 --port 7000 --ssl-certfile=cert.pem --ssl-keyfile=key.pem
```
4. Install the `mkcert` CA on any other device you want to access Odysseus from (e.g., for iOS, email the `rootCA.pem` to yourself, install the profile, and trust it in Certificate Trust Settings).
### Optional Dependencies
`requirements-optional.txt` contains packages that unlock extra features. It is not installed by default.
| Package | Feature unlocked |
|---------|-----------------|
| `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
| `ddgs` | DuckDuckGo as a search provider option. |
| `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
| `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
### Faster, reproducible installs with uv (optional)
[uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
```bash
uv venv venv --python 3.13
uv pip install -r requirements.txt
# then continue as usual: python setup.py, uvicorn, ...
```
`requirements.txt` is intentionally unpinned, so two installs at different times can produce different package versions. If you want a reproducible environment (e.g. across your own machines, or to roll back after a bad upgrade), snapshot and restore exact versions with:
```bash
uv pip compile requirements.txt -o requirements.lock # snapshot current resolution
uv pip sync requirements.lock # reproduce it exactly later
```
`requirements.lock` is gitignored and platform-specific (compile it on the OS you deploy to). Regenerate it deliberately when you want to take upgrades. The plain `uv pip install -r requirements.txt` keeps following the unpinned requirements like pip does.
### Outlook / Office 365 email
Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
current limitation and the planned integration direction.
## Security Notes
Odysseus is a self-hosted workspace with powerful local tools: shell access, file uploads, model downloads, web research, email/calendar integrations, and API tokens. Treat it like an admin console.
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
- Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
- Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
- Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
- Non-admin users do not get shell/Python/file read/write by default, and admin-only routes/tools such as MCP management, API tokens, webhooks, model/cookbook serving, backup/vault, and app settings are admin-gated. Other features are controlled by per-user privileges, so review each user's privileges before exposing a deployment.
- Rotate any API keys or tokens that were ever pasted into a shared chat, demo, screenshot, or log.
- If you enable API tokens or webhooks, create separate tokens per integration and delete unused ones.
- Prefer binding manual development runs to `127.0.0.1`; bind to `0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
- Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
### Private or proxied deployments
Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
1. Keep Odysseus on localhost, for example `127.0.0.1:7000`.
2. Terminate HTTPS at a trusted reverse proxy or private access gateway.
3. Put the authenticated Odysseus web/API entrypoint behind that layer.
4. Keep raw service and model ports internal-only.
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
`ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
Common internal-only ports from the default docs/compose setup:
| Port | Service |
|---|---|
| `7000` | Odysseus raw app port |
| `8080` | SearXNG |
| `8091` | ntfy |
| `8100` | ChromaDB host port for manual/compose access |
| `11434` | Ollama |
| `8000-8020` | Common local model/provider APIs |
## Configuration
Most setup is done inside the app with `/setup` or **Settings**. Use `.env`
for deployment-level defaults and secrets you want present before first boot.
Key settings:
| Variable | Default | Description |
|---|---|---|
| `LLM_HOST` | `localhost` | Your LLM server (e.g. `llm-host.local:8000`) |
| `LLM_HOSTS` | -- | Comma-separated list for model discovery |
| `OPENAI_API_KEY` | -- | Optional OpenAI key. Prefer adding providers in the app unless pre-seeding. |
| `SEARXNG_INSTANCE` | `http://localhost:8080` | SearXNG URL. Docker overrides this to `http://searxng:8080`. |
| `SEARXNG_SECRET` | generated on first Docker boot | Optional SearXNG cookie/CSRF secret. Leave blank unless you need to pin it. |
| `APP_BIND` | `127.0.0.1` | Docker Compose host bind address for the web UI. Use `0.0.0.0` only for intentional LAN/reverse-proxy access. |
| `APP_PORT` | `7000` | Docker Compose host port for the web UI. |
| `APP_DATA_DIR` | `./data` | Docker Compose host directory for application data volumes. |
| `APP_LOGS_DIR` | `./logs` | Docker Compose host directory for application logs. |
| `AUTH_ENABLED` | `true` | Enable/disable login |
| `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
| `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
| `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
| `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
| `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |
| `EMBEDDING_URL` | -- | OpenAI-compatible embeddings endpoint |
| `ODYSSEUS_CHAT_UPLOAD_MAX_BYTES` | `10485760` | Chat/agent attachment cap in bytes. Raise for larger local PDFs or text documents. |
| `ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES` | `104857600` | Gallery image upload cap in bytes (100 MB). |
| `ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES` | `26214400` | Gallery transform input cap in bytes (25 MB). |
| `ODYSSEUS_MEMORY_IMPORT_MAX_BYTES` | `10485760` | Memory import file cap in bytes (10 MB). |
| `ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES` | `26214400` | Personal document upload cap in bytes (25 MB). |
| `ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES` | `26214400` | Email compose attachment cap in bytes (25 MB). |
| `ODYSSEUS_STT_MAX_AUDIO_BYTES` | `26214400` | Speech-to-text audio cap in bytes (25 MB). |
| `ODYSSEUS_ICS_MAX_BYTES` | `10485760` | Calendar `.ics` import cap in bytes (10 MB). |
All upload-limit vars are validated (must be a positive integer) and optional; an invalid value fails fast at startup.
### Built-in MCP servers (optional setup)
Odysseus auto-registers a few built-in MCP servers at startup. The npx-based ones (currently the browser server, `@playwright/mcp`) only start when their npm package is already in the local npx cache. If a package isn't cached, that server is skipped with a startup log message explaining what to do, so a fresh install does not block on a multi-minute npm download or hang if Playwright system deps are missing.
To enable the browser MCP (page navigation, screenshots, vision), run once:
```bash
npx -y @playwright/mcp@latest --version
```
That installs `@playwright/mcp` plus Playwright (~300MB total). Restart Odysseus and the server will register at startup.
## Architecture
```
app.py # FastAPI entry point
core/ auth, database, middleware, constants
src/ llm_core, agent_loop, agent_tools, chat_processor, search/
routes/ chat, session, document, memory, model … endpoints
services/ docs, memory, search, hwfit (Cookbook) …
static/ index.html + app.js + style.css + js/ (modular front-end)
docs/ landing page (index.html) + preview clips
```
## Data
All user data lives in `data/` (gitignored): `app.db` (sessions, messages, documents),
`memory.json`, `presets.json`, `uploads/`, `personal_docs/`, `chroma/`, `settings.json`.
To back up or restore everything in `data/`, see the
[Backup & Restore guide](docs/backup-restore.md).
+90 -29
View File
@@ -6,6 +6,7 @@ Imports MemoryManager and MemoryVectorStore from the Odysseus codebase.
"""
import asyncio
import os
import sys
import time
from pathlib import Path
@@ -23,6 +24,55 @@ _memory_manager = None
_memory_vector = None
_initialized = False
_OWNER_ENV_KEYS = ("ODYSSEUS_MCP_MEMORY_OWNER", "ODYSSEUS_MEMORY_OWNER")
_OWNER_SCOPE_ERROR = (
"Error: Memory MCP owner is not configured for an owner-scoped memory store. "
"Set ODYSSEUS_MCP_MEMORY_OWNER for this server or use the owner-aware native memory tool."
)
def _configured_owner() -> str | None:
for key in _OWNER_ENV_KEYS:
owner = os.environ.get(key, "").strip()
if owner:
return owner
return None
def _entry_owner(entry: dict) -> str | None:
owner = entry.get("owner")
if owner is None:
return None
owner_text = str(owner).strip()
return owner_text or None
def _owner_scoped_store(entries: list[dict]) -> bool:
return any(_entry_owner(entry) for entry in entries if isinstance(entry, dict))
def _scope_entries() -> tuple[str | None, list[dict], list[dict], str | None]:
"""Return configured owner, all entries, visible entries, and optional error."""
entries = _memory_manager.load_all()
owner = _configured_owner()
if owner is None and _owner_scoped_store(entries):
return None, entries, [], _OWNER_SCOPE_ERROR
if owner is None:
visible = [
entry for entry in entries
if isinstance(entry, dict) and _entry_owner(entry) is None
]
else:
visible = [
entry for entry in entries
if isinstance(entry, dict) and _entry_owner(entry) == owner
]
return owner, entries, visible, None
def _text_result(text: str) -> list[TextContent]:
return [TextContent(type="text", text=text)]
def _ensure_init():
"""Lazy-init memory managers on first use."""
@@ -75,24 +125,26 @@ async def list_tools() -> list[Tool]:
@server.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if name != "manage_memory":
return [TextContent(type="text", text=f"Unknown tool: {name}")]
return _text_result(f"Unknown tool: {name}")
_ensure_init()
if not _memory_manager:
return [TextContent(type="text", text="Error: Memory manager not available")]
return _text_result("Error: Memory manager not available")
action = arguments.get("action", "")
if action == "list":
category_filter = arguments.get("category", "")
memories = _memory_manager.load()
_owner, _all_memories, memories, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
if category_filter:
memories = [m for m in memories if m.get("category", "").lower() == category_filter.lower()]
if not memories:
msg = "No memories found"
if category_filter:
msg += f" in category '{category_filter}'"
return [TextContent(type="text", text=msg + ".")]
return _text_result(msg + ".")
lines = [f"Found {len(memories)} memory entries:\n"]
for m in memories:
@@ -102,15 +154,17 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if len(text) > 150:
text = text[:150] + "..."
lines.append(f"- [{cat}] `{mid}` — {text}")
return [TextContent(type="text", text="\n".join(lines))]
return _text_result("\n".join(lines))
elif action == "add":
text = arguments.get("text", "")
category = arguments.get("category", "fact")
if not text:
return [TextContent(type="text", text="Error: Memory text cannot be empty")]
entry = _memory_manager.add_entry(text, source="ai_agent", category=category)
memories = _memory_manager.load_all()
return _text_result("Error: Memory text cannot be empty")
owner, memories, _visible, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
entry = _memory_manager.add_entry(text, source="ai_agent", category=category, owner=owner)
memories.append(entry)
_memory_manager.save(memories)
if _memory_vector and _memory_vector.healthy:
@@ -118,25 +172,28 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
_memory_vector.add(entry["id"], text)
except Exception:
pass
return [TextContent(type="text", text=f"Memory added: [{category}] {text} (id: {entry['id'][:8]})")]
return _text_result(f"Memory added: [{category}] {text} (id: {entry['id'][:8]})")
elif action == "edit":
memory_id = arguments.get("memory_id", "")
new_text = arguments.get("text", "")
if not memory_id or not new_text:
return [TextContent(type="text", text="Error: edit needs memory_id and text")]
memories = _memory_manager.load_all()
found = False
return _text_result("Error: edit needs memory_id and text")
_owner, memories, visible, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
full_id = None
for m in memories:
for m in visible:
if m.get("id", "").startswith(memory_id):
m["text"] = new_text
m["timestamp"] = int(time.time())
found = True
full_id = m["id"]
break
if not found:
return [TextContent(type="text", text=f"Error: Memory '{memory_id}' not found")]
if not full_id:
return _text_result(f"Error: Memory '{memory_id}' not found")
for m in memories:
if m.get("id") == full_id:
m["text"] = new_text
m["timestamp"] = int(time.time())
break
_memory_manager.save(memories)
if _memory_vector and _memory_vector.healthy and full_id:
try:
@@ -144,24 +201,26 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
_memory_vector.add(full_id, new_text)
except Exception:
pass
return [TextContent(type="text", text=f"Memory updated: {new_text}")]
return _text_result(f"Memory updated: {new_text}")
elif action == "delete":
memory_id = arguments.get("memory_id", "")
if not memory_id:
return [TextContent(type="text", text="Error: delete needs memory_id")]
memories = _memory_manager.load_all()
return _text_result("Error: delete needs memory_id")
_owner, memories, visible, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
full_id = None
deleted_text = ""
deleted_category = ""
for m in memories:
for m in visible:
if m.get("id", "").startswith(memory_id):
full_id = m["id"]
deleted_text = m.get("text", "")
deleted_category = m.get("category", "")
break
if not full_id:
return [TextContent(type="text", text=f"Error: Memory '{memory_id}' not found")]
return _text_result(f"Error: Memory '{memory_id}' not found")
memories = [m for m in memories if m.get("id") != full_id]
_memory_manager.save(memories)
if _memory_vector and _memory_vector.healthy and full_id:
@@ -171,30 +230,32 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
pass
cat = f"[{deleted_category}] " if deleted_category else ""
snippet = deleted_text if len(deleted_text) <= 120 else deleted_text[:117] + "..."
return [TextContent(type="text", text=f"Memory deleted: {cat}{snippet} (id: {memory_id})")]
return _text_result(f"Memory deleted: {cat}{snippet} (id: {memory_id})")
elif action == "search":
query = arguments.get("text", "")
if not query:
return [TextContent(type="text", text="Error: search needs text (query)")]
memories = _memory_manager.load()
return _text_result("Error: search needs text (query)")
_owner, _all_memories, memories, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
if hasattr(_memory_manager, 'get_relevant_memories'):
results = _memory_manager.get_relevant_memories(query, memories, threshold=0.05, max_items=20)
else:
query_lower = query.lower()
results = [m for m in memories if query_lower in m.get("text", "").lower()][:20]
if not results:
return [TextContent(type="text", text=f"No memories found matching '{query}'.")]
return _text_result(f"No memories found matching '{query}'.")
lines = [f"Found {len(results)} matching memories:\n"]
for m in results:
cat = m.get("category", "fact")
mid = m.get("id", "?")[:8]
text = m.get("text", "")
lines.append(f"- [{cat}] `{mid}` — {text}")
return [TextContent(type="text", text="\n".join(lines))]
return _text_result("\n".join(lines))
else:
return [TextContent(type="text", text=f"Error: Unknown action '{action}'. Use: list, add, edit, delete, search")]
return _text_result(f"Error: Unknown action '{action}'. Use: list, add, edit, delete, search")
async def run():
+2
View File
@@ -160,6 +160,8 @@ def setup_api_token_routes() -> APIRouter:
payload = await request.json()
except Exception:
payload = {}
if not isinstance(payload, dict):
payload = {}
with get_db_session() as db:
token = db.query(ApiToken).filter(ApiToken.id == token_id).first()
if not token:
+8 -8
View File
@@ -14,7 +14,7 @@ from core.database import Session as DBSession, ModelEndpoint
from src.llm_core import normalize_model_id
from src.endpoint_resolver import normalize_base
from src.context_compactor import maybe_compact, trim_for_context
from src.auth_helpers import get_current_user
from src.auth_helpers import effective_user
from src.prompt_security import untrusted_context_message
from routes.prefs_routes import _load_for_user as load_prefs_for_user
@@ -78,7 +78,7 @@ def _enforce_chat_privileges(request, sess) -> None:
which means unrestricted allowed_models / zero cap -> no-op for them.
"""
try:
user = get_current_user(request)
user = effective_user(request)
except Exception:
user = None
if not user:
@@ -346,11 +346,11 @@ def add_user_message(sess, chat_handler, preprocessed: PreprocessedMessage, inco
def fire_message_event(request, webhook_manager, session_id: str, sess, message: str, compare_mode: bool = False):
"""Fire webhook and event_bus events for a new user message."""
if webhook_manager and not compare_mode:
asyncio.create_task(webhook_manager.fire("chat.message", {
webhook_manager.fire_and_forget("chat.message", {
"session_id": session_id, "model": sess.model, "message": message[:2000],
}))
})
from src.event_bus import fire_event
user = get_current_user(request)
user = effective_user(request)
fire_event("message_sent", user)
@@ -577,7 +577,7 @@ async def build_chat_context(
fire_message_event(request, webhook_manager, session_id, sess, message, compare_mode)
# Resolve user prefs
user = get_current_user(request)
user = effective_user(request)
uprefs = load_prefs_for_user(user)
# Memory enabled?
@@ -1120,10 +1120,10 @@ def run_post_response_tasks(
# Webhook
if webhook_manager and not compare_mode:
asyncio.create_task(webhook_manager.fire("chat.completed", {
webhook_manager.fire_and_forget("chat.completed", {
"session_id": session_id, "model": sess.model,
"user_message": message, "response": full_response[:2000],
}))
})
# Auto-name
if needs_auto_name(sess.name):
+13 -10
View File
@@ -23,7 +23,7 @@ from src.endpoint_resolver import normalize_base as _normalize_base, build_chat_
from src.session_search import search_session_messages
from src.prompt_security import untrusted_context_message
from core.exceptions import SessionNotFoundError
from src.auth_helpers import get_current_user
from src.auth_helpers import effective_user, get_current_user
from routes.session_routes import _verify_session_owner
from routes.document_helpers import _owner_session_filter
from core.database import SessionLocal, get_session_mode, set_session_mode
@@ -126,7 +126,8 @@ def _clear_orphaned_session_endpoint(sess, owner: str | None = None) -> bool:
sess.model = ""
sess.headers = {}
return True
except Exception:
except Exception as e:
logger.warning("Failed to clear orphaned session endpoint", exc_info=e)
db.rollback()
return False
finally:
@@ -144,7 +145,8 @@ def _endpoint_cache_contains_model(endpoint, model: str) -> bool:
return True
try:
models = json.loads(raw) if isinstance(raw, str) else raw
except Exception:
except Exception as e:
logger.warning("Failed to parse cached models list, treating as containing model", exc_info=e)
return True
if not isinstance(models, list) or not models:
return True
@@ -236,7 +238,8 @@ def _recover_empty_session_model(sess, session_id: str, owner: str | None = None
is_chatgpt_subscription = False
try:
cached = json.loads(ep.cached_models) if isinstance(ep.cached_models, str) else (ep.cached_models or [])
except Exception:
except Exception as e:
logger.warning("Failed to parse cached_models for endpoint %r", getattr(ep, "id", "?"), exc_info=e)
cached = []
if not cached:
visible = []
@@ -360,7 +363,7 @@ def setup_chat_routes(
sess = session_manager.get_session(session)
except KeyError:
raise HTTPException(404, f"Session '{session}' not found")
owner = get_current_user(request)
owner = effective_user(request)
if _clear_orphaned_session_endpoint(sess, owner=owner):
raise HTTPException(400, "Selected model endpoint was removed. Pick another model in Settings.")
@@ -600,7 +603,7 @@ def setup_chat_routes(
# but BEFORE loading. Prevents cross-user session hijack.
_verify_session_owner(request, session)
sess = session_manager.get_session(session)
owner = get_current_user(request)
owner = effective_user(request)
if _clear_orphaned_session_endpoint(sess, owner=owner):
raise HTTPException(400, "Selected model endpoint was removed. Pick another model in Settings.")
# Issue #587: picker shows a model from the endpoint cache but
@@ -631,7 +634,7 @@ def setup_chat_routes(
_enforce_chat_privileges(request, sess)
# Ensure session has auth headers
resolve_session_auth(sess, session, owner=get_current_user(request))
resolve_session_auth(sess, session, owner=effective_user(request))
# Check for research_pending BEFORE mode persist overwrites it
do_research = str(use_research).lower() == "true"
@@ -646,8 +649,8 @@ def setup_chat_routes(
elif attachments:
try:
att_ids = [str(x) for x in json.loads(attachments)]
except Exception:
pass
except Exception as e:
logger.warning("Failed to parse attachments JSON, ignoring attachments", exc_info=e)
no_memory = str(form_data.get("no_memory", "")).lower() == "true"
pre_context_tool_policy = build_effective_tool_policy(
@@ -1482,7 +1485,7 @@ def setup_chat_routes(
if not q or not q.strip():
return []
_user = get_current_user(request)
_user = effective_user(request)
return [
result.to_dict()
for result in search_session_messages(
+1 -1
View File
@@ -790,7 +790,7 @@ def setup_codex_routes(
norm = dict(body or {})
sess = (norm.get("tmux_session") or norm.get("session_id") or "").strip()
model = (norm.get("model") or norm.get("repo_id") or "").strip()
host = (norm.get("host") or norm.get("remote_host") or "").strip()
host = validate_remote_host((norm.get("host") or norm.get("remote_host") or "").strip() or None) or ""
port = norm.get("port") or 8000
import re as _re
if not sess or not _re.fullmatch(r"[a-zA-Z0-9_-]+", sess):
+3 -1
View File
@@ -505,6 +505,8 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
" if u.startswith('KB'): return int(n * 1024)",
" return int(n)",
"def scan_ollama():",
" if any(m.get('is_ollama') for m in models): return",
" if os.name == 'nt' and not os.environ.get('ODYSSEUS_ALLOW_OLLAMA_CLI_SCAN'): return",
" if not shutil.which('ollama'): return",
" try:",
" p = subprocess.run(['ollama', 'list'], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, timeout=6)",
@@ -535,8 +537,8 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
" models.append({'repo_id':name,'size_bytes':size_bytes,'nb_files':1,'has_incomplete':False,'path':'ollama','backend':'ollama','is_ollama':True})",
" return",
"for _hf_cache in hf_cache_paths(): scan_hf(_hf_cache)",
"scan_ollama()",
"scan_ollama_api()",
"scan_ollama()",
]
for model_dir in model_dirs or []:
lines.append(f"scan_dir(os.path.expanduser({model_dir!r}))")
+4 -3
View File
@@ -503,7 +503,8 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
user = get_current_user(request)
try:
data = await request.json()
except Exception:
except Exception as e:
logger.warning("Failed to parse export request body, defaulting to empty", exc_info=e)
data = {}
ids = data.get("ids") or []
if not ids:
@@ -645,8 +646,8 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
try:
from src.agent_tools.document_tools import clear_active_document
clear_active_document(doc_id)
except Exception:
pass
except Exception as e:
logger.warning("Failed to clear active document %r on detach", doc_id, exc_info=e)
db.commit()
db.refresh(doc)
return _doc_to_dict(doc)
+134 -10
View File
@@ -13,6 +13,8 @@ and `email_pollers.py` (the background loops):
"""
import os
import base64
import time
import imaplib
import smtplib
import email as email_mod
@@ -38,6 +40,106 @@ from src.secret_storage import decrypt as _decrypt
logger = logging.getLogger(__name__)
def _xoauth2_raw(user: str, access_token: str) -> str:
"""The SASL XOAUTH2 initial-response string (unencoded).
Both smtplib.SMTP.auth() and imaplib.IMAP4.authenticate() base64-encode
the value their callback returns, so callers pass this raw form never
pre-encoded to avoid double base64.
"""
return f"user={user}\x01auth=Bearer {access_token}\x01\x01"
def _xoauth2_bytes(user: str, access_token: str) -> bytes:
"""Raw XOAUTH2 bytes for imaplib's authenticate() callback."""
return _xoauth2_raw(user, access_token).encode()
def make_oauth_state(account_id: str, owner: str) -> str:
"""Return an HMAC-signed, base64-encoded OAuth state token.
Encodes account_id + owner + a random nonce, signed with the app secret
so the callback can validate that the flow was initiated by an
authenticated, owning user (CSRF / state-forgery protection).
"""
import hmac as _hmac, hashlib as _hl, secrets as _sec
from src.secret_storage import _load_or_create_key
nonce = _sec.token_hex(16)
payload = json.dumps({"a": account_id, "o": owner, "n": nonce}, separators=(",", ":"))
sig = _hmac.new(_load_or_create_key(), payload.encode(), _hl.sha256).hexdigest()
return base64.urlsafe_b64encode(f"{payload}|{sig}".encode()).decode()
def verify_oauth_state(state: str) -> dict | None:
"""Verify an OAuth state token's HMAC signature.
Returns the decoded payload dict ({"a", "o", "n"}) on success, or None if
the token is malformed, tampered, or signed with a different key.
"""
import hmac as _hmac, hashlib as _hl
from src.secret_storage import _load_or_create_key
try:
decoded = base64.urlsafe_b64decode(state.encode()).decode()
payload, sig = decoded.rsplit("|", 1)
expected = _hmac.new(_load_or_create_key(), payload.encode(), _hl.sha256).hexdigest()
if not _hmac.compare_digest(sig, expected):
return None
return json.loads(payload)
except Exception:
return None
def _refresh_google_token(account_id: str) -> str | None:
"""Exchange the stored refresh token for a new access token and persist it."""
import httpx
from core.database import SessionLocal as _SL, EmailAccount as _EA
from src.secret_storage import encrypt as _enc, decrypt as _dec
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
client_secret = os.environ.get("GOOGLE_OAUTH_CLIENT_SECRET", "")
if not client_id or not client_secret:
return None
db = _SL()
try:
row = db.get(_EA, account_id)
if not row or not row.oauth_refresh_token:
return None
refresh_token = _dec(row.oauth_refresh_token or "")
if not refresh_token:
return None
resp = httpx.post("https://oauth2.googleapis.com/token", data={
"client_id": client_id,
"client_secret": client_secret,
"refresh_token": refresh_token,
"grant_type": "refresh_token",
}, timeout=10)
resp.raise_for_status()
data = resp.json()
access_token = data["access_token"]
row.oauth_access_token = _enc(access_token)
row.oauth_token_expiry = str(int(time.time()) + data.get("expires_in", 3600))
db.commit()
return access_token
except Exception:
logger.warning(f"Google token refresh failed for account {account_id}")
return None
finally:
db.close()
def _get_valid_google_token(account_id: str, cfg: dict) -> str | None:
"""Return a valid Google access token, refreshing if expired or missing."""
from src.secret_storage import decrypt as _dec
access_token = _dec(cfg.get("oauth_access_token") or "")
expiry_str = cfg.get("oauth_token_expiry") or ""
if access_token and expiry_str:
try:
if int(expiry_str) - 60 > time.time():
return access_token
except (ValueError, TypeError):
pass
return _refresh_google_token(account_id)
def _smtp_security_mode(cfg: dict) -> str:
raw = str(cfg.get("smtp_security") or "").strip().lower()
if raw in {"ssl", "starttls", "none"}:
@@ -54,20 +156,29 @@ def _send_smtp_message(cfg: dict, from_addr: str, recipients: list[str], message
port = int(cfg.get("smtp_port") or 465)
user = cfg.get("smtp_user") or ""
password = cfg.get("smtp_password") or ""
def _auth_smtp(smtp):
if cfg.get("oauth_provider") == "google":
token = _get_valid_google_token(cfg.get("account_id"), cfg)
if not token:
raise RuntimeError("Google OAuth token unavailable — reconnect the account")
smtp.ehlo()
smtp.auth("XOAUTH2", lambda challenge=None: _xoauth2_raw(user, token), initial_response_ok=True)
elif user and password:
smtp.login(user, password)
security = _smtp_security_mode(cfg)
if security == "ssl":
with smtplib.SMTP_SSL(host, port, timeout=timeout) as smtp:
if user and password:
smtp.login(user, password)
_auth_smtp(smtp)
smtp.sendmail(from_addr, recipients, message)
return
with smtplib.SMTP(host, port, timeout=timeout) as smtp:
if security == "starttls":
smtp.starttls()
if user and password:
smtp.login(user, password)
_auth_smtp(smtp)
smtp.sendmail(from_addr, recipients, message)
@@ -701,10 +812,16 @@ def _get_email_config(account_id: str | None = None, owner: str = "") -> dict:
"imap_password": _decrypt(row.imap_password or ""),
"imap_starttls": bool(row.imap_starttls),
"from_address": row.from_address or row.imap_user or "",
"oauth_provider": row.oauth_provider or "",
"oauth_access_token": row.oauth_access_token or "",
"oauth_refresh_token": row.oauth_refresh_token or "",
"oauth_token_expiry": row.oauth_token_expiry or "",
"display_name": row.display_name or "",
}
if not (cfg["smtp_host"] and cfg["smtp_user"] and cfg["smtp_password"]):
is_oauth = bool(cfg.get("oauth_provider"))
if not is_oauth and not (cfg["smtp_host"] and cfg["smtp_user"] and cfg["smtp_password"]):
logger.warning(f"SMTP not configured for account {row.name!r}")
if not (cfg["imap_host"] and cfg["imap_user"] and cfg["imap_password"]):
if not is_oauth and not (cfg["imap_host"] and cfg["imap_user"] and cfg["imap_password"]):
logger.warning(f"IMAP not configured for account {row.name!r}")
return cfg
finally:
@@ -825,12 +942,19 @@ def _imap_connect(account_id: str | None = None, owner: str = "",
timeout=timeout,
)
try:
conn.login(cfg["imap_user"], cfg["imap_password"])
if cfg.get("oauth_provider") == "google":
token = _get_valid_google_token(cfg.get("account_id"), cfg)
if not token:
raise RuntimeError("Google OAuth token unavailable — reconnect the account in Settings → Integrations")
conn.authenticate("XOAUTH2", lambda x: _xoauth2_bytes(cfg["imap_user"], token))
else:
conn.login(cfg["imap_user"], cfg["imap_password"])
except Exception:
# A failed AUTHENTICATE (e.g. an Office 365 app password on an
# MFA-enabled tenant, #3174) otherwise orphans the already-connected
# socket; close it before propagating so a misconfigured account
# can't leak one descriptor per retry / background poller pass.
# MFA-enabled tenant, #3174, or an expired/revoked OAuth token)
# otherwise orphans the already-connected socket; close it before
# propagating so a misconfigured account can't leak one descriptor
# per retry / background poller pass.
try:
conn.shutdown()
except Exception:
+146 -8
View File
@@ -13,7 +13,9 @@ handlers need. The split is mechanical — no behavior change.
"""
import asyncio
import os
import sqlite3 as _sql3
import time
import email as email_mod
import email.header
import email.utils
@@ -43,6 +45,7 @@ from routes.email_helpers import (
_load_settings, _save_settings, _get_email_config,
_send_smtp_message, _smtp_security_mode,
_IMAP_TIMEOUT_SECONDS, _open_imap_connection,
make_oauth_state, verify_oauth_state,
_imap_connect, _imap, _decode_header, _detect_sent_folder, _detect_drafts_folder,
_extract_attachment_text, _list_attachments_from_msg,
_extract_attachment_to_disk, _extract_html, _extract_text,
@@ -76,15 +79,16 @@ def _email_tag_owner_aliases(account_id: str | None, owner: str = "") -> list[st
cfg.get("smtp_user") or "",
cfg.get("from_address") or "",
])
except Exception:
except Exception as _e:
logger.warning("Failed to resolve email account alias", exc_info=_e)
resolved_account_id = None
row = db.get(_EA, resolved_account_id) if resolved_account_id else None
if row:
aliases.extend([row.owner or "", row.imap_user or "", row.from_address or ""])
finally:
db.close()
except Exception:
pass
except Exception as _e:
logger.warning("Failed to load email aliases", exc_info=_e)
out = []
for a in aliases:
a = (a or "").strip()
@@ -285,7 +289,9 @@ def _group_uid_fetch_records(msg_data) -> list:
def _smtp_ready(cfg: dict) -> bool:
return bool(cfg.get("smtp_host") and cfg.get("smtp_user") and cfg.get("smtp_password"))
if not cfg.get("smtp_host") or not cfg.get("smtp_user"):
return False
return bool(cfg.get("smtp_password") or cfg.get("oauth_provider"))
def _resolve_send_config(account_id: str | None = None, owner: str = "") -> dict:
@@ -2021,7 +2027,7 @@ def setup_email_routes():
outer = MIMEMultipart("alternative")
body_container = outer
outer["From"] = cfg["from_address"]
outer["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
outer["To"] = to
if cc:
outer["Cc"] = cc
@@ -2285,6 +2291,7 @@ def setup_email_routes():
try:
cfg = _resolve_send_config(req.account_id, owner=owner)
except Exception as e:
logger.warning(f"No SMTP-capable account resolved: {e}")
return {"success": False, "error": str(e) or "No SMTP-capable email account configured"}
# Use 'mixed' if we have attachments, 'alternative' otherwise
@@ -2297,7 +2304,7 @@ def setup_email_routes():
outer = MIMEMultipart("alternative")
body_container = outer
outer["From"] = cfg["from_address"]
outer["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
outer["To"] = req.to
if req.cc:
outer["Cc"] = req.cc
@@ -2348,6 +2355,10 @@ def setup_email_routes():
_account_id = cfg.get("account_id") or req.account_id # capture for the IMAP append in the closure
_in_reply_to = (req.in_reply_to or "").strip()
_oauth_provider = cfg.get("oauth_provider") or ""
_oauth_access_token = cfg.get("oauth_access_token") or ""
_oauth_refresh_token = cfg.get("oauth_refresh_token") or ""
_oauth_token_expiry = cfg.get("oauth_token_expiry") or ""
def _deliver():
try:
@@ -2358,6 +2369,11 @@ def setup_email_routes():
"smtp_security": _smtp_security,
"smtp_user": _smtp_user,
"smtp_password": _smtp_pw,
"account_id": _account_id,
"oauth_provider": _oauth_provider,
"oauth_access_token": _oauth_access_token,
"oauth_refresh_token": _oauth_refresh_token,
"oauth_token_expiry": _oauth_token_expiry,
},
_from,
_recipients,
@@ -2470,7 +2486,7 @@ def setup_email_routes():
msg.attach(MIMEText(_draft_html, "html", "utf-8"))
else:
msg = MIMEText(req.body, "plain", "utf-8")
msg["From"] = cfg["from_address"]
msg["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
msg["To"] = req.to
if req.cc:
msg["Cc"] = req.cc
@@ -3122,6 +3138,8 @@ def setup_email_routes():
"from_address": r.from_address or "",
"has_imap_password": bool(r.imap_password),
"has_smtp_password": bool(r.smtp_password),
"oauth_provider": r.oauth_provider or "",
"display_name": r.display_name or "",
})
return {"accounts": out}
finally:
@@ -3154,6 +3172,7 @@ def setup_email_routes():
smtp_user=(data.get("smtp_user") or "").strip(),
smtp_password=_enc(data.get("smtp_password") or ""),
from_address=(data.get("from_address") or "").strip(),
display_name=(data.get("display_name") or "").strip(),
# SECURITY: stamp the creator so all subsequent reads / mutations
# can filter by user. Without this every new account leaks to
# every other user.
@@ -3188,7 +3207,7 @@ def setup_email_routes():
if not row:
return {"ok": False, "error": "Account not found"}
# Simple fields
for key in ("name", "imap_host", "imap_user", "smtp_host", "smtp_user", "from_address"):
for key in ("name", "imap_host", "imap_user", "smtp_host", "smtp_user", "from_address", "display_name"):
if key in data:
setattr(row, key, (data[key] or "").strip())
for key in ("imap_port", "smtp_port"):
@@ -3377,4 +3396,123 @@ def setup_email_routes():
finally:
db.close()
# ── Google OAuth2 routes ──
@router.get("/oauth/google/authorize")
async def google_oauth_authorize(account_id: str = Query(...), request: Request = None, owner: str = Depends(require_user)):
import urllib.parse
_assert_owns_account(account_id, owner)
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
if not client_id:
raise HTTPException(400, "GOOGLE_OAUTH_CLIENT_ID not set — add it to .env")
redirect_uri = (
os.environ.get("GOOGLE_OAUTH_REDIRECT_URI")
or f"http://{request.headers.get('host', 'localhost:7000')}/api/email/oauth/google/callback"
)
state = make_oauth_state(account_id, owner)
params = urllib.parse.urlencode({
"client_id": client_id,
"redirect_uri": redirect_uri,
"response_type": "code",
"scope": "https://mail.google.com/ email",
"access_type": "offline",
"prompt": "consent",
"state": state,
})
from fastapi.responses import RedirectResponse as _RR
return _RR(f"https://accounts.google.com/o/oauth2/v2/auth?{params}")
@router.get("/oauth/google/callback")
async def google_oauth_callback(
code: str = Query(None),
state: str = Query(None),
error: str = Query(None),
request: Request = None,
):
import urllib.parse
from fastapi.responses import RedirectResponse as _RR
if error:
return _RR("/?section=integrations&email_oauth_error=google_error")
if not code or not state:
return _RR("/?section=integrations&email_oauth_error=missing_code")
state_data = verify_oauth_state(state)
if not state_data:
return _RR("/?section=integrations&email_oauth_error=invalid_state")
account_id = state_data.get("a", "")
owner = state_data.get("o", "")
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
client_secret = os.environ.get("GOOGLE_OAUTH_CLIENT_SECRET", "")
redirect_uri = (
os.environ.get("GOOGLE_OAUTH_REDIRECT_URI")
or f"http://{request.headers.get('host', 'localhost:7000')}/api/email/oauth/google/callback"
)
import httpx as _httpx
try:
resp = _httpx.post("https://oauth2.googleapis.com/token", data={
"code": code,
"client_id": client_id,
"client_secret": client_secret,
"redirect_uri": redirect_uri,
"grant_type": "authorization_code",
}, timeout=10)
resp.raise_for_status()
data = resp.json()
except Exception:
logger.warning("Google token exchange failed")
return _RR("/?section=integrations&email_oauth_error=token_exchange_failed")
access_token = data.get("access_token", "")
refresh_token = data.get("refresh_token", "")
expiry = str(int(time.time()) + data.get("expires_in", 3600))
# Fetch the email address from userinfo so we can auto-fill imap_user.
email_addr = ""
display_name = ""
try:
ui = _httpx.get("https://www.googleapis.com/oauth2/v1/userinfo",
headers={"Authorization": f"Bearer {access_token}"}, timeout=10)
if ui.is_success:
ui_data = ui.json()
email_addr = ui_data.get("email", "")
display_name = ui_data.get("name", "")
except Exception:
pass
from core.database import SessionLocal, EmailAccount
from src.secret_storage import encrypt as _enc
db = SessionLocal()
try:
row = db.query(EmailAccount).filter(EmailAccount.id == account_id).first()
if not row:
return _RR("/?section=integrations&email_oauth_error=account_not_found")
# SECURITY: verify the account belongs to the initiating user.
if owner and row.owner and row.owner != owner:
logger.warning("OAuth callback owner mismatch — rejecting token write")
return _RR("/?section=integrations&email_oauth_error=ownership_error")
row.oauth_provider = "google"
row.oauth_access_token = _enc(access_token)
if refresh_token:
row.oauth_refresh_token = _enc(refresh_token)
row.oauth_token_expiry = expiry
# Auto-fill Google IMAP/SMTP settings if not already configured.
if not row.imap_host:
row.imap_host = "imap.gmail.com"
row.imap_port = 993
row.imap_starttls = False
if not row.smtp_host:
row.smtp_host = "smtp.gmail.com"
row.smtp_port = 587
if email_addr:
if not row.imap_user:
row.imap_user = email_addr
if not row.smtp_user:
row.smtp_user = email_addr
if not row.from_address:
row.from_address = email_addr
if not row.name or row.name == row.id:
row.name = email_addr
if display_name and not row.display_name:
row.display_name = display_name
db.commit()
finally:
db.close()
return _RR("/?section=integrations&email_oauth_success=1")
return router
+1
View File
@@ -9,6 +9,7 @@ from pathlib import Path
from fastapi import APIRouter, HTTPException, Form, Depends
from core.constants import EMBEDDING_ENDPOINT_FILE, FASTEMBED_CACHE_DIR
from core.middleware import require_admin
from src.runtime_paths import get_app_root
logger = logging.getLogger(__name__)
+2 -5
View File
@@ -232,8 +232,6 @@ def setup_gallery_routes() -> APIRouter:
@router.post("/api/gallery/{image_id}/replace")
async def gallery_replace(request: Request, image_id: str):
"""Replace an existing gallery image file with a new one."""
from pathlib import Path
user = get_current_user(request)
db = SessionLocal()
try:
@@ -249,9 +247,8 @@ def setup_gallery_routes() -> APIRouter:
raise HTTPException(400, "No image provided")
content = await read_upload_limited(file, GALLERY_UPLOAD_MAX_BYTES, "Gallery replacement")
img_dir = Path(GENERATED_IMAGES_DIR)
img_dir.mkdir(parents=True, exist_ok=True)
img_path = img_dir / _sanitize_gallery_filename(img.filename)
GALLERY_IMAGE_DIR.mkdir(parents=True, exist_ok=True)
img_path = _gallery_image_path(img.filename)
img_path.write_bytes(content)
# Refresh dimensions in case the editor resized the canvas.
+11 -8
View File
@@ -26,7 +26,7 @@ from src.endpoint_resolver import (
build_models_url,
build_headers,
)
from src.auth_helpers import _auth_disabled, owner_filter
from src.auth_helpers import _auth_disabled, effective_user, owner_filter
logger = logging.getLogger(__name__)
@@ -1255,13 +1255,16 @@ def setup_model_routes(model_discovery):
# Require auth; "" is the unconfigured single-user mode, treated as
# "see everything" by _fetch_models.
try:
from src.auth_helpers import get_current_user as _gcu
owner = _gcu(request) or ""
except Exception:
owner = ""
# Reject anonymous in configured deployments — no leaking the model
# list to unauthenticated callers.
try:
if getattr(request.state, "api_token", False):
scopes = set(getattr(request.state, "api_token_scopes", []) or [])
if "chat" not in scopes:
raise HTTPException(403, "API token is not scoped for chat")
if not getattr(request.state, "api_token_owner", None):
raise HTTPException(403, "API token has no owner")
owner = effective_user(request) or ""
# Reject anonymous in configured deployments — no leaking the model
# list to unauthenticated callers.
auth_mgr = getattr(request.app.state, "auth_manager", None)
if not owner and not _auth_disabled() and auth_mgr is not None and getattr(auth_mgr, "is_configured", False):
raise HTTPException(401, "Not authenticated")
+12 -4
View File
@@ -10,7 +10,7 @@ from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel
from core.database import SessionLocal, Note
from src.auth_helpers import get_current_user
from src.auth_helpers import require_user
from src.constants import DATA_DIR
from sqlalchemy.orm.attributes import flag_modified
@@ -570,7 +570,16 @@ def setup_note_routes(task_scheduler=None):
router = APIRouter(prefix="/api/notes", tags=["notes"])
def _owner(request: Request) -> Optional[str]:
return get_current_user(request)
# require_user, not bare get_current_user: a request that reaches
# these owner-scoped routes with NO identity (auth-middleware
# regression, SSRF from a sibling service) must fail closed (401)
# when auth is configured — not be treated as the single-user mode
# and handed blanket access to every account's notes. The documented
# anonymous modes (AUTH_ENABLED=false, LOCALHOST_BYPASS on loopback,
# unconfigured first-run) still resolve to None, the single-user
# path. fire_reminder below already gated this way; the CRUD routes
# did not.
return require_user(request) or None
def _is_admin_or_single_user(request: Request, user: str | None) -> bool:
if user == "internal-tool":
@@ -805,8 +814,7 @@ def setup_note_routes(task_scheduler=None):
Returns {synthesis, email_sent}.
"""
# Gate against anonymous callers — LLM synthesis can burn tokens.
from src.auth_helpers import require_user as _ru
user = _ru(request)
user = require_user(request)
body = await request.json()
note_id = str(body.get("note_id") or "").strip()
if not note_id:
+2 -2
View File
@@ -278,8 +278,8 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
# Delete file from disk if it's in uploads dir
deleted_from_disk = False
try:
abs_target = os.path.abspath(filepath)
base_abs = os.path.abspath(UPLOADS_DIR)
abs_target = os.path.realpath(filepath)
base_abs = os.path.realpath(UPLOADS_DIR)
in_uploads = (
abs_target == base_abs
or os.path.commonpath([abs_target, base_abs]) == base_abs
+3 -3
View File
@@ -11,7 +11,7 @@ from core.session_manager import SessionManager
from core.models import ChatMessage
from src.request_models import SessionResponse
from core.database import Session as DbSession, SessionLocal, Document, GalleryImage, utcnow_naive
from src.auth_helpers import get_current_user, effective_user, _auth_disabled, owner_filter
from src.auth_helpers import effective_user, _auth_disabled, owner_filter
from src.session_actions import is_session_recently_active
@@ -328,7 +328,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
endpoint_id: str = Form(""),
):
skip_val = str(skip_validation).lower() == "true"
user = get_current_user(request)
user = effective_user(request)
endpoint_api_key = ""
endpoint_base_url = ""
_reject_raw_endpoint_url_for_non_admin(request, user, endpoint_id, endpoint_url)
@@ -477,7 +477,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
db.close()
# Switch model/endpoint mid-session
if model is not None and endpoint_url is not None:
user = get_current_user(request)
user = effective_user(request)
_reject_raw_endpoint_url_for_non_admin(request, user, endpoint_id, endpoint_url)
endpoint_api_key = ""
endpoint_base_url = ""
+5 -5
View File
@@ -7,7 +7,7 @@ from fastapi import APIRouter, Request, File, UploadFile, HTTPException
from typing import List
import logging
from core.middleware import require_admin
from src.auth_helpers import get_current_user
from src.auth_helpers import effective_user
from src.upload_handler import count_recent_uploads
logger = logging.getLogger(__name__)
@@ -78,7 +78,7 @@ def setup_upload_routes(upload_handler):
for u in files:
try:
meta = upload_handler.save_upload(u, client_ip, owner=get_current_user(request))
meta = upload_handler.save_upload(u, client_ip, owner=effective_user(request))
out.append({
"id": meta["id"],
"name": meta["name"],
@@ -138,7 +138,7 @@ def setup_upload_routes(upload_handler):
original_name = info.get("name", file_id)
auth_mgr = getattr(request.app.state, "auth_manager", None)
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
current_user = get_current_user(request)
current_user = effective_user(request)
file_owner = info.get("owner") if info else None
if auth_configured:
if not current_user:
@@ -204,7 +204,7 @@ def setup_upload_routes(upload_handler):
info = _load_upload_info(file_id)
auth_mgr = getattr(request.app.state, "auth_manager", None)
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
current_user = get_current_user(request)
current_user = effective_user(request)
file_owner = info.get("owner") if info else None
if auth_configured:
if not current_user:
@@ -247,7 +247,7 @@ def setup_upload_routes(upload_handler):
raise HTTPException(404, "File not found")
auth_mgr = getattr(request.app.state, "auth_manager", None)
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
current_user = get_current_user(request)
current_user = effective_user(request)
file_owner = info.get("owner")
if auth_configured:
if not current_user:
+2 -3
View File
@@ -1,6 +1,5 @@
"""Webhook, API Token, and sync chat routes."""
import asyncio
import uuid
import logging
from typing import Optional
@@ -385,10 +384,10 @@ def setup_webhook_routes(
sess.add_message(ChatMessage("assistant", reply))
session_manager.save_sessions()
asyncio.create_task(webhook_manager.fire("chat.completed", {
webhook_manager.fire_and_forget("chat.completed", {
"session_id": session_id, "model": sess.model,
"user_message": message[:2000], "response": reply[:2000],
}))
})
return {"response": reply, "session_id": session_id, "model": sess.model}
+4
View File
@@ -19,6 +19,10 @@ GPU_BANDWIDTH = {
"6950 xt": 576, "6900 xt": 512, "6800 xt": 512, "6800": 512, "6700 xt": 384, "6600 xt": 256, "6600": 224,
"mi300x": 5300, "mi300": 5300, "mi250x": 3277, "mi250": 3277, "mi210": 1638, "mi100": 1229,
"9070 xt": 624, "9070": 488, "9060 xt": 322, "9060": 322,
# NVIDIA GB10 Grace-Blackwell superchip (DGX Spark). Unified LPDDR5X memory,
# not Apple Silicon, so it lives in the generic GPU table — the Apple-only
# lookup never matches it (its name carries no "apple").
"gb10": 273,
}
# Pre-sort keys by length descending for correct substring matching
+163 -14
View File
@@ -15,6 +15,8 @@ from urllib.parse import urljoin, urlparse
import httpx
from bs4 import BeautifulSoup
from src.constants import WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES, WEB_FETCH_USER_AGENT
from .analytics import RateLimitError, error_logger
from .cache import (
CONTENT_CACHE_DIR,
@@ -89,18 +91,128 @@ def _public_http_url(url: str) -> bool:
return False
def _get_public_url(url: str, headers: dict, timeout: int, max_redirects: int = 5) -> httpx.Response:
class BodyTooLargeError(Exception):
"""The server declared a body larger than the hard fetch ceiling."""
def __init__(self, url: str, declared_bytes: int):
self.url = url
self.declared_bytes = declared_bytes
super().__init__(
f"response body is {declared_bytes:,} bytes, over the "
f"{WEB_FETCH_HARD_MAX_BYTES:,}-byte hard cap"
)
class _CappedFetch:
"""Result of a size-capped streaming GET.
Carries just what fetch_webpage_content needs from an httpx.Response,
plus the cap bookkeeping: the (possibly truncated) body, whether the
cap cut it short, and the size the server declared via Content-Length
(wire bytes; None when absent).
"""
__slots__ = ("status_code", "headers", "content", "truncated",
"declared_bytes", "encoding", "url")
def __init__(self, status_code, headers, content, truncated,
declared_bytes, encoding, url):
self.status_code = status_code
self.headers = headers
self.content = content
self.truncated = truncated
self.declared_bytes = declared_bytes
self.encoding = encoding
self.url = url
@property
def text(self) -> str:
return self.content.decode(self.encoding or "utf-8", errors="replace")
def raise_for_status(self):
if self.status_code >= 400:
request = httpx.Request("GET", self.url)
raise httpx.HTTPStatusError(
f"HTTP {self.status_code} for {self.url}",
request=request,
response=httpx.Response(self.status_code, request=request),
)
def _get_public_url(url: str, headers: dict, timeout: int, max_redirects: int = 5,
max_bytes: int = None) -> "_CappedFetch":
"""Capped streaming GET with SSRF-guarded manual redirects.
The body is streamed and buffering stops at ``max_bytes`` (default: the
soft cap), so an oversized resource cannot be pulled into memory or the
content cache in full. When Content-Length already declares a body over
the hard ceiling, the fetch is refused before any body bytes are read.
"""
cap = min(max_bytes or WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES)
current = url
for _ in range(max_redirects + 1):
if not _public_http_url(current):
raise httpx.RequestError("Blocked private/internal URL", request=httpx.Request("GET", current))
response = httpx.get(current, headers=headers, timeout=timeout, follow_redirects=False)
if response.status_code not in (301, 302, 303, 307, 308):
return response
location = response.headers.get("location")
if not location:
return response
current = urljoin(str(response.url), location)
# Force identity transfer-encoding. With gzip/deflate the wire bytes
# (and Content-Length) can be a small fraction of the decoded body, so
# a tiny compressed response could pass the hard-cap preflight and then
# expand past the ceiling in a single decoded chunk before the streamed
# cap below can slice it. Identity makes Content-Length the true body
# size and keeps each streamed chunk bounded by the network read.
req_headers = dict(headers or {})
req_headers["Accept-Encoding"] = "identity"
with httpx.stream("GET", current, headers=req_headers, timeout=timeout,
follow_redirects=False) as response:
if response.status_code in (301, 302, 303, 307, 308):
location = response.headers.get("location")
if not location:
return _CappedFetch(response.status_code, response.headers, b"",
False, None, response.encoding, str(response.url))
current = urljoin(str(response.url), location)
continue
# A server can ignore the identity request and still return a
# compressed body; httpx.iter_bytes would then decode it, and a tiny
# gzip can balloon into one decoded chunk far past the cap before we
# slice. Refuse a compressed Content-Encoding so the streamed cap
# stays a real memory bound (Content-Length is the compressed wire
# length here, so the preflight and size metadata are unreliable too).
enc = (response.headers.get("content-encoding") or "").strip().lower()
if enc and enc != "identity":
raise httpx.RequestError(
f"Refusing compressed response (Content-Encoding: {enc}) after "
"requesting identity: cannot bound decoded body size",
request=httpx.Request("GET", current),
)
declared = None
raw_len = response.headers.get("content-length")
if raw_len and raw_len.isdigit():
declared = int(raw_len)
# Refuse before buffering anything when the server already tells
# us the body exceeds the absolute ceiling (Content-Length is wire
# bytes; the decompressed body can only be larger).
if declared is not None and declared > WEB_FETCH_HARD_MAX_BYTES:
raise BodyTooLargeError(current, declared)
chunks = []
read = 0
truncated = False
# We requested identity above, so iter_bytes yields the raw body in
# network-read-sized chunks (no decompression expansion); the cap
# therefore bounds what we actually buffer.
for chunk in response.iter_bytes():
read += len(chunk)
if read > cap:
keep = cap - (read - len(chunk))
if keep > 0:
chunks.append(chunk[:keep])
truncated = True
break
chunks.append(chunk)
return _CappedFetch(response.status_code, response.headers,
b"".join(chunks), truncated, declared,
response.encoding, str(response.url))
raise httpx.RequestError("Too many redirects", request=httpx.Request("GET", current))
# PDF extraction (optional dependency)
@@ -222,9 +334,19 @@ def _empty_result(url: str, error: str = "") -> dict:
# ----------------------------------------------------------------------
# Main content fetcher
# ----------------------------------------------------------------------
def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) -> dict:
"""Fetch and extract meaningful content from a webpage with caching."""
cache_key = generate_cache_key(url)
def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0,
max_bytes: int = None) -> dict:
"""Fetch and extract meaningful content from a webpage with caching.
``max_bytes`` raises the download budget per call (clamped to the hard
cap); the default is the soft cap. When the body is cut short the result
carries ``truncated``/``fetched_bytes``/``total_bytes`` so callers can
tell the model the content is partial (#3812).
"""
effective_cap = min(max_bytes or WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES)
# The cap is part of the cache identity: a truncated soft-cap fetch must
# not be served to a later full-budget request for the same URL.
cache_key = generate_cache_key(f"{url}#cap={effective_cap}")
cache_file = CONTENT_CACHE_DIR / f"{cache_key}.cache"
# Check cache
@@ -247,18 +369,24 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
# Fetch
try:
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
"User-Agent": WEB_FETCH_USER_AGENT,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate",
# identity so the streamed size cap in _get_public_url stays honest
# (a compressed body can decode to far more than Content-Length).
"Accept-Encoding": "identity",
"Connection": "keep-alive",
}
response = _get_public_url(url, headers=headers, timeout=timeout)
response = _get_public_url(url, headers=headers, timeout=timeout,
max_bytes=effective_cap)
if response.status_code == 429:
raise RateLimitError(f"Rate limit hit for {url} (attempt {retry_attempt})")
response.raise_for_status()
except BodyTooLargeError as e:
error_logger.warning(f"Refused oversized body for {url}: {e}")
return _empty_result(url, f"TooLarge: {e}")
except httpx.HTTPStatusError as e:
error_logger.warning(f"HTTP {e.response.status_code} fetching {url}: {e}")
return _empty_result(url, f"HTTP {e.response.status_code}: {e}")
@@ -269,9 +397,27 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
error_logger.error(str(e))
return _empty_result(url, str(e))
# Size bookkeeping shared by every content branch below. getattr keeps
# plain httpx.Response stand-ins (tests) working without the cap fields.
_size_fields = {
"truncated": getattr(response, "truncated", False),
"fetched_bytes": len(response.content),
"total_bytes": getattr(response, "declared_bytes", None),
}
# PDF handling
content_type = response.headers.get("Content-Type", "").lower()
if "application/pdf" in content_type or url.lower().endswith(".pdf"):
if _size_fields["truncated"]:
# A PDF cut mid-stream is not parseable; unlike text there is no
# useful partial result, so report the budget problem instead.
_declared = _size_fields["total_bytes"]
return _empty_result(
url,
f"TooLarge: PDF exceeds the {effective_cap:,}-byte fetch budget"
+ (f" (size {_declared:,} bytes)" if _declared else "")
+ "; retry with a larger budget if it fits under the hard cap",
)
if pdf_extract_text is None:
logger.error("pdfminer.six is not installed; cannot extract PDF text.")
pdf_text = ""
@@ -295,6 +441,7 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
"js_message": "",
"success": bool(pdf_text),
"error": "" if pdf_text else "Failed to extract PDF text",
**_size_fields,
}
_cache_result(cache_file, cache_key, result, url)
return result
@@ -329,6 +476,7 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
"js_message": "",
"success": bool(text_body),
"error": "" if text_body else "Empty response body",
**_size_fields,
}
_cache_result(cache_file, cache_key, result, url)
return result
@@ -391,6 +539,7 @@ def fetch_webpage_content(url: str, timeout: int = 5, retry_attempt: int = 0) ->
"js_message": js_message,
"success": True,
"error": "",
**_size_fields,
}
_cache_result(cache_file, cache_key, result, url)
return result
+4 -6
View File
@@ -9,14 +9,12 @@ from urllib.parse import urljoin, urlparse, parse_qs
import httpx
from bs4 import BeautifulSoup
from src.constants import SEARXNG_INSTANCE
from src.constants import SEARXNG_INSTANCE, REQUEST_TIMEOUT, WEB_FETCH_USER_AGENT
from .analytics import RateLimitError, error_logger
from .query import build_enhanced_query
logger = logging.getLogger(__name__)
REQUEST_TIMEOUT = 20
# Provider registry — maps setting value to (label, needs_key, needs_url)
PROVIDER_INFO = {
"searxng": ("SearXNG", False, True),
@@ -140,7 +138,7 @@ def searxng_search_api(query: str, count: Optional[int] = None, categories: str
count = count if count is not None else _get_result_count()
instance = _get_search_instance()
api_key = ""
headers = {"User-Agent": "Mozilla/5.0"}
headers = {"User-Agent": WEB_FETCH_USER_AGENT}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
# News/fresh queries do badly in the 'general' category — it favours
@@ -252,7 +250,7 @@ def searxng_search(query, max_results=10):
"""Search using SearXNG instance - parsing HTML."""
instance = _get_search_instance()
api_key = ""
req_headers = {"User-Agent": "Mozilla/5.0"}
req_headers = {"User-Agent": WEB_FETCH_USER_AGENT}
if api_key:
req_headers["Authorization"] = f"Bearer {api_key}"
try:
@@ -391,7 +389,7 @@ def duckduckgo_search(query: str, count: Optional[int] = None, time_filter: Opti
response = httpx.get(
"https://html.duckduckgo.com/html/",
params={"q": query, "kp": _safesearch_for("duckduckgo_html")},
headers={"User-Agent": "Mozilla/5.0"},
headers={"User-Agent": WEB_FETCH_USER_AGENT},
timeout=REQUEST_TIMEOUT,
)
response.raise_for_status()
+5 -5
View File
@@ -524,7 +524,7 @@ def get_builtin_overrides() -> dict:
ov = get_setting("builtin_tool_overrides", {})
return ov if isinstance(ov, dict) else {}
except Exception as e:
logger.warning('Failed to load builtin tool overrides: %s', e)
logger.warning("Failed to load builtin tool overrides, using defaults", exc_info=e)
return {}
@@ -929,8 +929,8 @@ def _build_system_prompt(
try:
from src.user_time import current_datetime_context_message
_datetime_message = current_datetime_context_message()
except Exception:
pass
except Exception as e:
logger.warning("Failed to build datetime context message", exc_info=e)
# Document context is kept as a SEPARATE message (not merged into the tool
# prompt) so the context trimmer doesn't destroy it when truncating the
@@ -973,8 +973,8 @@ def _build_system_prompt(
try:
from src.pdf_form_doc import find_source_upload_id
_is_form_backed = bool(find_source_upload_id(active_document.current_content or ""))
except Exception:
pass
except Exception as e:
logger.warning("Failed to detect if document is form-backed, assuming plain", exc_info=e)
if _is_form_backed:
doc_ctx = (
+32 -2
View File
@@ -57,13 +57,23 @@ class WebSearchTool:
class WebFetchTool:
async def execute(self, content: str, ctx: dict) -> dict:
from src.search.content import fetch_webpage_content
from src.constants import WEB_FETCH_HARD_MAX_BYTES
raw = content.strip()
url = ""
max_bytes = None
if raw.startswith("{"):
try:
parsed = json.loads(raw)
if isinstance(parsed, dict):
url = str(parsed.get("url") or "").strip()
# Download-budget override (#3812): "full": true raises the
# budget to the hard cap; an explicit max_bytes is clamped
# to the hard cap downstream. Default stays the soft cap.
if parsed.get("full") is True:
max_bytes = WEB_FETCH_HARD_MAX_BYTES
mb = parsed.get("max_bytes")
if isinstance(mb, int) and mb > 0:
max_bytes = mb
except json.JSONDecodeError:
url = ""
if not url:
@@ -78,7 +88,7 @@ class WebFetchTool:
loop = asyncio.get_running_loop()
try:
result = await asyncio.wait_for(
loop.run_in_executor(None, lambda: fetch_webpage_content(url, timeout=10)),
loop.run_in_executor(None, lambda: fetch_webpage_content(url, timeout=10, max_bytes=max_bytes)),
timeout=30,
)
except asyncio.TimeoutError:
@@ -94,8 +104,28 @@ class WebFetchTool:
return {"error": f"web_fetch: {url}: {err}", "exit_code": 1}
return {"error": f"web_fetch: {url}: no readable text content (not HTML, or the page needs JS/login)", "exit_code": 1}
# Tell the model when the download budget cut the body short and how
# to get the rest, instead of silently presenting a partial page as
# the whole thing.
size_note = ""
if result.get("truncated"):
fetched = result.get("fetched_bytes") or 0
total = result.get("total_bytes")
total_txt = f" of {total:,} bytes" if total else ""
size_note = (
f"[partial content: download stopped at {fetched:,} bytes{total_txt}. "
f'Re-call with {{"url": "{url}", "full": true}} to fetch up to '
f"{WEB_FETCH_HARD_MAX_BYTES:,} bytes.]\n\n"
)
# The notice must lead the output so the MAX_OUTPUT_CHARS trim below can
# never drop it. The title is untrusted, uncapped page content, so a
# giant title ahead of the notice could push it out of range; keep the
# notice first and cap the title as a second guard.
if len(title) > 300:
title = title[:300] + "..."
header = (f"# {title}\n" if title else "") + f"Source: {url}\n\n"
output = header + text
output = size_note + header + text
if len(output) > MAX_OUTPUT_CHARS:
output = output[:MAX_OUTPUT_CHARS] + "\n\n[...truncated]"
return {"output": output, "exit_code": 0}
+12 -3
View File
@@ -1613,7 +1613,9 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
"""
import base64
import httpx
import os
from pathlib import Path
from src.url_safety import check_outbound_url
lines = content.strip().split("\n")
prompt = lines[0].strip() if lines else ""
@@ -1779,8 +1781,15 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
elif img.get("url"):
# Download external URL and save locally (DALL-E returns temp URLs)
result_url = img["url"]
ok, reason = check_outbound_url(
result_url,
block_private=os.getenv("IMAGE_BLOCK_PRIVATE_IPS", "false").lower() == "true",
)
if not ok:
return {"error": f"Image API returned unsafe image URL: {reason}"}
try:
dl_resp = httpx.get(img["url"], timeout=60)
dl_resp = httpx.get(result_url, timeout=60)
if dl_resp.status_code == 200:
img_dir = Path(GENERATED_IMAGES_DIR)
img_dir.mkdir(parents=True, exist_ok=True)
@@ -1790,10 +1799,10 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
image_url = f"/api/generated-image/{filename}"
image_id = _save_to_gallery(filename)
else:
image_url = img["url"] # fallback to external URL
image_url = result_url # fallback to external URL
except Exception as _dl_e:
logger.warning(f"Failed to download DALL-E image: {_dl_e}")
image_url = img["url"] # fallback to external URL
image_url = result_url # fallback to external URL
else:
return {"error": "Image API returned unexpected format (no b64_json or url)"}
+3 -2
View File
@@ -14,6 +14,7 @@ import subprocess
import sys
from core.platform_compat import IS_WINDOWS, which_tool
from src.runtime_paths import get_app_root
logger = logging.getLogger(__name__)
@@ -81,7 +82,7 @@ _BUILTIN_NPX_SERVERS = {
"name": "Built-in: Browser",
"command": "npx",
"args": ["-y", "@playwright/mcp@latest", "--headless", "--caps", "vision"],
},
}
}
# Global flag to disable MCP if there are compatibility issues
@@ -94,7 +95,7 @@ async def register_builtin_servers(mcp_manager):
logger.info("Built-in MCP servers disabled via ODYSSEUS_DISABLE_MCP")
return
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
base_dir = get_app_root()
python = sys.executable
async def _connect_python_server(server_id: str, script_path: str, name: str):
+3 -2
View File
@@ -5,6 +5,7 @@ from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field, field_validator
from src.constants import DATA_DIR as _DATA_DIR_CONST
from src.runtime_paths import get_app_root
# Cross-platform OS flag, exposed here so callers can `from src.config import
# IS_WINDOWS`. Defined locally (a trivial `os.name == "nt"`) rather than imported
@@ -19,7 +20,7 @@ IS_WINDOWS = os.name == "nt"
class DataConfig(BaseSettings):
"""Configuration for data storage and file handling."""
# Base directory
base_dir: Path = Field(default=Path(__file__).parent.parent, description="Base directory for the application")
base_dir: Path = Field(default=Path(get_app_root()), description="Base directory for the application")
# Data paths
data_dir: Path = Field(default=Path(_DATA_DIR_CONST), description="Main data directory")
@@ -138,7 +139,7 @@ class AppConfig(BaseSettings):
if isinstance(v, dict) and "base_dir" in v:
base_dir = v["base_dir"]
else:
base_dir = Path(__file__).parent.parent
base_dir = Path(get_app_root())
# Convert string paths to Path objects relative to base_dir
data_dir = Path(_DATA_DIR_CONST)
+26 -3
View File
@@ -2,12 +2,14 @@
"""Application-wide constants and configuration values."""
import os
from src.runtime_paths import get_app_root, get_default_data_dir
APP_VERSION = "1.0.0"
# Base paths
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + "/"
BASE_DIR = os.path.join(get_app_root(), "")
STATIC_DIR = os.path.join(BASE_DIR, "static")
DATA_DIR = os.getenv("ODYSSEUS_DATA_DIR", os.path.join(BASE_DIR, "data"))
DATA_DIR = os.getenv("ODYSSEUS_DATA_DIR", get_default_data_dir())
# Data file paths
# Single source of truth: every persisted file/dir lives under DATA_DIR, which
@@ -55,7 +57,13 @@ MEMORY_VECTORS_DIR = os.path.join(DATA_DIR, "memory_vectors")
# Paths with an intentional dedicated env override, defaulting under DATA_DIR.
MAIL_ATTACHMENTS_DIR = os.getenv("ODYSSEUS_MAIL_ATTACHMENTS_DIR", os.path.join(DATA_DIR, "mail-attachments"))
FASTEMBED_CACHE_DIR = os.getenv("FASTEMBED_CACHE_PATH", os.path.join(DATA_DIR, "fastembed_cache"))
# `or` (not os.getenv's default arg) so a PRESENT-but-EMPTY value falls back to
# the default. docker-compose.yml injects `FASTEMBED_CACHE_PATH=${FASTEMBED_CACHE_PATH:-}`,
# which sets the var to "" when the host hasn't defined it. os.getenv(name, default)
# only returns the default when the var is ABSENT, so the empty string would win →
# os.makedirs("") raises [Errno 2] No such file or directory: '' → FastEmbed fails to
# init and all vector features (RAG, semantic memory, tool index) silently degrade.
FASTEMBED_CACHE_DIR = os.getenv("FASTEMBED_CACHE_PATH") or os.path.join(DATA_DIR, "fastembed_cache")
# Agent tool output limits (single source of truth — imported by tool_execution.py,
# tool_implementations.py, agent_tools.py, and any other module that needs them)
@@ -63,11 +71,26 @@ MAX_OUTPUT_CHARS = 10_000 # cap for bash/python/web_search/web_fetch outpu
MAX_READ_CHARS = 20_000 # cap for read_file / document preview
MAX_DIFF_LINES = 400 # cap for edit_file unified-diff display
# web_fetch response-size policy (#3812). MAX_OUTPUT_CHARS above only trims
# what the agent SEES; these caps bound what the server downloads, parses,
# and writes to the content cache. The soft cap is the default download
# budget; the agent can raise it per call (full/max_bytes) but never past
# the hard cap, so a model can't decide to pull a multi-GB file.
WEB_FETCH_SOFT_MAX_BYTES = 2_000_000 # default download budget (2 MB)
WEB_FETCH_HARD_MAX_BYTES = 20_000_000 # absolute ceiling, even with override (20 MB)
# API Configuration
MAX_CONTEXT_MESSAGES = 90
REQUEST_TIMEOUT = 20
OPENAI_COMPAT_PATH = "/v1/chat/completions"
# Outbound UA for web_fetch / web_search scraping; common desktop UA so pages serve normal HTML.
WEB_FETCH_USER_AGENT = os.environ.get(
"WEB_FETCH_USER_AGENT",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
)
# Environment variables with defaults
DEFAULT_HOST = os.getenv("LLM_HOST", "localhost")
LLM_HOSTS = [h.strip() for h in os.getenv("LLM_HOSTS", "").split(",") if h.strip()]
+3 -2
View File
@@ -161,11 +161,13 @@ async def _tick() -> None:
# Re-read state once before writing so we capture any updates from
# concurrent UI syncs.
stopped_any = False
successfully_stopped_sids = set()
for sid, host, port in to_stop:
ok = await _stop_serve(sid, host, port)
logger.info(f"cookbook_serve_lifecycle: stop {sid} (host={host or 'local'}): {'ok' if ok else 'failed'}")
if ok:
stopped_any = True
successfully_stopped_sids.add(sid)
# Drop the auto-registered endpoint so the model picker and
# the chat router don't keep pointing at a dead server.
for t in tasks:
@@ -188,12 +190,11 @@ async def _tick() -> None:
except Exception:
fresh = state
fresh_tasks = tasks
stopped_sids = {sid for sid, _, _ in to_stop}
for ft in fresh_tasks:
if not isinstance(ft, dict):
continue
ft_sid = ft.get("sessionId") or ft.get("id")
if ft_sid in stopped_sids:
if ft_sid in successfully_stopped_sids:
ft["status"] = "stopped"
ft["_scheduledStopAtMs"] = None
ft["_lastStatusFlipAt"] = now_ms
+2
View File
@@ -31,6 +31,8 @@ import numpy as np
import httpx
from typing import List, Optional
from src.runtime_paths import get_app_root
logger = logging.getLogger(__name__)
_DEFAULT_MODEL = "all-minilm:l6-v2"
+4 -3
View File
@@ -283,7 +283,8 @@ def _is_ollama_native_url(url: str) -> bool:
"""Return True for native Ollama API URLs, including Ollama Cloud."""
try:
parsed = urlparse(url or "")
except Exception:
except Exception as e:
logger.warning("Failed to parse URL for Ollama detection", exc_info=e)
return False
host = parsed.hostname or ""
path = (parsed.path or "").rstrip("/")
@@ -1345,8 +1346,8 @@ def list_model_ids(
r = httpx.get(root + "/api/tags", timeout=timeout)
r.raise_for_status()
return [m.get("name") or m.get("model") for m in (r.json().get("models") or []) if m.get("name") or m.get("model")]
except Exception:
pass
except Exception as e:
logger.warning("Failed to fetch model list from configured endpoint", exc_info=e)
return []
def normalize_model_id(
+3 -1
View File
@@ -11,6 +11,8 @@ import os
import re
from typing import Any, Dict, List, Optional, Set, Tuple
from src.runtime_paths import get_app_root
logger = logging.getLogger(__name__)
def _format_mcp_connection_error(name: str, command: str = "", args: Optional[List[str]] = None, error: Exception = None) -> str:
@@ -508,7 +510,7 @@ class McpManager:
return False
script_rel, name = _BUILTIN_SERVERS[server_id]
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
base_dir = get_app_root()
script_path = os.path.join(base_dir, script_rel)
# Clean up old connection
+1
View File
@@ -7,6 +7,7 @@ import time
from pathlib import Path
from src.constants import RAG_DIR
from src.runtime_paths import get_app_root
logger = logging.getLogger(__name__)
+30
View File
@@ -0,0 +1,30 @@
"""Helpers for resolving runtime paths in source and frozen builds."""
import os
import sys
def get_app_root() -> str:
"""Return the app root directory.
In normal source runs, this is the repository root. In a frozen Windows
build, it is the bundle content root (PyInstaller's internal directory)
so bundled runtime folders like `static/`, `scripts/`, and `data/` stay
together with the executable payload.
"""
if getattr(sys, "frozen", False):
return getattr(sys, "_MEIPASS", os.path.dirname(os.path.abspath(sys.executable)))
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def get_default_data_dir() -> str:
"""Return the default path to the data directory.
In normal runs, this is a 'data' subdirectory under the app root.
In frozen builds, it is a persistent user directory (~/.odysseus/data)
to prevent SQLite databases and other persistent files from being
written to the ephemeral, temporary extraction bundle directory.
"""
if getattr(sys, "frozen", False):
return os.path.join(os.path.expanduser("~"), ".odysseus", "data")
return os.path.join(get_app_root(), "data")
+24 -5
View File
@@ -236,6 +236,29 @@ def _digest_windows(now):
]
def _checkin_calendar_events(db, owner, start, end):
"""Calendar events in [start, end] for ONE owner, for the check-in digest.
Ownership lives on CalendarCal.owner; events inherit it via calendar_id.
The digest query had no owner scope, so it pulled EVERY user's events into
one user's check-in (a cross-tenant leak of summaries/locations). Scope it
by joining CalendarCal, mirroring routes/calendar_routes.list_events.
"""
from core.database import CalendarEvent as _CE, CalendarCal as _CC
return (
db.query(_CE)
.join(_CC, _CE.calendar_id == _CC.id)
.filter(
_CC.owner == owner,
_CE.dtstart >= start,
_CE.dtstart <= end,
_CE.status != "cancelled",
)
.order_by(_CE.dtstart)
.all()
)
class TaskScheduler:
def __init__(self, session_manager):
self._session_manager = session_manager
@@ -1127,11 +1150,7 @@ class TaskScheduler:
# Strip timezone for naive DB comparison
_s = start.replace(tzinfo=None) if start.tzinfo else start
_e = end.replace(tzinfo=None) if end.tzinfo else end
evs = _db.query(_CE).filter(
_CE.dtstart >= _s,
_CE.dtstart <= _e,
_CE.status != "cancelled",
).order_by(_CE.dtstart).all()
evs = _checkin_calendar_events(_db, task.owner, _s, _e)
if not evs:
continue
# Group by importance for richer output
+17 -6
View File
@@ -1579,10 +1579,10 @@ async def do_manage_calendar(content: str, owner: Optional[str] = None) -> Dict:
text = str(raw).strip().lower()
if text in {"none", "no", "off", "false"}:
return None
m = re.search(r"(\d+)\s*(?:m|min|minute|minutes)\b", text)
m = re.search(r"(\d+)\s*(?:minutes?|mins?|m)\b", text)
if m:
return max(0, int(m.group(1)))
m = re.search(r"(\d+)\s*(?:h|hr|hour|hours)\b", text)
m = re.search(r"(\d+)\s*(?:hours?|hrs?|h)\b", text)
if m:
return max(0, int(m.group(1)) * 60)
if text.isdigit():
@@ -1595,7 +1595,7 @@ async def do_manage_calendar(content: str, owner: Optional[str] = None) -> Dict:
return desc
reminder_only = re.compile(
r"^\s*(?:remind(?:er)?|alarm)\s*:?\s*\d+\s*"
r"(?:m|min|minute|minutes|h|hr|hour|hours)\b.*$",
r"(?:minutes?|mins?|m|hours?|hrs?|h)\b.*$",
re.I,
)
return "" if reminder_only.match(desc) else desc
@@ -3797,7 +3797,7 @@ async def do_resolve_contact(content: str, owner: Optional[str] = None) -> Dict:
if not name:
return {"error": "name is required", "exit_code": 1}
contacts = {} # email -> {name, source}
contacts = {} # email_or_phone -> {name, source, phone?}
# 1. CardDAV (Radicale) — structured contacts. Call in-process: a
# server-side httpx GET to /api/contacts/search carries no session
@@ -3812,10 +3812,18 @@ async def do_resolve_contact(content: str, owner: Optional[str] = None) -> Dict:
match = q in hay_name or any(q in (e or "").lower() for e in c.get("emails", []))
if not match:
continue
has_email = False
for email in (c.get("emails") or []):
email = (email or "").strip().lower()
if email and "@" in email:
contacts[email] = {"name": c.get("name") or email, "source": "contacts"}
has_email = True
# Fall back to phone numbers when the contact has no email address
if not has_email:
for phone in (c.get("phones") or []):
phone = (phone or "").strip()
if phone:
contacts[phone] = {"name": c.get("name") or phone, "source": "contacts", "phone": phone}
except Exception:
pass
@@ -3835,8 +3843,11 @@ async def do_resolve_contact(content: str, owner: Optional[str] = None) -> Dict:
return {"output": f"No contacts found matching '{name}'.", "exit_code": 0}
lines = [f"Contacts matching '{name}':"]
for email, info in contacts.items():
lines.append(f"- {info['name']} <{email}> ({info['source']})")
for key, info in contacts.items():
if info.get("phone"):
lines.append(f"- {info['name']} — phone: {info['phone']} ({info['source']})")
else:
lines.append(f"- {info['name']} <{key}> ({info['source']})")
return {"output": "\n".join(lines), "exit_code": 0}
+4 -3
View File
@@ -68,11 +68,12 @@ FUNCTION_TOOL_SCHEMAS = [
"type": "function",
"function": {
"name": "web_fetch",
"description": "Fetch and read the text content of a specific URL the user names (e.g. 'check example.com', 'what's on this page <url>'). Use when you already have a concrete URL/domain. NOT for open-ended searches (use web_search) or 'research X' jobs (use trigger_research).",
"description": "Fetch and read the text content of a specific URL the user names (e.g. 'check example.com', 'what's on this page <url>'). Use when you already have a concrete URL/domain. NOT for open-ended searches (use web_search) or 'research X' jobs (use trigger_research). Downloads are size-budgeted; a '[partial content: ...]' notice in the result means the body was cut short and you can re-call with full=true for the rest.",
"parameters": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "The URL or domain to fetch (http/https; a bare domain like example.com is fine)"}
"url": {"type": "string", "description": "The URL or domain to fetch (http/https; a bare domain like example.com is fine)"},
"full": {"type": "boolean", "description": "Raise the download budget to the hard cap for large pages/files. Use only after a result reported partial content."}
},
"required": ["url"]
}
@@ -1008,7 +1009,7 @@ FUNCTION_TOOL_SCHEMAS = [
"type": "function",
"function": {
"name": "resolve_contact",
"description": "Look up a contact's email address by name. Searches CardDAV address book and sent email history. Use when the user says 'message [name]' or 'email [name]' without an email address.",
"description": "Look up a contact by name. Searches CardDAV address book and sent email history. Returns email addresses (when available) or phone numbers. Use when the user says 'message [name]', 'email [name]', or asks for someone's contact details.",
"parameters": {
"type": "object",
"properties": {
+2 -1
View File
@@ -87,7 +87,8 @@ import * as Modals from './modalManager.js';
}
function _accountCanSend(account) {
return !!(account && account.smtp_host && account.smtp_user && account.has_smtp_password);
if (!account || !account.smtp_host || !account.smtp_user) return false;
return !!(account.has_smtp_password || account.oauth_provider);
}
async function _resolveComposeSendAccountId() {
+128 -10
View File
@@ -2913,13 +2913,14 @@ async function initEmailAccountsSettings() {
// IMAP and SMTP. Dovecot is IMAP-only here; the host is intentionally
// blank because it may live on another machine (DNS, LAN, Tailscale).
const PROVIDERS = {
gmail: { label: 'Gmail', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 465 } },
migadu: { label: 'Migadu', imap: { host: 'imap.migadu.com', port: 993, starttls: false }, smtp: { host: 'smtp.migadu.com', port: 465 } },
icloud: { label: 'iCloud', imap: { host: 'imap.mail.me.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.me.com', port: 587 } },
outlook: { label: 'Outlook / Office 365', imap: { host: 'outlook.office365.com', port: 993, starttls: false }, smtp: { host: 'smtp.office365.com', port: 587 } },
fastmail: { label: 'Fastmail', imap: { host: 'imap.fastmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.fastmail.com', port: 465 } },
yahoo: { label: 'Yahoo', imap: { host: 'imap.mail.yahoo.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.yahoo.com', port: 465 } },
dovecot: { label: 'Dovecot IMAP (no SMTP)', imap: { host: '', port: 31143, starttls: false }, smtp: { host: '', port: 465 } },
gmail: { label: 'Gmail', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 465 } },
google_workspace: { label: 'Google Workspace / .edu', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 587 }, oauth: 'google' },
migadu: { label: 'Migadu', imap: { host: 'imap.migadu.com', port: 993, starttls: false }, smtp: { host: 'smtp.migadu.com', port: 465 } },
icloud: { label: 'iCloud', imap: { host: 'imap.mail.me.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.me.com', port: 587 } },
outlook: { label: 'Outlook / Office 365', imap: { host: 'outlook.office365.com', port: 993, starttls: false }, smtp: { host: 'smtp.office365.com', port: 587 } },
fastmail: { label: 'Fastmail', imap: { host: 'imap.fastmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.fastmail.com', port: 465 } },
yahoo: { label: 'Yahoo', imap: { host: 'imap.mail.yahoo.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.yahoo.com', port: 465 } },
dovecot: { label: 'Dovecot IMAP (no SMTP)', imap: { host: '', port: 31143, starttls: false }, smtp: { host: '', port: 465 } },
};
const _providerOptions = Object.entries(PROVIDERS)
.map(([k, v]) => `<option value="${k}">${esc(v.label)}</option>`)
@@ -2932,11 +2933,17 @@ async function initEmailAccountsSettings() {
<div id="eaf-provider-note" style="display:none;font-size:11px;line-height:1.5;padding:8px 10px;margin:2px 0 4px;border:1px solid color-mix(in srgb, var(--fg) 15%, transparent);border-left:3px solid var(--accent, var(--red));border-radius:4px;background:color-mix(in srgb, var(--fg) 4%, transparent);"></div>
<div class="settings-row"><label class="settings-label">Name${_hint('Optional label for this account (e.g. “Work” or “Personal”). Leave blank to use the email address.')}</label><input id="eaf-name" class="settings-input" placeholder="(optional leave blank to use email)" value="${esc(a.name || '')}"></div>
<div class="settings-row"><label class="settings-label">Email${_hint('Your email address. Used as the From: header on outgoing mail and as the display label when Name is blank.')}</label><input id="eaf-from" class="settings-input" placeholder="you@example.com" value="${esc(a.from_address || '')}"></div>
<div class="settings-row"><label class="settings-label">Display Name${_hint('Your name as it appears in the From: field of emails you send, e.g. Jane Smith. Auto-filled from Google during OAuth.')}</label><input id="eaf-display-name" class="settings-input" placeholder="Your Name" value="${esc(a.display_name || '')}"></div>
<div id="eaf-oauth-section" style="display:none;margin:8px 0;padding:10px;border:1px solid var(--border);border-radius:6px;background:color-mix(in srgb,var(--accent,#50fa7b) 6%,transparent)">
<div style="font-size:11px;font-weight:600;margin-bottom:6px">Google OAuth2 required for Workspace / .edu accounts</div>
<div id="eaf-oauth-status" style="font-size:11px;opacity:0.7;margin-bottom:6px">${a.oauth_provider === 'google' ? '✓ Connected via Google OAuth' : 'Not connected — click below to authorize'}</div>
<button type="button" id="eaf-oauth-btn" class="admin-btn-add" style="font-size:11px">${a.oauth_provider === 'google' ? 'Reconnect with Google' : 'Connect with Google'}</button>
</div>
<div style="font-size:11px;font-weight:600;opacity:0.6;margin:6px 0 2px">IMAP (Receiving)</div>
<div class="settings-row"><label class="settings-label">Host${_hint('Your IMAP server, e.g. imap.gmail.com, imap.migadu.com, a LAN host, or a Tailscale IP for Dovecot.')}</label><input id="eaf-imap-host" class="settings-input" value="${esc(a.imap_host || '')}"></div>
<div class="settings-row"><label class="settings-label">Port${_hint('993 for IMAPS (most providers), 143 for plain or STARTTLS. Local servers often use a custom port like 31143.')}</label><input id="eaf-imap-port" class="settings-input" type="number" value="${esc(a.imap_port || 993)}" style="max-width:100px"></div>
<div class="settings-row"><label class="settings-label">Username${_hint('Usually your full email address.')}</label><input id="eaf-imap-user" class="settings-input" value="${esc(a.imap_user || '')}"></div>
<div class="settings-row"><label class="settings-label">Password${_hint('Your IMAP login password. Use an app-specific password if your provider requires 2FA. Outlook / Office 365 generally requires OAuth and will not work with a normal password here.')}</label><input id="eaf-imap-pass" class="settings-input" type="password" placeholder="${isEdit && a.has_imap_password ? '(unchanged)' : ''}"></div>
<div class="eaf-password-section"><div class="settings-row"><label class="settings-label">Password${_hint('Your IMAP login password. Use an app-specific password if your provider requires 2FA. Outlook / Office 365 generally requires OAuth and will not work with a normal password here.')}</label><input id="eaf-imap-pass" class="settings-input" type="password" placeholder="${isEdit && a.has_imap_password ? '(unchanged)' : ''}"></div></div>
<div class="settings-row"><label class="settings-label">STARTTLS${_hint('Turn ON for port 143/587 to upgrade plain to TLS. Turn OFF for port 993 (IMAPS — already encrypted) or a local server with no TLS configured.')}</label><label class="admin-switch"><input type="checkbox" id="eaf-imap-starttls" ${a.imap_starttls !== false ? 'checked' : ''}><span class="admin-slider"></span></label></div>
<div style="font-size:11px;font-weight:600;opacity:0.6;margin:8px 0 2px">SMTP (Sending) <span style="font-weight:normal;opacity:0.7"> optional, leave blank for read-only</span></div>
<div class="settings-row"><label class="settings-label">Host${_hint('Your outgoing-mail server, e.g. smtp.gmail.com, smtp.migadu.com. Leave blank to make this account read-only.')}</label><input id="eaf-smtp-host" class="settings-input" value="${esc(a.smtp_host || '')}"></div>
@@ -2959,6 +2966,16 @@ async function initEmailAccountsSettings() {
</div>
`;
// Show/hide OAuth section and password fields based on provider selection.
function _syncOauthUI(providerKey) {
const p = PROVIDERS[providerKey];
const isOauth = !!(p && p.oauth);
el('eaf-oauth-section').style.display = isOauth ? '' : 'none';
formEl.querySelectorAll('.eaf-password-section').forEach(r => {
r.style.display = isOauth ? 'none' : '';
});
}
const eafProviderNotes = {
outlook: {
title: 'Outlook / Office 365 needs OAuth',
@@ -2983,13 +3000,41 @@ async function initEmailAccountsSettings() {
el('eaf-provider').addEventListener('change', (e) => {
_renderEafProviderNote(e.target.value);
const p = PROVIDERS[e.target.value];
if (!p) return;
if (!p) { _syncOauthUI(''); return; }
el('eaf-imap-host').value = p.imap.host;
el('eaf-imap-port').value = p.imap.port;
el('eaf-imap-starttls').checked = !!p.imap.starttls;
el('eaf-smtp-host').value = p.smtp.host;
el('eaf-smtp-port').value = p.smtp.port;
el('eaf-smtp-security').value = p.smtp.security || ((parseInt(p.smtp.port || 465) === 587) ? 'starttls' : 'ssl');
_syncOauthUI(e.target.value);
});
// Init OAuth UI for accounts already connected via OAuth.
if (a.oauth_provider === 'google') _syncOauthUI('google_workspace');
// "Connect with Google" button — save the account first, then redirect to OAuth.
el('eaf-oauth-btn').addEventListener('click', async () => {
// Must save the account first to get an account_id to pass to the OAuth flow.
const body = {
name: el('eaf-name').value.trim() || el('eaf-from').value.trim(),
from_address: el('eaf-from').value.trim(),
imap_host: el('eaf-imap-host').value.trim(),
imap_port: parseInt(el('eaf-imap-port').value) || 993,
imap_user: el('eaf-imap-user').value.trim(),
imap_starttls: el('eaf-imap-starttls').checked,
smtp_host: el('eaf-smtp-host').value.trim(),
smtp_port: parseInt(el('eaf-smtp-port').value) || 587,
smtp_user: el('eaf-imap-user').value.trim(),
};
if (!body.name) { el('eaf-msg').textContent = 'Enter a Name or Email first'; el('eaf-msg').style.color = 'var(--red)'; return; }
const url = isEdit ? `/api/email/accounts/${a.id}` : '/api/email/accounts';
const method = isEdit ? 'PUT' : 'POST';
const r = await fetch(url, { method, credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
const d = await r.json();
if (!d.ok) { el('eaf-msg').textContent = d.error || 'Save failed'; el('eaf-msg').style.color = 'var(--red)'; return; }
const accId = isEdit ? a.id : d.id;
window.location.href = `/api/email/oauth/google/authorize?account_id=${encodeURIComponent(accId)}`;
});
el('eaf-smtp-security').value = _smtpSecurity(a);
@@ -3009,6 +3054,7 @@ async function initEmailAccountsSettings() {
const body = {
name: el('eaf-name').value.trim(),
from_address: el('eaf-from').value.trim(),
display_name: el('eaf-display-name').value.trim(),
imap_host: el('eaf-imap-host').value.trim(),
imap_port: parseInt(el('eaf-imap-port').value) || 993,
imap_user: el('eaf-imap-user').value.trim(),
@@ -4317,6 +4363,7 @@ async function initUnifiedIntegrations() {
// it may be remote (DNS, LAN, Tailscale), not localhost.
const PROVIDERS = {
gmail: { label: 'Gmail', emailEx: 'you@gmail.com', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 465 } },
google_workspace: { label: 'Google Workspace / .edu', emailEx: 'you@yourschool.edu', imap: { host: 'imap.gmail.com', port: 993, starttls: false }, smtp: { host: 'smtp.gmail.com', port: 587 }, oauth: 'google' },
migadu: { label: 'Migadu', emailEx: 'you@yourdomain.com', imap: { host: 'imap.migadu.com', port: 993, starttls: false }, smtp: { host: 'smtp.migadu.com', port: 465 } },
icloud: { label: 'iCloud', emailEx: 'you@icloud.com', imap: { host: 'imap.mail.me.com', port: 993, starttls: false }, smtp: { host: 'smtp.mail.me.com', port: 587 } },
outlook: { label: 'Outlook / Office 365', emailEx: 'you@outlook.com', imap: { host: 'outlook.office365.com', port: 993, starttls: false }, smtp: { host: 'smtp.office365.com', port: 587 } },
@@ -4334,6 +4381,7 @@ async function initUnifiedIntegrations() {
const PROV_LOGO = {
'': _customLogo,
gmail: _letterLogo('G', '#ea4335'),
google_workspace: _letterLogo('G', '#ea4335'),
migadu: _letterLogo('M', '#3aa39d'),
icloud: _letterLogo('i', '#3693f3'),
outlook: _letterLogo('O', '#0078d4'),
@@ -4362,11 +4410,17 @@ async function initUnifiedIntegrations() {
<div id="uf-email-provider-note" style="display:none;font-size:11px;line-height:1.5;padding:8px 10px;margin:2px 0 4px;border:1px solid color-mix(in srgb, var(--fg) 15%, transparent);border-left:3px solid var(--accent, var(--red));border-radius:4px;background:color-mix(in srgb, var(--fg) 4%, transparent);"></div>
<div class="settings-row"><label class="settings-label">Name${_hint('Optional label for this account (e.g. “Work” or “Personal”). Leave blank to use the email address.')}</label><input id="uf-email-name" class="settings-input" placeholder="(optional leave blank to use email)"></div>
<div class="settings-row"><label class="settings-label">Email${_hint('Your email address. Used as the From: header on outgoing mail and as the display label when Name is blank.')}</label><input id="uf-email-from" class="settings-input" placeholder="you@example.com"></div>
<div class="settings-row"><label class="settings-label">Display Name${_hint('Your name as it appears in the From: field of emails you send, e.g. Jane Smith. Auto-filled from Google during OAuth.')}</label><input id="uf-display-name" class="settings-input" placeholder="Your Name"></div>
<div id="uf-oauth-section" style="display:none;margin:8px 0;padding:10px;border:1px solid var(--border);border-radius:6px;background:color-mix(in srgb,var(--accent,#50fa7b) 6%,transparent)">
<div style="font-size:11px;font-weight:600;margin-bottom:6px">Google OAuth2 required for Workspace / .edu accounts</div>
<div id="uf-oauth-status" style="font-size:11px;opacity:0.7;margin-bottom:6px">${existing && existing.oauth_provider === 'google' ? '✓ Connected via Google OAuth' : 'Not connected — click below to authorize'}</div>
<button type="button" id="uf-oauth-btn" class="admin-btn-add" style="font-size:11px">${existing && existing.oauth_provider === 'google' ? 'Reconnect with Google' : 'Connect with Google'}</button>
</div>
<div style="font-size:11px;font-weight:600;opacity:0.6;margin:4px 0 2px;display:flex;align-items:center;gap:5px;"><svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="color:var(--accent, var(--red));flex-shrink:0;" aria-hidden="true"><polyline points="22 12 16 12 14 15 10 15 8 12 2 12"/><path d="M5.45 5.11 2 12v6a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2v-6l-3.45-6.89A2 2 0 0 0 16.76 4H7.24a2 2 0 0 0-1.79 1.11z"/></svg>IMAP (Receiving)</div>
<div class="settings-row"><label class="settings-label">Host${_hint('Your IMAP server, e.g. imap.gmail.com, imap.migadu.com, a LAN host, or a Tailscale IP for Dovecot.')}</label><input id="uf-imap-host" class="settings-input" placeholder="imap.example.com"></div>
<div class="settings-row"><label class="settings-label">Port${_hint('993 for IMAPS (most providers), 143 for plain or STARTTLS. Local servers often use a custom port like 31143.')}</label><input id="uf-imap-port" class="settings-input" type="number" placeholder="993" style="max-width:100px"></div>
<div class="settings-row"><label class="settings-label">Username${_hint('Yes — your full email address goes here too (e.g. you@gmail.com). Same as the Email field above for almost every provider.')}</label><input id="uf-imap-user" class="settings-input" placeholder="you@example.com"></div>
<div class="settings-row"><label class="settings-label">Password${_hint('For Gmail, iCloud, and Yahoo: paste your App Password (NOT your normal account password). For Migadu and Fastmail, your mailbox password usually works. Outlook / Office 365 generally requires OAuth and will not work with this password form.')}</label><input id="uf-imap-pass" class="settings-input" type="password" placeholder="${placeholderPass}"></div>
<div class="uf-password-section"><div class="settings-row"><label class="settings-label">Password${_hint('For Gmail, iCloud, and Yahoo: paste your App Password (NOT your normal account password). For Migadu and Fastmail, your mailbox password usually works. Outlook / Office 365 generally requires OAuth and will not work with this password form.')}</label><input id="uf-imap-pass" class="settings-input" type="password" placeholder="${placeholderPass}"></div></div>
<div class="settings-row"><label class="settings-label">STARTTLS${_hint('Turn ON for port 143/587 to upgrade plain to TLS. Turn OFF for port 993 (IMAPS — already encrypted) or a local server with no TLS configured.')}</label><label class="admin-switch" style="margin-left:0"><input type="checkbox" id="uf-imap-starttls" checked><span class="admin-slider"></span></label></div>
<div style="font-size:11px;font-weight:600;opacity:0.6;margin:8px 0 2px;display:flex;align-items:center;gap:5px;"><svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="color:var(--accent, var(--red));flex-shrink:0;" aria-hidden="true"><line x1="22" y1="2" x2="11" y2="13"/><polygon points="22 2 15 22 11 13 2 9 22 2"/></svg>SMTP (Sending) <span style="font-weight:normal;opacity:0.7"> optional, leave blank for read-only</span></div>
<div class="settings-row"><label class="settings-label">Host${_hint('Your outgoing-mail server, e.g. smtp.gmail.com. Leave blank to make this account read-only.')}</label><input id="uf-smtp-host" class="settings-input" placeholder="smtp.example.com"></div>
@@ -4491,6 +4545,16 @@ async function initUnifiedIntegrations() {
</div>`;
};
// Show/hide the OAuth section and password fields based on provider selection.
function _syncOauthUI(providerKey) {
const p = PROVIDERS[providerKey];
const isOauth = !!(p && p.oauth);
el('uf-oauth-section').style.display = isOauth ? '' : 'none';
formEl.querySelectorAll('.uf-password-section').forEach(r => {
r.style.display = isOauth ? 'none' : '';
});
}
// Custom dropdown wire-up — the native <select> stays in the DOM as the
// data source and accessibility target, but the visible UI is a button +
// popup so each provider row can render with its SVG logo. Selecting an
@@ -4547,6 +4611,7 @@ async function initUnifiedIntegrations() {
el('uf-email-provider').addEventListener('change', (e) => {
const key = e.target.value;
_renderProviderNote(key);
_syncOauthUI(key);
const p = PROVIDERS[key];
if (!p) return;
el('uf-imap-host').value = p.imap.host;
@@ -4562,6 +4627,23 @@ async function initUnifiedIntegrations() {
}
});
// Init OAuth UI for accounts already connected via OAuth.
if (existing && existing.oauth_provider === 'google') _syncOauthUI('google_workspace');
// "Connect with Google" — save the account first, then redirect to OAuth.
el('uf-oauth-btn').addEventListener('click', async () => {
const body = _collectBody();
if (!body.name) body.name = body.from_address;
if (!body.name) { el('uf-email-msg').textContent = 'Enter a Name or Email first'; el('uf-email-msg').style.color = 'var(--red)'; return; }
const url = isEdit ? `/api/email/accounts/${editId}` : '/api/email/accounts';
const method = isEdit ? 'PUT' : 'POST';
const r = await fetch(url, { method, credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
const d = await r.json();
if (!(d.ok || d.id)) { el('uf-email-msg').textContent = d.error || 'Save failed'; el('uf-email-msg').style.color = 'var(--red)'; return; }
const accId = isEdit ? editId : d.id;
window.location.href = `/api/email/oauth/google/authorize?account_id=${encodeURIComponent(accId)}`;
});
// "Same as IMAP" toggle — hide the SMTP creds rows when on.
const _syncSmtpSame = () => {
const same = el('uf-smtp-same').checked;
@@ -4574,6 +4656,7 @@ async function initUnifiedIntegrations() {
if (existing) {
el('uf-email-name').value = existing.name || '';
el('uf-email-from').value = existing.from_address || '';
el('uf-display-name').value = existing.display_name || '';
el('uf-imap-host').value = existing.imap_host || '';
el('uf-imap-port').value = existing.imap_port || 993;
el('uf-imap-user').value = existing.imap_user || '';
@@ -4622,6 +4705,7 @@ async function initUnifiedIntegrations() {
const body = {
name: el('uf-email-name').value.trim(),
from_address: el('uf-email-from').value.trim(),
display_name: el('uf-display-name').value.trim(),
imap_host: el('uf-imap-host').value.trim(),
imap_port: parseInt(el('uf-imap-port').value) || 993,
imap_user: el('uf-imap-user').value.trim(),
@@ -5650,6 +5734,40 @@ export function close() {
}
}
// Handle redirect back from Google OAuth2 — open settings to integrations and show status.
(function _handleOauthRedirect() {
const sp = new URLSearchParams(window.location.search);
if (!sp.has('email_oauth_success') && !sp.has('email_oauth_error')) return;
// Strip params from URL without a page reload.
const clean = window.location.pathname + window.location.hash;
window.history.replaceState(null, '', clean);
const success = sp.has('email_oauth_success');
const errMsg = sp.get('email_oauth_error') || '';
// Open settings → integrations after the app has initialised.
function _tryOpen() {
if (window.settingsModule && typeof window.settingsModule.open === 'function') {
window.settingsModule.open('integrations');
// Brief toast-style banner.
const banner = document.createElement('div');
banner.textContent = success
? '✓ Google account connected — email is ready'
: `Google OAuth failed: ${errMsg || 'unknown error'}`;
Object.assign(banner.style, {
position: 'fixed', bottom: '24px', left: '50%', transform: 'translateX(-50%)',
background: success ? 'var(--accent, #50fa7b)' : 'var(--red, #ff5555)',
color: '#000', padding: '8px 18px', borderRadius: '6px', fontSize: '12px',
fontWeight: '600', zIndex: '99999', pointerEvents: 'none',
boxShadow: '0 2px 12px rgba(0,0,0,0.3)',
});
document.body.appendChild(banner);
setTimeout(() => banner.remove(), 4000);
} else {
setTimeout(_tryOpen, 100);
}
}
_tryOpen();
})();
const settingsModule = { open, close, initIntegrations, initUnifiedIntegrations, syncAdminVisibility, refreshAiModelEndpoints };
+104
View File
@@ -0,0 +1,104 @@
from src import ai_interaction
class _GenerationResponse:
status_code = 200
text = ""
def __init__(self, image_url):
self._image_url = image_url
def json(self):
return {"data": [{"url": self._image_url}]}
class _DownloadResponse:
status_code = 503
content = b""
def _patch_generation(monkeypatch, image_url):
async def _post(self, url, json, headers):
return _GenerationResponse(image_url)
class _AsyncClient:
def __init__(self, *args, **kwargs):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *exc):
return False
post = _post
import httpx
import src.settings as settings
monkeypatch.setattr(settings, "load_settings", lambda: {})
monkeypatch.setattr(httpx, "AsyncClient", _AsyncClient)
monkeypatch.setattr(
ai_interaction,
"_resolve_model",
lambda model_spec, owner=None: (
"https://api.openai.example/v1/chat/completions",
"dall-e-3",
{"Authorization": "Bearer test"},
),
)
async def test_generate_image_validates_provider_url_before_download(monkeypatch):
import httpx
import src.url_safety as url_safety
provider_url = "https://images.example.com/generated.png?sig=abc"
events = []
_patch_generation(monkeypatch, provider_url)
def _check_outbound_url(url, *, block_private=False):
events.append(("check", url, block_private))
return True, "ok"
def _get(url, *, timeout):
events.append(("get", url, timeout))
return _DownloadResponse()
monkeypatch.setattr(url_safety, "check_outbound_url", _check_outbound_url)
monkeypatch.setattr(httpx, "get", _get)
result = await ai_interaction.do_generate_image("draw a chair\ndall-e-3")
assert result["image_url"] == provider_url
assert events == [
("check", provider_url, False),
("get", provider_url, 60),
]
async def test_generate_image_rejects_unsafe_provider_url_without_download(monkeypatch):
import httpx
import src.url_safety as url_safety
unsafe_url = "http://169.254.169.254/latest/meta-data"
events = []
_patch_generation(monkeypatch, unsafe_url)
def _check_outbound_url(url, *, block_private=False):
events.append(("check", url, block_private))
return False, "link-local address blocked (SSRF metadata risk): 169.254.169.254"
def _get(url, *, timeout):
raise AssertionError("unsafe provider image URL must not be downloaded")
monkeypatch.setattr(url_safety, "check_outbound_url", _check_outbound_url)
monkeypatch.setattr(httpx, "get", _get)
result = await ai_interaction.do_generate_image("draw a chair\ndall-e-3")
assert result["error"] == (
"Image API returned unsafe image URL: "
"link-local address blocked (SSRF metadata risk): 169.254.169.254"
)
assert events == [("check", unsafe_url, False)]
+3
View File
@@ -219,6 +219,9 @@ class _WebhookManager:
async def fire(self, event, payload):
return None
def fire_and_forget(self, event, payload):
return None
def _install_sync_chat_stubs(monkeypatch):
# FastAPI checks for python_multipart at import time when Form is used;
+74
View File
@@ -502,3 +502,77 @@ def test_delete_token_owner_check_skipped_when_auth_disabled(monkeypatch, token_
resp = delete_token(request=req, token_id="tok123")
assert resp == {"status": "deleted"}
fake_session.delete.assert_called_once_with(fake_token)
# ---------------------------------------------------------------------------
# 7. PATCH /api/tokens/{id} — non-object JSON bodies must not 500
# ---------------------------------------------------------------------------
def test_update_token_with_array_body_does_not_500(monkeypatch, token_routes_mod):
"""PATCH body of [] must be normalised to {} and not raise."""
monkeypatch.setenv("AUTH_ENABLED", "true")
mod = token_routes_mod
token = SimpleNamespace(
id="tok123", name="original", owner="alice",
token_prefix="ody_orig", scopes="email:read", is_active=True,
)
fake_session = MagicMock()
fake_session.query.return_value.filter.return_value.first.return_value = token
monkeypatch.setattr(mod, "get_db_session", lambda: _db_ctx(fake_session))
invalidator = MagicMock()
req = _patch_request(invalidator, [])
update_token = _get_handler(mod, "PATCH", "/tokens/{token_id}")
resp = asyncio.run(update_token(request=req, token_id="tok123"))
# Name and scopes must be unchanged — payload was normalised to {}
assert token.name == "original"
assert token.scopes == "email:read"
assert resp["name"] == "original"
def test_update_token_with_null_body_does_not_500(monkeypatch, token_routes_mod):
"""PATCH body of null must be normalised to {} and not raise."""
monkeypatch.setenv("AUTH_ENABLED", "true")
mod = token_routes_mod
token = SimpleNamespace(
id="tok123", name="original", owner="alice",
token_prefix="ody_orig", scopes="chat", is_active=True,
)
fake_session = MagicMock()
fake_session.query.return_value.filter.return_value.first.return_value = token
monkeypatch.setattr(mod, "get_db_session", lambda: _db_ctx(fake_session))
invalidator = MagicMock()
req = _patch_request(invalidator, None)
update_token = _get_handler(mod, "PATCH", "/tokens/{token_id}")
resp = asyncio.run(update_token(request=req, token_id="tok123"))
assert token.name == "original"
assert token.scopes == "chat"
def test_update_token_normal_object_still_works(monkeypatch, token_routes_mod):
"""Normal dict payload continues to update fields as before."""
monkeypatch.setenv("AUTH_ENABLED", "true")
mod = token_routes_mod
token = SimpleNamespace(
id="tok123", name="original", owner="alice",
token_prefix="ody_orig", scopes="email:read", is_active=True,
)
fake_session = MagicMock()
fake_session.query.return_value.filter.return_value.first.return_value = token
monkeypatch.setattr(mod, "get_db_session", lambda: _db_ctx(fake_session))
invalidator = MagicMock()
req = _patch_request(invalidator, {"name": "updated"})
update_token = _get_handler(mod, "PATCH", "/tokens/{token_id}")
resp = asyncio.run(update_token(request=req, token_id="tok123"))
assert token.name == "updated"
assert resp["name"] == "updated"
invalidator.assert_called_once()
@@ -0,0 +1,88 @@
"""do_manage_calendar must honour abbreviated reminder phrasings like "mins"/"hrs".
`_reminder_minutes` parsed the reminder offset with regexes anchored on
`(?:m|min|minute|minutes)\b` / `(?:h|hr|hour|hours)\b`. The trailing `\b`
made the very common plural abbreviations "mins" and "hrs" fail to match
(after "min" the next char "s" is a word char, so no boundary), so a request
like ``reminder_minutes: "5 mins"`` silently produced no reminder at all
even though the sibling duration parser (no `\b`) already accepted them.
"""
import json
import sys
import uuid
import pytest
from tests.helpers.import_state import clear_fake_database_modules
from tests.helpers.sqlite_db import make_temp_sqlite
clear_fake_database_modules()
import core.database as cdb
from core.database import Note
_TS, _ENGINE, _TMPDB = make_temp_sqlite(cdb.Base.metadata)
@pytest.fixture(autouse=True)
def _bind_temp_db(monkeypatch):
monkeypatch.setitem(sys.modules, "core.database", cdb)
parent = sys.modules.get("core")
if parent is not None:
monkeypatch.setattr(parent, "database", cdb, raising=False)
monkeypatch.setattr(cdb, "SessionLocal", _TS)
yield
async def _create_with_reminder(reminder, owner):
from src.tool_implementations import do_manage_calendar
payload = {
"action": "create_event",
"summary": "Dentist",
# Far-future so the reminder is never "already passed".
"dtstart": "2030-01-01T10:00:00",
"reminder_minutes": reminder,
}
return await do_manage_calendar(json.dumps(payload), owner=owner)
@pytest.mark.parametrize("reminder,expected", [
("5 mins", 5),
("10 mins", 10),
("2 hrs", 120),
("1 hr", 60),
("15 minutes", 15), # regression: long form still works
("30m", 30), # regression: bare unit still works
])
async def test_reminder_minutes_accepts_abbreviations(reminder, expected):
owner = "tester-" + uuid.uuid4().hex[:6]
res = await _create_with_reminder(reminder, owner)
assert res.get("exit_code") == 0, res
assert f"reminder {expected} min before" in res.get("response", ""), res
db = _TS()
try:
note = (
db.query(Note)
.filter(Note.owner == owner, Note.title == "Reminder: Dentist")
.first()
)
assert note is not None, "reminder note should have been created"
finally:
db.close()
async def test_no_reminder_when_offset_absent():
owner = "tester-" + uuid.uuid4().hex[:6]
from src.tool_implementations import do_manage_calendar
payload = {
"action": "create_event",
"summary": "No Reminder Event",
"dtstart": "2030-02-01T10:00:00",
}
res = await do_manage_calendar(json.dumps(payload), owner=owner)
assert res.get("exit_code") == 0, res
assert "reminder set" not in res.get("response", ""), res
+7 -7
View File
@@ -30,7 +30,7 @@ class _Session:
def test_allowed_models_legacy_empty_list_remains_unrestricted(monkeypatch):
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
_enforce_chat_privileges(
_Request({"allowed_models": [], "max_messages_per_day": 0}),
@@ -39,7 +39,7 @@ def test_allowed_models_legacy_empty_list_remains_unrestricted(monkeypatch):
def test_allowed_models_explicit_empty_restricted_list_blocks_all_models(monkeypatch):
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
with pytest.raises(HTTPException) as exc:
_enforce_chat_privileges(
@@ -56,7 +56,7 @@ def test_allowed_models_explicit_empty_restricted_list_blocks_all_models(monkeyp
def test_allowed_models_nonempty_list_still_restricts_without_new_flag(monkeypatch):
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
_enforce_chat_privileges(
_Request({"allowed_models": ["provider/model-a"], "max_messages_per_day": 0}),
@@ -70,7 +70,7 @@ def test_allowed_models_nonempty_list_still_restricts_without_new_flag(monkeypat
def test_no_restriction_allows_any_model(monkeypatch):
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
privs = {"allowed_models": [], "block_all_models": False, "max_messages_per_day": 0}
_enforce_chat_privileges(_Request(privs), _Session("provider/model-a"))
@@ -78,7 +78,7 @@ def test_no_restriction_allows_any_model(monkeypatch):
def test_specific_allowlist_blocks_models_outside_it(monkeypatch):
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
privs = {
"allowed_models": ["gpt-4"],
@@ -92,7 +92,7 @@ def test_specific_allowlist_blocks_models_outside_it(monkeypatch):
def test_block_all_models_blocks_regardless_of_allowed_models_contents(monkeypatch):
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "alice")
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "alice")
# Even if allowed_models contains entries, block_all_models wins.
privs = {
@@ -111,7 +111,7 @@ def test_block_all_models_blocks_regardless_of_allowed_models_contents(monkeypat
def test_admin_user_is_never_blocked(monkeypatch):
from core.auth import ADMIN_PRIVILEGES
monkeypatch.setattr("routes.chat_helpers.get_current_user", lambda request: "admin")
monkeypatch.setattr("routes.chat_helpers.effective_user", lambda request: "admin")
class _AdminAuthManager:
def get_privileges(self, username):
+70
View File
@@ -0,0 +1,70 @@
"""Check-in calendar digest must be scoped to the task owner.
The digest query selected CalendarEvent with no owner scope, so a scheduled
check-in for one user pulled EVERY user's calendar events (summaries,
locations) into their digest a cross-tenant leak. Ownership lives on
CalendarCal.owner; the query must join it, like routes/calendar_routes.
"""
import tempfile
import uuid
from datetime import datetime
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import NullPool
import core.database as cdb
from core.database import CalendarEvent, CalendarCal
from src.task_scheduler import _checkin_calendar_events
_TMPDB = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
_ENGINE = create_engine(f"sqlite:///{_TMPDB.name}", connect_args={"check_same_thread": False}, poolclass=NullPool)
cdb.Base.metadata.create_all(_ENGINE)
_TS = sessionmaker(bind=_ENGINE, autoflush=False, autocommit=False)
def _seed():
db = _TS()
try:
db.query(CalendarEvent).delete(); db.query(CalendarCal).delete()
db.add(CalendarCal(id="calA", owner="alice", name="A"))
db.add(CalendarCal(id="calB", owner="bob", name="B"))
db.add(CalendarEvent(uid="a1", calendar_id="calA", summary="Alice mtg",
dtstart=datetime(2026, 6, 10, 9, 0),
dtend=datetime(2026, 6, 10, 10, 0), status="confirmed"))
db.add(CalendarEvent(uid="b1", calendar_id="calB", summary="Bob secret",
dtstart=datetime(2026, 6, 10, 10, 0),
dtend=datetime(2026, 6, 10, 11, 0), status="confirmed"))
db.commit()
finally:
db.close()
def test_digest_only_returns_owner_events():
_seed()
db = _TS()
try:
s, e = datetime(2026, 6, 1), datetime(2026, 6, 30)
alice = _checkin_calendar_events(db, "alice", s, e)
assert [ev.summary for ev in alice] == ["Alice mtg"] # not Bob's
bob = _checkin_calendar_events(db, "bob", s, e)
assert [ev.summary for ev in bob] == ["Bob secret"]
finally:
db.close()
def test_cancelled_excluded_and_window_respected():
_seed()
db = _TS()
try:
db2 = _TS()
db2.add(CalendarEvent(uid="a2", calendar_id="calA", summary="cancelled",
dtstart=datetime(2026, 6, 11),
dtend=datetime(2026, 6, 11, 1, 0), status="cancelled"))
db2.commit(); db2.close()
s, e = datetime(2026, 6, 1), datetime(2026, 6, 30)
out = _checkin_calendar_events(db, "alice", s, e)
assert "cancelled" not in [ev.summary for ev in out]
finally:
db.close()
+50
View File
@@ -7,12 +7,39 @@ in ``remoteHost`` would be injected into that command.
These pin validation on the host/port before they reach the ssh string, matching
the validators the rest of the cookbook routes already apply.
"""
import asyncio
import pytest
from fastapi import HTTPException
from starlette.requests import Request
import routes.codex_routes as codex_routes
def _route_endpoint(path: str, method: str):
router = codex_routes.setup_codex_routes()
for route in router.routes:
if route.path == path and method in route.methods:
return route.endpoint
raise AssertionError(f"{method} {path} route not found")
def _launch_request() -> Request:
request = Request(
{
"type": "http",
"method": "POST",
"path": "/api/codex/cookbook/adopt",
"headers": [],
"state": {},
}
)
request.state.api_token = True
request.state.api_token_owner = "alice"
request.state.api_token_scopes = ["cookbook:launch"]
return request
def test_rejects_remote_host_with_shell_metacharacters():
task = {"remoteHost": "box; rm -rf ~", "sshPort": ""}
with pytest.raises(HTTPException) as exc:
@@ -47,3 +74,26 @@ def test_default_ssh_port_omits_flag():
)
assert host == "box"
assert port_flag == ""
def test_adopt_rejects_ssh_option_host_before_shell(monkeypatch):
calls = []
async def fail_if_shell_runs(*args, **kwargs):
calls.append((args, kwargs))
raise RuntimeError("shell should not run for invalid host")
monkeypatch.setattr(asyncio, "create_subprocess_shell", fail_if_shell_runs)
endpoint = _route_endpoint("/api/codex/cookbook/adopt", "POST")
body = {
"tmux_session": "serve_abc123",
"model": "org/model",
"host": "-oProxyCommand=sh",
}
with pytest.raises(HTTPException) as exc:
asyncio.run(endpoint(_launch_request(), body))
assert exc.value.status_code == 400
assert calls == []
+32 -2
View File
@@ -13,6 +13,9 @@ import json
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
from fastapi import HTTPException
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
# core.database instantiates SQLAlchemy declarative classes at import time, which
@@ -225,12 +228,34 @@ def test_models_route_scopes_api_token_to_token_owner(monkeypatch):
endpoints = _call_models_route(
monkeypatch,
rows,
_request(api_token=True, api_token_owner="alice", current_user="api"),
_request(
api_token=True,
api_token_owner="alice",
api_token_scopes=["chat"],
current_user="api",
),
)
assert _endpoint_names(endpoints) == ["alice-endpoint", "shared-endpoint"]
def test_models_route_rejects_api_token_without_chat_scope(monkeypatch):
monkeypatch.setattr(companion_routes, "get_current_user", lambda request: "api")
with pytest.raises(HTTPException) as exc:
_models_route()(
_request(
api_token=True,
api_token_owner="alice",
api_token_scopes=["todos:read"],
current_user="api",
)
)
assert exc.value.status_code == 403
assert "chat scope" in exc.value.detail
def test_models_route_unresolved_owner_returns_only_shared_rows(monkeypatch):
rows = [
_ep(1, "alice-endpoint", "alice"),
@@ -242,7 +267,12 @@ def test_models_route_unresolved_owner_returns_only_shared_rows(monkeypatch):
endpoints = _call_models_route(
monkeypatch,
rows,
_request(api_token=True, api_token_owner=None, current_user="api"),
_request(
api_token=True,
api_token_owner=None,
api_token_scopes=["chat"],
current_user="api",
),
)
assert _endpoint_names(endpoints) == ["shared-endpoint"]
+44
View File
@@ -786,6 +786,50 @@ def test_cached_model_scan_reports_plain_dir_gguf(tmp_path):
assert ggufs[3]["quant"] == "BF16"
def test_cached_model_scan_uses_ollama_api_before_cli_and_windows_opt_in():
script = _cached_model_scan_script()
assert "scan_ollama_api()\nscan_ollama()" in script
assert "if any(m.get('is_ollama') for m in models): return" in script
assert "os.name == 'nt'" in script
assert "ODYSSEUS_ALLOW_OLLAMA_CLI_SCAN" in script
@pytest.mark.skipif(os.name != "nt", reason="Windows Ollama CLI startup guard")
def test_cached_model_scan_does_not_launch_ollama_cli_on_windows(tmp_path):
"""Official Ollama for Windows can auto-start the tray/server on `ollama list`.
The read-only cache scanner must not invoke that CLI unless explicitly opted in.
"""
marker = tmp_path / "ollama-called.txt"
fake_ollama = tmp_path / "ollama.cmd"
fake_ollama.write_text(
"@echo off\r\n"
f'echo called>"{marker}"\r\n'
"echo NAME ID SIZE MODIFIED\r\n"
"echo local-model:latest abc 1 GB now\r\n",
encoding="utf-8",
)
empty_home = tmp_path / "home"
empty_home.mkdir()
scan_py = tmp_path / "scan_cache.py"
scan_py.write_text(_cached_model_scan_script(), encoding="utf-8")
env = dict(os.environ)
env["PATH"] = str(tmp_path) + os.pathsep + env.get("PATH", "")
env["HOME"] = str(empty_home)
env.pop("ODYSSEUS_ALLOW_OLLAMA_CLI_SCAN", None)
proc = subprocess.run(
[sys.executable, str(scan_py)],
check=True,
capture_output=True,
text=True,
env=env,
)
assert marker.exists() is False
assert all(m.get("backend") != "ollama" for m in json.loads(proc.stdout))
def test_cached_model_scan_uses_huggingface_cache_env(tmp_path):
"""Docker recreates can leave the persisted HF cache outside HOME.
The Serve scanner should honor the cache env path instead of only ~/.cache.
+50
View File
@@ -0,0 +1,50 @@
import json
import pytest
from src import cookbook_serve_lifecycle as lifecycle
@pytest.mark.asyncio
async def test_tick_persists_only_successfully_stopped_serves(tmp_path, monkeypatch):
state_path = tmp_path / "cookbook_state.json"
state_path.write_text(
json.dumps({
"tasks": [
{
"id": "stop-succeeds",
"type": "serve",
"status": "running",
"_scheduledStopAtMs": 0,
},
{
"id": "stop-fails",
"type": "serve",
"status": "running",
"_scheduledStopAtMs": 0,
},
]
}),
encoding="utf-8",
)
async def fake_stop_serve(session_id, remote_host="", ssh_port=""):
return session_id == "stop-succeeds"
async def fake_delete_endpoint(task):
return None
monkeypatch.setattr(lifecycle, "COOKBOOK_STATE_FILE", str(state_path))
monkeypatch.setattr(lifecycle, "_stop_serve", fake_stop_serve)
monkeypatch.setattr(lifecycle, "_delete_endpoint_for_task", fake_delete_endpoint)
await lifecycle._tick()
tasks = {
task["id"]: task
for task in json.loads(state_path.read_text(encoding="utf-8"))["tasks"]
}
assert tasks["stop-succeeds"]["status"] == "stopped"
assert tasks["stop-succeeds"]["_scheduledStopAtMs"] is None
assert tasks["stop-fails"]["status"] == "running"
assert tasks["stop-fails"]["_scheduledStopAtMs"] == 0
+580
View File
@@ -0,0 +1,580 @@
"""Tests for the Google OAuth2 email helpers.
Covers the security-critical surface added for Google Workspace / .edu
IMAP/SMTP support:
- `make_oauth_state` / `verify_oauth_state` HMAC-signed OAuth state so the
callback can't be CSRF'd or have its account_id/owner tampered with.
- `_smtp_ready` an OAuth account (no stored password) must still count as
send-capable; a host+user-only account without password or OAuth must not.
- `_xoauth2_raw` / `_xoauth2_bytes` SASL XOAUTH2 framing for SMTP/IMAP.
- `_refresh_google_token` token refresh stores result encrypted; failure is
silent (no token/secret in logs or return value).
- `_get_valid_google_token` uses cached token when fresh; calls refresh when
expired.
- `google_oauth_callback` (real route) invalid/tampered/missing state and
provider errors return generic redirects with no PII; owner mismatch refuses
the token write; a valid owner writes encrypted tokens only to the intended
account.
- `list_email_accounts` (real route) exposes OAuth status but never token
values.
- `_imap_connect` password accounts use login(); OAuth accounts use XOAUTH2.
Route tests pull the live endpoint out of `setup_email_routes()` and call it
directly they pin the real handler, not a re-implementation. The ASGI app is
not booted; outbound HTTP is mocked and the DB is an isolated in-memory SQLite.
"""
import base64
import json
import time
import unittest.mock as mock
import pytest
# ── OAuth state signing ──────────────────────────────────────────
def test_oauth_state_round_trips_account_and_owner():
from routes.email_helpers import make_oauth_state, verify_oauth_state
state = make_oauth_state("acct-123", "user@example.com")
payload = verify_oauth_state(state)
assert payload is not None
assert payload["a"] == "acct-123"
assert payload["o"] == "user@example.com"
assert payload["n"] # nonce present
def test_oauth_state_nonce_is_unique_per_call():
from routes.email_helpers import make_oauth_state, verify_oauth_state
a = verify_oauth_state(make_oauth_state("acct", "o"))
b = verify_oauth_state(make_oauth_state("acct", "o"))
assert a["n"] != b["n"]
def test_oauth_state_rejects_tampered_account_id():
from routes.email_helpers import make_oauth_state, verify_oauth_state
state = make_oauth_state("acct-123", "user@example.com")
decoded = base64.urlsafe_b64decode(state.encode()).decode()
payload_str, sig = decoded.rsplit("|", 1)
payload = json.loads(payload_str)
payload["a"] = "evil-acct" # attacker swaps the target account
forged = base64.urlsafe_b64encode(
(json.dumps(payload, separators=(",", ":")) + "|" + sig).encode()
).decode()
assert verify_oauth_state(forged) is None
def test_oauth_state_rejects_forged_signature():
from routes.email_helpers import make_oauth_state, verify_oauth_state
state = make_oauth_state("acct-123", "user@example.com")
decoded = base64.urlsafe_b64decode(state.encode()).decode()
payload_str, _ = decoded.rsplit("|", 1)
forged = base64.urlsafe_b64encode((payload_str + "|" + "deadbeef" * 8).encode()).decode()
assert verify_oauth_state(forged) is None
@pytest.mark.parametrize("garbage", ["", "not-base64-at-all", "###", "a|b|c"])
def test_oauth_state_rejects_garbage(garbage):
from routes.email_helpers import verify_oauth_state
assert verify_oauth_state(garbage) is None
# ── _smtp_ready: OAuth accounts have no password but can still send ──
def test_smtp_ready_true_for_oauth_account_without_password():
from routes.email_routes import _smtp_ready
cfg = {
"smtp_host": "smtp.gmail.com",
"smtp_user": "me@nyu.edu",
"smtp_password": "",
"oauth_provider": "google",
}
assert _smtp_ready(cfg) is True
def test_smtp_ready_true_for_password_account():
from routes.email_routes import _smtp_ready
cfg = {
"smtp_host": "smtp.example.com",
"smtp_user": "me@example.com",
"smtp_password": "app-password",
"oauth_provider": "",
}
assert _smtp_ready(cfg) is True
def test_smtp_ready_false_without_password_or_oauth():
from routes.email_routes import _smtp_ready
cfg = {
"smtp_host": "smtp.example.com",
"smtp_user": "me@example.com",
"smtp_password": "",
"oauth_provider": "",
}
assert _smtp_ready(cfg) is False
def test_smtp_ready_false_without_host():
from routes.email_routes import _smtp_ready
cfg = {"smtp_host": "", "smtp_user": "me@x.com", "oauth_provider": "google"}
assert _smtp_ready(cfg) is False
# ── XOAUTH2 SASL framing ─────────────────────────────────────────
def test_xoauth2_raw_is_unencoded_sasl_frame():
from routes.email_helpers import _xoauth2_raw
assert _xoauth2_raw("me@nyu.edu", "tok123") == "user=me@nyu.edu\x01auth=Bearer tok123\x01\x01"
def test_xoauth2_bytes_is_raw_frame_encoded():
from routes.email_helpers import _xoauth2_bytes
assert _xoauth2_bytes("me@nyu.edu", "tok123") == b"user=me@nyu.edu\x01auth=Bearer tok123\x01\x01"
# ── Helpers for in-memory DB fixtures ────────────────────────────
def _make_db():
"""Return (Session, SessionFactory) backed by an isolated in-memory SQLite DB.
Used to test DB-touching helpers without the real database.
The factory lets tests open a fresh session after the helper closes its own.
"""
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from core.database import Base
engine = create_engine("sqlite:///:memory:", connect_args={"check_same_thread": False})
Base.metadata.create_all(engine)
Factory = sessionmaker(bind=engine)
return Factory(), Factory
def _make_account(session, account_id="acct-1", owner="alice", **kwargs):
"""Insert a minimal EmailAccount row and return it."""
from core.database import EmailAccount
row = EmailAccount(
id=account_id,
owner=owner,
name=kwargs.get("name", "Test"),
from_address=kwargs.get("from_address", "test@example.com"),
imap_host=kwargs.get("imap_host", "imap.gmail.com"),
imap_port=kwargs.get("imap_port", 993),
imap_user=kwargs.get("imap_user", "test@example.com"),
smtp_host=kwargs.get("smtp_host", "smtp.gmail.com"),
smtp_port=kwargs.get("smtp_port", 587),
smtp_user=kwargs.get("smtp_user", "test@example.com"),
)
for k, v in kwargs.items():
if hasattr(row, k):
setattr(row, k, v)
session.add(row)
session.commit()
return row
# ── Token encryption at rest ─────────────────────────────────────
def test_refresh_token_stored_encrypted_not_raw():
"""_refresh_google_token must encrypt the new access token before writing it
to the DB storing the raw token string would expose credentials at rest."""
from src.secret_storage import encrypt as _enc, decrypt as _dec
from core.database import EmailAccount
raw_token = "ya29.test_access_token_raw"
db, Factory = _make_db()
_make_account(db, account_id="acct-r", owner="bob",
oauth_refresh_token=_enc("refresh-tok-xyz"))
db.close()
fake_resp = mock.MagicMock()
fake_resp.raise_for_status = mock.MagicMock()
fake_resp.json.return_value = {"access_token": raw_token, "expires_in": 3600}
with mock.patch("httpx.post", return_value=fake_resp), \
mock.patch("core.database.SessionLocal", Factory), \
mock.patch("routes.email_helpers.os.environ.get", side_effect=lambda k, d="": {
"GOOGLE_OAUTH_CLIENT_ID": "cid", "GOOGLE_OAUTH_CLIENT_SECRET": "csec"
}.get(k, d)):
from routes.email_helpers import _refresh_google_token
result = _refresh_google_token("acct-r")
verify_db = Factory()
row = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-r").first()
stored = row.oauth_access_token
verify_db.close()
assert result == raw_token, "function should return the plain access token to callers"
assert stored != raw_token, "raw token must not be stored directly in the DB"
assert _dec(stored) == raw_token, "stored value must decrypt back to the raw token"
def test_refresh_stores_encrypted_expiry_not_token():
"""oauth_token_expiry stores only a timestamp, never the token value."""
from src.secret_storage import encrypt as _enc
from core.database import EmailAccount
db, Factory = _make_db()
_make_account(db, account_id="acct-e", owner="bob",
oauth_refresh_token=_enc("ref-tok"))
db.close()
fake_resp = mock.MagicMock()
fake_resp.raise_for_status = mock.MagicMock()
fake_resp.json.return_value = {"access_token": "ya29.secret", "expires_in": 3600}
with mock.patch("httpx.post", return_value=fake_resp), \
mock.patch("core.database.SessionLocal", Factory), \
mock.patch("routes.email_helpers.os.environ.get", side_effect=lambda k, d="": {
"GOOGLE_OAUTH_CLIENT_ID": "cid", "GOOGLE_OAUTH_CLIENT_SECRET": "csec"
}.get(k, d)):
from routes.email_helpers import _refresh_google_token
_refresh_google_token("acct-e")
verify_db = Factory()
row = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-e").first()
expiry = row.oauth_token_expiry
verify_db.close()
assert "ya29" not in (expiry or ""), \
"token_expiry must be a timestamp, not the token string"
# ── Real OAuth callback route ─────────────────────────────────────
#
# These pull the actual google_oauth_callback endpoint out of the router and
# invoke it — they pin the real route's behaviour, not a re-implementation, so
# they fail if the ownership/state guards are ever removed or weakened.
def _callback_endpoint():
"""Return the live google_oauth_callback endpoint from the email router."""
from routes.email_routes import setup_email_routes
router = setup_email_routes()
for route in router.routes:
if route.path == "/api/email/oauth/google/callback" and "GET" in getattr(route, "methods", set()):
return route.endpoint
raise AssertionError("google_oauth_callback route not found")
class _FakeRequest:
"""Minimal stand-in for starlette Request — the callback only reads headers."""
headers = {"host": "localhost:7000"}
def _location(resp):
"""Pull the redirect target out of a RedirectResponse."""
return resp.headers["location"]
@pytest.mark.asyncio
async def test_callback_missing_code_returns_generic_error():
"""No `code` query param → generic error redirect, with no account id, owner,
or state echoed back into the URL."""
from routes.email_helpers import make_oauth_state
callback = _callback_endpoint()
state = make_oauth_state("acct-1", "alice")
resp = await callback(code=None, state=state, error=None, request=_FakeRequest())
loc = _location(resp)
assert "email_oauth_error=missing_code" in loc
assert "acct-1" not in loc, "account id must not appear in redirect URL"
assert "alice" not in loc, "owner must not appear in redirect URL"
@pytest.mark.asyncio
async def test_callback_provider_error_returns_generic_error():
"""An `error` from Google → generic error redirect, no raw provider text."""
callback = _callback_endpoint()
resp = await callback(code=None, state=None, error="access_denied", request=_FakeRequest())
loc = _location(resp)
assert "email_oauth_error=google_error" in loc
assert "access_denied" not in loc, "raw provider error must not leak into redirect"
@pytest.mark.asyncio
async def test_callback_tampered_state_returns_generic_error_no_leak():
"""Tampered/invalid state → invalid_state redirect; the auth code and any
token must never appear in the redirect URL."""
callback = _callback_endpoint()
resp = await callback(code="4/secret-auth-code", state="not-a-valid-state",
error=None, request=_FakeRequest())
loc = _location(resp)
assert "email_oauth_error=invalid_state" in loc
assert "4/secret-auth-code" not in loc, "auth code must not leak into redirect"
assert "token" not in loc
@pytest.mark.asyncio
async def test_callback_owner_mismatch_does_not_write_tokens():
"""A signed, valid state whose owner does not match the target account's
owner must NOT write tokens this blocks one authenticated user from
binding their Google account onto another user's mailbox row.
"""
from routes.email_helpers import make_oauth_state
from core.database import EmailAccount
db, Factory = _make_db()
_make_account(db, account_id="acct-x", owner="alice")
db.close()
# Token-exchange + userinfo would succeed — the point is the ownership gate
# rejects the write *before* trusting them.
token_resp = mock.MagicMock()
token_resp.raise_for_status = mock.MagicMock()
token_resp.json.return_value = {"access_token": "ya29.attacker", "refresh_token": "r", "expires_in": 3600}
userinfo_resp = mock.MagicMock()
userinfo_resp.is_success = True
userinfo_resp.json.return_value = {"email": "bob@evil.com", "name": "Bob"}
# State is genuinely signed, but for owner "bob" — not the row owner "alice".
state = make_oauth_state("acct-x", "bob")
with mock.patch("httpx.post", return_value=token_resp), \
mock.patch("httpx.get", return_value=userinfo_resp), \
mock.patch("core.database.SessionLocal", Factory):
callback = _callback_endpoint()
resp = await callback(code="4/code", state=state, error=None, request=_FakeRequest())
loc = _location(resp)
assert "email_oauth_error=ownership_error" in loc
verify_db = Factory()
row = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-x").first()
token_after = row.oauth_access_token
verify_db.close()
assert token_after is None, "no token may be written when ownership check fails"
@pytest.mark.asyncio
async def test_callback_valid_owner_writes_encrypted_tokens_to_intended_account():
"""A signed state whose owner matches the target account writes the tokens —
and only to that account, stored encrypted (raw token never persisted)."""
from routes.email_helpers import make_oauth_state
from src.secret_storage import decrypt as _dec
from core.database import EmailAccount
db, Factory = _make_db()
_make_account(db, account_id="acct-v", owner="alice", imap_host="", smtp_host="")
_make_account(db, account_id="acct-other", owner="alice") # must stay untouched
db.close()
raw_access = "ya29.legit_access_token"
raw_refresh = "1//legit_refresh_token"
token_resp = mock.MagicMock()
token_resp.raise_for_status = mock.MagicMock()
token_resp.json.return_value = {"access_token": raw_access, "refresh_token": raw_refresh, "expires_in": 3600}
userinfo_resp = mock.MagicMock()
userinfo_resp.is_success = True
userinfo_resp.json.return_value = {"email": "alice@nyu.edu", "name": "Alice"}
state = make_oauth_state("acct-v", "alice")
with mock.patch("httpx.post", return_value=token_resp), \
mock.patch("httpx.get", return_value=userinfo_resp), \
mock.patch("core.database.SessionLocal", Factory):
callback = _callback_endpoint()
resp = await callback(code="4/code", state=state, error=None, request=_FakeRequest())
assert "email_oauth_success=1" in _location(resp)
verify_db = Factory()
target = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-v").first()
other = verify_db.query(EmailAccount).filter(EmailAccount.id == "acct-other").first()
verify_db.close()
assert target.oauth_provider == "google"
assert target.oauth_access_token != raw_access, "access token must be stored encrypted"
assert _dec(target.oauth_access_token) == raw_access
assert _dec(target.oauth_refresh_token) == raw_refresh
assert other.oauth_access_token is None, "tokens must only touch the intended account"
# ── Token refresh scenarios ───────────────────────────────────────
def test_get_valid_google_token_uses_cached_when_fresh():
"""_get_valid_google_token must NOT call refresh when the stored token is
still valid (expiry - 60s buffer > now). Refresh is an outbound HTTP call
that should only happen when genuinely needed."""
from src.secret_storage import encrypt as _enc
from routes.email_helpers import _get_valid_google_token
future_expiry = str(int(time.time()) + 7200) # 2 hours from now
cfg = {
"account_id": "acct-fresh",
"oauth_access_token": _enc("ya29.fresh_token"),
"oauth_token_expiry": future_expiry,
}
with mock.patch("routes.email_helpers._refresh_google_token") as mock_refresh:
result = _get_valid_google_token("acct-fresh", cfg)
assert result == "ya29.fresh_token"
mock_refresh.assert_not_called()
def test_get_valid_google_token_refreshes_when_expired():
"""_get_valid_google_token must call refresh when the token is expired."""
from src.secret_storage import encrypt as _enc
from routes.email_helpers import _get_valid_google_token
past_expiry = str(int(time.time()) - 10) # already expired
cfg = {
"account_id": "acct-exp",
"oauth_access_token": _enc("ya29.old_token"),
"oauth_token_expiry": past_expiry,
}
with mock.patch("routes.email_helpers._refresh_google_token", return_value="ya29.new_token") as mock_refresh:
result = _get_valid_google_token("acct-exp", cfg)
mock_refresh.assert_called_once_with("acct-exp")
assert result == "ya29.new_token"
def test_refresh_failure_returns_none_no_secret_raised():
"""When the refresh HTTP call fails, _refresh_google_token must return None
silently. It must not raise an exception or surface token/secret details."""
from src.secret_storage import encrypt as _enc
db, Factory = _make_db()
_make_account(db, account_id="acct-fail", owner="dave",
oauth_refresh_token=_enc("ref-tok"))
db.close()
failing_resp = mock.MagicMock()
failing_resp.raise_for_status.side_effect = Exception("401 Unauthorized")
with mock.patch("httpx.post", return_value=failing_resp), \
mock.patch("core.database.SessionLocal", Factory), \
mock.patch("routes.email_helpers.os.environ.get", side_effect=lambda k, d="": {
"GOOGLE_OAUTH_CLIENT_ID": "cid", "GOOGLE_OAUTH_CLIENT_SECRET": "csec"
}.get(k, d)):
from routes.email_helpers import _refresh_google_token
result = _refresh_google_token("acct-fail")
assert result is None, "failed refresh must return None, not raise"
def test_refresh_without_credentials_returns_none():
"""_refresh_google_token must return None immediately when the OAuth client
credentials are not configured no DB query, no HTTP call."""
with mock.patch("routes.email_helpers.os.environ.get", return_value=""):
from routes.email_helpers import _refresh_google_token
result = _refresh_google_token("acct-any")
assert result is None
# ── Password-account regression ───────────────────────────────────
def test_imap_connect_uses_login_for_password_accounts():
"""Existing password-auth IMAP accounts must still call conn.login() and
must NOT trigger the XOAUTH2 authenticate path."""
from routes.email_helpers import _imap_connect
mock_conn = mock.MagicMock()
# _imap_connect calls _get_email_config internally — mock it to return our cfg.
cfg = {
"imap_host": "imap.gmail.com",
"imap_port": 993,
"imap_starttls": False,
"imap_user": "me@gmail.com",
"imap_password": "app-password-xyz",
"oauth_provider": "",
"account_id": "acct-pw",
}
with mock.patch("routes.email_helpers._open_imap_connection", return_value=mock_conn), \
mock.patch("routes.email_helpers._get_email_config", return_value=cfg):
_imap_connect("acct-pw", owner="alice")
mock_conn.login.assert_called_once_with("me@gmail.com", "app-password-xyz")
mock_conn.authenticate.assert_not_called()
def test_imap_connect_uses_xoauth2_for_oauth_accounts():
"""OAuth accounts must call conn.authenticate('XOAUTH2', ...) and must NOT
call conn.login() which would fail without a password."""
from routes.email_helpers import _imap_connect
from src.secret_storage import encrypt as _enc
mock_conn = mock.MagicMock()
future_expiry = str(int(time.time()) + 7200)
cfg = {
"imap_host": "imap.gmail.com",
"imap_port": 993,
"imap_starttls": False,
"imap_user": "me@nyu.edu",
"imap_password": "",
"oauth_provider": "google",
"account_id": "acct-oauth",
"oauth_access_token": _enc("ya29.live_token"),
"oauth_token_expiry": future_expiry,
}
with mock.patch("routes.email_helpers._open_imap_connection", return_value=mock_conn), \
mock.patch("routes.email_helpers._get_email_config", return_value=cfg):
_imap_connect("acct-oauth", owner="alice")
mock_conn.authenticate.assert_called_once()
assert mock_conn.authenticate.call_args[0][0] == "XOAUTH2"
mock_conn.login.assert_not_called()
@pytest.mark.asyncio
async def test_account_list_response_does_not_expose_token_values():
"""The /accounts list route is the client-facing account inventory. It must
expose `oauth_provider` (so the UI can show OAuth status) but never the
access/refresh token values, encrypted or otherwise only boolean
has_*_password flags and the provider name."""
from routes.email_routes import setup_email_routes
from src.secret_storage import encrypt as _enc
raw_access = "ya29.super_secret_access_token"
raw_refresh = "1//super_secret_refresh_token"
db, Factory = _make_db()
_make_account(db, account_id="acct-list", owner="alice",
oauth_provider="google",
oauth_access_token=_enc(raw_access),
oauth_refresh_token=_enc(raw_refresh))
db.close()
router = setup_email_routes()
list_accounts = None
for route in router.routes:
if route.path == "/api/email/accounts" and "GET" in getattr(route, "methods", set()):
list_accounts = route.endpoint
break
assert list_accounts is not None, "accounts list route not found"
with mock.patch("core.database.SessionLocal", Factory):
result = await list_accounts(owner="alice")
blob = json.dumps(result)
assert raw_access not in blob, "raw access token must not appear in list response"
assert raw_refresh not in blob, "raw refresh token must not appear in list response"
assert _enc(raw_access) not in blob, "encrypted token must not be sent to the client either"
acct = result["accounts"][0]
assert acct["oauth_provider"] == "google" # status is exposed
assert "oauth_access_token" not in acct # token value is not
assert "oauth_refresh_token" not in acct
+69
View File
@@ -0,0 +1,69 @@
"""Regression: FASTEMBED_CACHE_DIR must tolerate a PRESENT-but-EMPTY
FASTEMBED_CACHE_PATH.
docker-compose.yml injects ``FASTEMBED_CACHE_PATH=${FASTEMBED_CACHE_PATH:-}``,
which sets the variable to ``""`` when the host has not defined it. The old
``os.getenv("FASTEMBED_CACHE_PATH", default)`` only used the default when the
variable was ABSENT, so an empty value made ``FASTEMBED_CACHE_DIR == ""``
``os.makedirs("")`` raised ``[Errno 2] No such file or directory: ''``
FastEmbed failed to initialise and every vector feature (RAG, semantic memory,
tool index) silently degraded on the default Docker stack.
These tests pin the fix: empty is treated like absent use the DATA_DIR
default, while an explicit non-empty override is still honoured.
"""
from __future__ import annotations
import importlib
import os
import src.constants as constants
def _reload_with(monkeypatch, value):
"""Reload src.constants with FASTEMBED_CACHE_PATH set to ``value`` (or
removed when ``value`` is None) and return the reloaded module."""
if value is None:
monkeypatch.delenv("FASTEMBED_CACHE_PATH", raising=False)
else:
monkeypatch.setenv("FASTEMBED_CACHE_PATH", value)
return importlib.reload(constants)
def _restore(monkeypatch):
"""Return the module to its env-default state so reloading it here does
not leak a test-specific FASTEMBED_CACHE_DIR into other tests."""
monkeypatch.delenv("FASTEMBED_CACHE_PATH", raising=False)
importlib.reload(constants)
def test_empty_fastembed_cache_path_falls_back_to_default(monkeypatch):
"""The bug: an empty FASTEMBED_CACHE_PATH (exactly what Docker injects)
must fall back to the DATA_DIR default, never the empty string."""
try:
mod = _reload_with(monkeypatch, "")
assert mod.FASTEMBED_CACHE_DIR, "empty env must not yield an empty path"
assert mod.FASTEMBED_CACHE_DIR == os.path.join(mod.DATA_DIR, "fastembed_cache")
finally:
_restore(monkeypatch)
def test_unset_fastembed_cache_path_uses_default(monkeypatch):
"""Sanity: an absent variable also resolves to the default."""
try:
mod = _reload_with(monkeypatch, None)
assert mod.FASTEMBED_CACHE_DIR == os.path.join(mod.DATA_DIR, "fastembed_cache")
finally:
_restore(monkeypatch)
def test_explicit_fastembed_cache_path_is_respected(monkeypatch):
"""A real explicit override must still win — the fix only changes the
empty-value handling, not the documented FASTEMBED_CACHE_PATH override."""
custom = os.path.join("custom", "fastembed-cache")
try:
mod = _reload_with(monkeypatch, custom)
assert mod.FASTEMBED_CACHE_DIR == custom
finally:
_restore(monkeypatch)
@@ -2,7 +2,14 @@ import os
from pathlib import Path
import pytest
from fastapi import FastAPI
from fastapi import HTTPException
from fastapi.testclient import TestClient
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import NullPool
from core.database import Base, GalleryImage
def _gallery_module():
@@ -53,6 +60,57 @@ def test_gallery_image_path_rejects_symlink_escape(tmp_path, monkeypatch):
assert exc.value.status_code == 400
def test_gallery_replace_rejects_symlink_escape(tmp_path, monkeypatch):
gallery_routes = _gallery_module()
image_dir = tmp_path / "generated_images"
image_dir.mkdir()
outside = tmp_path / "outside.png"
outside.write_bytes(b"outside image root")
link = image_dir / "escape.png"
try:
os.symlink(outside, link)
except (AttributeError, NotImplementedError, OSError) as exc:
pytest.skip(f"symlinks unavailable: {exc}")
engine = create_engine(
f"sqlite:///{tmp_path / 'gallery.db'}",
connect_args={"check_same_thread": False},
poolclass=NullPool,
)
Base.metadata.create_all(engine)
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
db = SessionLocal()
try:
db.add(
GalleryImage(
id="img-1",
filename="escape.png",
prompt="escape",
owner="alice",
is_active=True,
)
)
db.commit()
finally:
db.close()
monkeypatch.setattr(gallery_routes, "GALLERY_IMAGE_DIR", image_dir)
monkeypatch.setattr(gallery_routes, "SessionLocal", SessionLocal)
monkeypatch.setattr(gallery_routes, "get_current_user", lambda request: "alice")
app = FastAPI()
app.include_router(gallery_routes.setup_gallery_routes())
client = TestClient(app)
response = client.post(
"/api/gallery/img-1/replace",
files={"image": ("replacement.png", b"replacement bytes", "image/png")},
)
assert response.status_code == 400
assert outside.read_bytes() == b"outside image root"
def test_gallery_file_operations_use_confining_resolver():
source = Path("routes/gallery_routes.py").read_text(encoding="utf-8")
+1 -1
View File
@@ -79,7 +79,7 @@ def _build_context_harness(monkeypatch, chat_helpers, history):
monkeypatch.setattr(chat_helpers, "extract_preset", fake_extract_preset)
monkeypatch.setattr(chat_helpers, "add_user_message", fake_add_user_message)
monkeypatch.setattr(chat_helpers, "load_prefs_for_user", lambda user: {})
monkeypatch.setattr(chat_helpers, "get_current_user", lambda request: "tester")
monkeypatch.setattr(chat_helpers, "effective_user", lambda request: "tester")
monkeypatch.setattr(chat_helpers, "normalize_model_id", lambda endpoint_url, model, **kwargs: None)
monkeypatch.setattr(chat_helpers, "maybe_compact", fake_maybe_compact)
monkeypatch.setattr(chat_helpers, "trim_for_context", lambda messages, context_length: messages)
+150
View File
@@ -0,0 +1,150 @@
import asyncio
import mcp_servers.memory_server as memory_server
from src.memory import MemoryManager
class FakeVector:
healthy = True
def __init__(self):
self.added = []
self.removed = []
def add(self, memory_id, text):
self.added.append((memory_id, text))
def remove(self, memory_id):
self.removed.append(memory_id)
def _tool_text(arguments):
result = asyncio.run(memory_server.call_tool("manage_memory", arguments))
return result[0].text
def _entry(manager, text, owner=None, memory_id=None, category="fact"):
entry = manager.add_entry(text, owner=owner, category=category)
if memory_id:
entry["id"] = memory_id
return entry
def _configure_server(monkeypatch, manager, vector=None):
monkeypatch.setattr(memory_server, "_memory_manager", manager)
monkeypatch.setattr(memory_server, "_memory_vector", vector)
monkeypatch.setattr(memory_server, "_initialized", True)
for key in memory_server._OWNER_ENV_KEYS:
monkeypatch.delenv(key, raising=False)
def test_mcp_memory_uses_configured_owner_for_all_operations(monkeypatch, tmp_path):
manager = MemoryManager(str(tmp_path))
vector = FakeVector()
alice = _entry(
manager,
"Alice likes green tea",
owner="alice",
memory_id="aaaaaaaa-0000-0000-0000-000000000000",
)
bob = _entry(
manager,
"Bob likes espresso",
owner="bob",
memory_id="bbbbbbbb-0000-0000-0000-000000000000",
)
manager.save([alice, bob])
_configure_server(monkeypatch, manager, vector)
monkeypatch.setenv("ODYSSEUS_MCP_MEMORY_OWNER", "alice")
list_text = _tool_text({"action": "list"})
assert "Alice likes green tea" in list_text
assert "Bob likes espresso" not in list_text
search_text = _tool_text({"action": "search", "text": "likes"})
assert "Alice likes green tea" in search_text
assert "Bob likes espresso" not in search_text
add_text = _tool_text({
"action": "add",
"text": "Alice prefers concise notes",
"category": "preference",
})
assert "Memory added" in add_text
added = next(
entry for entry in manager.load_all()
if entry["text"] == "Alice prefers concise notes"
)
assert added["owner"] == "alice"
assert vector.added == [(added["id"], "Alice prefers concise notes")]
edit_text = _tool_text({
"action": "edit",
"memory_id": bob["id"][:8],
"text": "Bob changed",
})
assert edit_text == "Error: Memory 'bbbbbbbb' not found"
bob_after_edit = next(
entry for entry in manager.load_all()
if entry["id"] == bob["id"]
)
assert bob_after_edit["text"] == "Bob likes espresso"
delete_text = _tool_text({"action": "delete", "memory_id": bob["id"][:8]})
assert delete_text == "Error: Memory 'bbbbbbbb' not found"
assert any(entry["id"] == bob["id"] for entry in manager.load_all())
def test_mcp_memory_fails_closed_without_owner_for_owner_scoped_store(monkeypatch, tmp_path):
manager = MemoryManager(str(tmp_path))
alice = _entry(manager, "Alice private memory", owner="alice", memory_id="aaaaaaaa-0000")
bob = _entry(manager, "Bob private memory", owner="bob", memory_id="bbbbbbbb-0000")
manager.save([alice, bob])
_configure_server(monkeypatch, manager, FakeVector())
before = manager.load_all()
actions = [
{"action": "list"},
{"action": "search", "text": "private"},
{"action": "add", "text": "new ownerless memory"},
{"action": "edit", "memory_id": alice["id"][:8], "text": "changed"},
{"action": "delete", "memory_id": alice["id"][:8]},
]
for arguments in actions:
assert _tool_text(arguments).startswith("Error: Memory MCP owner is not configured")
assert manager.load_all() == before
def test_mcp_memory_preserves_ownerless_local_behavior(monkeypatch, tmp_path):
manager = MemoryManager(str(tmp_path))
legacy = _entry(
manager,
"Legacy local memory",
memory_id="llllllll-0000-0000-0000-000000000000",
)
manager.save([legacy])
_configure_server(monkeypatch, manager, FakeVector())
assert "Legacy local memory" in _tool_text({"action": "list"})
assert "Legacy local memory" in _tool_text({"action": "search", "text": "legacy"})
add_text = _tool_text({"action": "add", "text": "Another local memory"})
assert "Memory added" in add_text
added = next(
entry for entry in manager.load_all()
if entry["text"] == "Another local memory"
)
assert "owner" not in added
assert _tool_text({
"action": "edit",
"memory_id": legacy["id"][:8],
"text": "Updated local memory",
}) == "Memory updated: Updated local memory"
assert any(entry["text"] == "Updated local memory" for entry in manager.load_all())
delete_text = _tool_text({"action": "delete", "memory_id": legacy["id"][:8]})
assert delete_text.startswith("Memory deleted:")
assert all(entry["id"] != legacy["id"] for entry in manager.load_all())
+76 -1
View File
@@ -1286,6 +1286,14 @@ class _ImmediateThread:
self.target()
class _NoopThread:
def __init__(self, target, daemon=None):
self.target = target
def start(self):
return None
def _wait_for(predicate, timeout=2.0):
deadline = time.time() + timeout
while time.time() < deadline:
@@ -1313,6 +1321,7 @@ def _route_ep(
pinned_models=None,
refresh_mode="auto",
refresh_timeout=None,
owner=None,
):
return SimpleNamespace(
id=id,
@@ -1329,7 +1338,7 @@ def _route_ep(
model_refresh_interval=None,
model_refresh_timeout=refresh_timeout,
supports_tools=None,
owner=None,
owner=owner,
created_at=None,
updated_at=None,
)
@@ -1342,6 +1351,72 @@ def _route_request():
)
def test_api_models_rejects_api_token_without_chat_scope(monkeypatch):
router = model_routes.setup_model_routes(model_discovery=None)
def fail_session():
raise AssertionError("model DB should not be queried without chat scope")
monkeypatch.setattr(model_routes, "SessionLocal", fail_session)
request = SimpleNamespace(
state=SimpleNamespace(
current_user="api",
api_token=True,
api_token_owner="alice",
api_token_scopes=["documents:read"],
),
app=SimpleNamespace(
state=SimpleNamespace(
auth_manager=SimpleNamespace(is_configured=True, is_admin=lambda user: False),
),
),
)
with pytest.raises(HTTPException) as exc:
_route_endpoint(router, "/api/models")(request)
assert exc.value.status_code == 403
assert "chat" in str(exc.value.detail)
def test_api_models_scopes_api_token_to_token_owner(monkeypatch):
rows = [
_route_ep("alice", "http://alice.example/v1", cached_models=["alice-model"], owner="alice"),
_route_ep("shared", "http://shared.example/v1", cached_models=["shared-model"], owner=None),
_route_ep("bob", "http://bob.example/v1", cached_models=["bob-model"], owner="bob"),
]
db = _RouteDb(rows)
router = model_routes.setup_model_routes(model_discovery=None)
admin_checks = []
monkeypatch.setattr(model_routes, "ModelEndpoint", _RouteModelEndpoint)
monkeypatch.setattr(model_routes, "SessionLocal", lambda: db)
monkeypatch.setattr(threading, "Thread", _NoopThread)
request = SimpleNamespace(
state=SimpleNamespace(
current_user="api",
api_token=True,
api_token_owner="alice",
api_token_scopes=["chat"],
),
app=SimpleNamespace(
state=SimpleNamespace(
auth_manager=SimpleNamespace(
is_configured=True,
is_admin=lambda user: admin_checks.append(user) or False,
),
),
),
)
result = _route_endpoint(router, "/api/models")(request)
assert [item["endpoint_name"] for item in result["items"]] == ["alice", "shared"]
assert admin_checks == ["alice"]
def test_api_models_returns_cached_proxy_models_without_refresh_probe(monkeypatch):
row = _route_ep(
"proxy",
+188
View File
@@ -0,0 +1,188 @@
"""Owner-scoped note routes must fail closed when the request has no identity.
The notes CRUD routes resolved the acting user with bare get_current_user().
A request that reached them carrying no identity (auth-middleware regression,
SSRF from a sibling service) therefore came through as user=None and the
queries treat None as the single-user mode, i.e. blanket access to every
account's notes: list everything, read/update/delete/pin/archive any row,
reorder globally.
require_user() already encodes the correct policy 401 when auth is
configured, while the documented anonymous modes (AUTH_ENABLED=false,
LOCALHOST_BYPASS on loopback, unconfigured first-run) still pass and
fire-reminder in the same file already used it. The CRUD routes now resolve
the owner through it too.
Test transport note: these drive the ASGI app through ``httpx.ASGITransport``
+ ``httpx.AsyncClient`` rather than ``starlette.testclient.TestClient``.
TestClient runs the app inside a background event-loop thread spun up by
``anyio.from_thread.start_blocking_portal`` and then dispatches each sync
endpoint onto *another* worker thread; on some anyio/httpx/platform
combinations that two-thread handshake deadlocks and ``TestClient(app).get(...)``
simply hangs. ASGITransport runs the whole request on the test's own event
loop no portal thread, no BaseHTTPMiddleware so the suite is portable.
Identity is injected by a pure-ASGI shim that writes the same
``request.state`` fields the real auth middleware sets.
"""
import uuid
from types import SimpleNamespace
import httpx
import pytest
from fastapi import FastAPI
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import NullPool
import core.database as cdb
from core.database import Note
import routes.note_routes as nr
# A deliberately NON-loopback peer. require_user has loopback fall-throughs
# (unconfigured first-run, LOCALHOST_BYPASS); pinning a public-looking client
# keeps every assertion below about the *configured-auth* path and not an
# accidental loopback bypass — the same reason the old fixture leaned on
# TestClient's non-loopback "testclient" host.
_PEER = ("203.0.113.7", 54321)
class _Identity:
"""Pure-ASGI shim mirroring what the auth middleware writes onto
request.state. Pure-ASGI on purpose it stays off Starlette's
BaseHTTPMiddleware + sync-TestClient path, the source of the
``TestClient(app).get(...)`` hang. No x-test-user header => no identity,
the exact state an auth-middleware regression would produce."""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] == "http":
headers = dict(scope.get("headers") or [])
state = scope.setdefault("state", {})
user = headers.get(b"x-test-user")
if user:
state["current_user"] = user.decode()
if headers.get(b"x-test-api-token"):
state["current_user"] = "api"
state["api_token"] = True
await self.app(scope, receive, send)
def _temp_db(tmp_path):
"""Note routes over a fresh temp DB; returns the session factory."""
engine = create_engine(
f"sqlite:///{tmp_path / 'notes.db'}",
connect_args={"check_same_thread": False},
poolclass=NullPool,
)
cdb.Base.metadata.create_all(engine)
return sessionmaker(bind=engine)
def _build_app(factory, *, configured=True):
app = FastAPI()
app.state.auth_manager = SimpleNamespace(is_configured=configured)
app.include_router(nr.setup_note_routes())
return _Identity(app)
def _client(app):
"""AsyncClient over the ASGI app with a non-loopback peer. Caller drives
it inside ``async with``."""
transport = httpx.ASGITransport(app=app, client=_PEER)
return httpx.AsyncClient(transport=transport, base_url="http://notes.test")
@pytest.fixture
def env(monkeypatch, tmp_path):
"""Configured-auth world: AUTH_ENABLED=true, auth_manager.is_configured,
no LOCALHOST_BYPASS. Identity comes only from the x-test-user header
(mirroring the auth middleware); no header => no identity, the exact state
an auth-middleware regression leaves behind. Seeds one note each for alice
and bob. Returns (app, factory)."""
factory = _temp_db(tmp_path)
monkeypatch.setattr(nr, "SessionLocal", factory)
monkeypatch.setenv("AUTH_ENABLED", "true")
monkeypatch.delenv("LOCALHOST_BYPASS", raising=False)
app = _build_app(factory)
db = factory()
db.add(Note(id="note-alice", owner="alice", title="a", content="x",
items='[{"text": "t", "done": false}]'))
db.add(Note(id="note-bob", owner="bob", title="b", content="y"))
db.commit()
db.close()
return app, factory
async def test_no_identity_fails_closed_on_every_owner_scoped_route(env):
app, _ = env
async with _client(app) as c:
assert (await c.get("/api/notes")).status_code == 401
assert (await c.get("/api/notes/note-alice")).status_code == 401
assert (await c.put("/api/notes/note-alice", json={"title": "pwn"})).status_code == 401
assert (await c.delete("/api/notes/note-alice")).status_code == 401
assert (await c.post("/api/notes/note-alice/pin")).status_code == 401
assert (await c.post("/api/notes/note-alice/archive")).status_code == 401
assert (await c.post("/api/notes/note-alice/items/0/toggle")).status_code == 401
assert (await c.post("/api/notes/reorder", json={"ids": ["note-bob", "note-alice"]})).status_code == 401
assert (await c.post("/api/notes", json={"title": "ghost"})).status_code == 401
async def test_no_identity_did_not_mutate_anything(env):
app, factory = env
async with _client(app) as c:
await c.put("/api/notes/note-alice", json={"title": "pwn"})
await c.post("/api/notes/note-alice/pin")
await c.delete("/api/notes/note-bob")
db = factory()
rows = {n.id: n for n in db.query(Note).all()}
db.close()
assert set(rows) == {"note-alice", "note-bob"}
assert rows["note-alice"].title == "a"
assert not rows["note-alice"].pinned
async def test_authenticated_user_still_scoped_to_own_notes(env):
app, _ = env
alice = {"x-test-user": "alice"}
async with _client(app) as c:
listed = (await c.get("/api/notes", headers=alice)).json()["notes"]
assert [n["id"] for n in listed] == ["note-alice"]
assert (await c.get("/api/notes/note-alice", headers=alice)).status_code == 200
# Someone else's note stays a 404 (don't reveal it exists).
assert (await c.get("/api/notes/note-bob", headers=alice)).status_code == 404
assert (await c.put("/api/notes/note-alice", json={"title": "mine"}, headers=alice)).status_code == 200
async def test_api_token_pseudo_user_is_rejected(env):
"""Bearer tokens must use the scope-aware API routes (require_user's
existing contract), not slip into cookie-session routes as user 'api'."""
app, _ = env
async with _client(app) as c:
r = await c.get("/api/notes", headers={"x-test-api-token": "1"})
assert r.status_code == 403
async def test_auth_disabled_keeps_single_user_mode_working(monkeypatch, tmp_path):
"""AUTH_ENABLED=false is the operator's explicit anonymous mode: no
identity must still mean full single-user access (issue #622 contract),
even with a stale configured auth.json on disk."""
factory = _temp_db(tmp_path)
monkeypatch.setattr(nr, "SessionLocal", factory)
monkeypatch.setenv("AUTH_ENABLED", "false")
app = _build_app(factory)
db = factory()
db.add(Note(id="n1", owner=None, title="solo", content="x"))
db.commit()
db.close()
async with _client(app) as c:
assert [n["id"] for n in (await c.get("/api/notes")).json()["notes"]] == ["n1"]
assert (await c.put("/api/notes/n1", json={"title": "still mine"})).status_code == 200
assert (await c.post("/api/notes/n1/pin")).status_code == 200
@@ -0,0 +1,74 @@
import asyncio
import os
from pathlib import Path
from routes import personal_routes
class _FakePersonalDocs:
def __init__(self):
self.excluded = []
def exclude_file(self, filepath):
self.excluded.append(filepath)
class _FakeRAG:
def __init__(self):
self.deleted_sources = []
def delete_by_source(self, filepath):
self.deleted_sources.append(filepath)
return 1
def _delete_endpoint(personal_docs):
router = personal_routes.setup_personal_routes(personal_docs, None, True)
for route in router.routes:
if getattr(route, "path", "") == "/api/personal/file" and "DELETE" in getattr(route, "methods", set()):
return route.endpoint
raise AssertionError("DELETE /api/personal/file endpoint not found")
def test_delete_file_refuses_symlink_directory_escape(tmp_path, monkeypatch):
uploads = tmp_path / "uploads"
uploads.mkdir()
outside = tmp_path / "outside"
outside.mkdir()
victim = outside / "victim.txt"
victim.write_text("keep me", encoding="utf-8")
os.symlink(outside, uploads / "linked")
docs = _FakePersonalDocs()
rag = _FakeRAG()
monkeypatch.setattr(personal_routes, "UPLOADS_DIR", str(uploads))
monkeypatch.setattr(personal_routes, "get_rag_manager", lambda: rag)
filepath = str(uploads / "linked" / "victim.txt")
result = asyncio.run(_delete_endpoint(docs)(filepath=filepath, owner="alice", _admin=None))
assert result["deleted_from_disk"] is False
assert victim.read_text(encoding="utf-8") == "keep me"
assert docs.excluded == [filepath]
assert rag.deleted_sources == [filepath]
def test_delete_file_removes_regular_file_inside_upload_root(tmp_path, monkeypatch):
uploads = tmp_path / "uploads"
uploads.mkdir()
uploaded_file = uploads / "alice" / "notes.txt"
uploaded_file.parent.mkdir()
uploaded_file.write_text("delete me", encoding="utf-8")
docs = _FakePersonalDocs()
rag = _FakeRAG()
monkeypatch.setattr(personal_routes, "UPLOADS_DIR", str(uploads))
monkeypatch.setattr(personal_routes, "get_rag_manager", lambda: rag)
filepath = str(uploaded_file)
result = asyncio.run(_delete_endpoint(docs)(filepath=filepath, owner="alice", _admin=None))
assert result["deleted_from_disk"] is True
assert not uploaded_file.exists()
assert docs.excluded == [filepath]
assert rag.deleted_sources == [filepath]
+21 -16
View File
@@ -1,16 +1,18 @@
"""Regression guard for issue #1390 — the README banner / ASCII art was not in a
fenced code block, so GitHub's markdown collapsed its leading whitespace and the
box-drawing rules, rendering it misaligned instead of monospace-as-typed.
"""Regression guard for the README title presentation.
This pins that the decorative banner stays inside a ``` code fence.
Originally (#1390) the README opened with an ASCII-art banner that had to live
inside a ``` code fence, otherwise GitHub's markdown collapsed its leading
whitespace and box-drawing rules and rendered it misaligned. The README refresh
(#4306) dropped that banner in favour of a centered wordmark image, so the guard
now pins the wordmark identity instead, while still catching the original failure
mode if an un-fenced ASCII banner is ever reintroduced.
"""
from pathlib import Path
README = Path(__file__).resolve().parent.parent / "README.md"
# Distinctive bits of the banner (box-drawing rule + the kaomoji version line).
# Box-drawing rule from the legacy ASCII banner (the #1390 failure mode).
_RULE = "" * 10
_BANNER_LINE = "Odysseus vers. 1.0"
def _fenced_segments(text: str):
@@ -20,15 +22,18 @@ def _fenced_segments(text: str):
return parts[1::2]
def test_readme_banner_is_inside_a_code_fence():
def test_readme_opens_with_wordmark_title():
# The README must still open with a recognizable Odysseus title: now the
# centered wordmark image rather than an H1 / ASCII banner.
head = "\n".join(README.read_text(encoding="utf-8").splitlines()[:15])
assert 'alt="Odysseus"' in head, "README must open with the Odysseus wordmark image"
def test_reintroduced_ascii_banner_stays_fenced():
# Defensive: if a box-drawing banner is ever added back, it must be fenced so
# GitHub renders it monospace-as-typed (the original #1390 regression).
text = README.read_text(encoding="utf-8")
assert _BANNER_LINE in text, "banner line missing from README"
if _RULE not in text:
return
inside = "\n".join(_fenced_segments(text))
assert _BANNER_LINE in inside, "banner version line must be inside a ``` code fence"
assert _RULE in inside, "banner rule line must be inside a ``` code fence"
def test_readme_title_stays_a_heading():
# The H1 must remain a real heading, not get swallowed into the fence.
first = README.read_text(encoding="utf-8").splitlines()[0]
assert first.strip() == "# Odysseus"
assert _RULE in inside, "ASCII banner rule must be inside a ``` code fence"
+1 -1
View File
@@ -385,7 +385,7 @@ async def test_build_chat_context_incognito_does_not_duplicate_current_user_mess
monkeypatch.setattr(chat_helpers, "extract_preset", fake_extract_preset)
monkeypatch.setattr(chat_helpers, "add_user_message", fake_add_user_message)
monkeypatch.setattr(chat_helpers, "load_prefs_for_user", lambda user: {})
monkeypatch.setattr(chat_helpers, "get_current_user", lambda request: "tester")
monkeypatch.setattr(chat_helpers, "effective_user", lambda request: "tester")
monkeypatch.setattr(chat_helpers, "normalize_model_id", lambda endpoint_url, model, **kwargs: None)
monkeypatch.setattr(chat_helpers, "maybe_compact", fake_maybe_compact)
monkeypatch.setattr(chat_helpers, "trim_for_context", lambda messages, context_length: messages)
+50
View File
@@ -0,0 +1,50 @@
import os
import sys
from unittest import mock
import pytest
from src.runtime_paths import get_app_root, get_default_data_dir
def test_get_app_root_normal_run():
"""Verify that get_app_root returns the repository root parent of src/ when not frozen."""
with mock.patch.object(sys, "frozen", False, create=True):
app_root = get_app_root()
# Verify it is a valid directory path and matches expected parent structure
assert os.path.isdir(app_root)
assert os.path.exists(os.path.join(app_root, "src"))
def test_get_app_root_frozen_with_meipass():
"""Verify that get_app_root returns the sys._MEIPASS directory when frozen by PyInstaller."""
mock_meipass = os.path.abspath("mock_meipass_dir")
with mock.patch.object(sys, "frozen", True, create=True), \
mock.patch.object(sys, "_MEIPASS", mock_meipass, create=True):
app_root = get_app_root()
assert app_root == mock_meipass
def test_get_app_root_frozen_without_meipass():
"""Verify that get_app_root falls back to the sys.executable parent directory when frozen but _MEIPASS is absent."""
mock_exe_path = os.path.join(os.path.abspath("mock_exe_dir"), "Odysseus.exe")
with mock.patch.object(sys, "frozen", True, create=True), \
mock.patch.object(sys, "executable", mock_exe_path, create=True):
# Remove sys._MEIPASS if it exists in the test process environment
if hasattr(sys, "_MEIPASS"):
delattr(sys, "_MEIPASS")
app_root = get_app_root()
assert app_root == os.path.abspath("mock_exe_dir")
def test_get_default_data_dir_normal():
"""Verify that get_default_data_dir resolves to get_app_root() / 'data' when not frozen."""
with mock.patch.object(sys, "frozen", False, create=True):
res = get_default_data_dir()
assert res == os.path.join(get_app_root(), "data")
def test_get_default_data_dir_frozen():
"""Verify that get_default_data_dir resolves to a persistent user path under ~ when frozen."""
with mock.patch.object(sys, "frozen", True, create=True):
res = get_default_data_dir()
expected = os.path.join(os.path.expanduser("~"), ".odysseus", "data")
assert res == expected
@@ -58,7 +58,7 @@ def test_content_fetcher_extracts_og_image_and_body_fallback(module, tmp_path, m
monkeypatch.setattr(module, "CONTENT_CACHE_DIR", tmp_path)
module.content_cache_index.clear()
monkeypatch.setattr(module, "_get_public_url", lambda url, headers, timeout: _FakeResponse(html))
monkeypatch.setattr(module, "_get_public_url", lambda url, headers, timeout, **kwargs: _FakeResponse(html))
result = module.fetch_webpage_content("https://example.com/parity-test")
@@ -82,7 +82,7 @@ def test_fetch_webpage_content_returns_empty_result_on_http_status_error(status_
monkeypatch.setattr(
service_content,
"_get_public_url",
lambda url, headers, timeout: _FakeErrorResponse(status_code),
lambda url, headers, timeout, **kwargs: _FakeErrorResponse(status_code),
)
result = service_content.fetch_webpage_content(f"https://example.com/status-{status_code}")
@@ -119,7 +119,7 @@ def test_fetch_webpage_content_429_takes_distinct_rate_limit_path(tmp_path, monk
monkeypatch.setattr(
service_content,
"_get_public_url",
lambda url, headers, timeout: _FakeRateLimitResponse(),
lambda url, headers, timeout, **kwargs: _FakeRateLimitResponse(),
)
result = service_content.fetch_webpage_content("https://example.com/rate-limited")
+13 -4
View File
@@ -121,9 +121,12 @@ def test_docker_compose_binds_web_ui_to_loopback_by_default():
def test_readme_native_quickstart_uses_loopback():
readme = Path("README.md").read_text(encoding="utf-8")
assert "python -m uvicorn app:app --host 127.0.0.1 --port 7000" in readme
assert "0.0.0.0` only when you intentionally want" in readme
# The README refresh (#4306) moved the native quickstart into docs/setup.md,
# so accept the loopback guidance from either the README or the setup guide.
docs = Path("README.md").read_text(encoding="utf-8")
docs += "\n" + Path("docs/setup.md").read_text(encoding="utf-8")
assert "python -m uvicorn app:app --host 127.0.0.1 --port 7000" in docs
assert "0.0.0.0` only when you intentionally want" in docs
def test_ollama_cookbook_runner_does_not_force_public_bind():
@@ -901,7 +904,13 @@ def test_web_fetch_guard_blocks_redirect_into_private(monkeypatch):
url = "http://public.example/start"
headers = {"location": "http://169.254.169.254/latest/meta-data/"}
monkeypatch.setattr(httpx, "get", lambda url, **kwargs: _Resp())
from contextlib import contextmanager
@contextmanager
def _fake_stream(method, url, **kwargs):
yield _Resp()
monkeypatch.setattr(httpx, "stream", _fake_stream)
with _pytest.raises(httpx.RequestError) as exc:
content._get_public_url("http://public.example/start", headers={}, timeout=5)
+1 -1
View File
@@ -52,6 +52,6 @@ def test_chat_endpoint_recovery_paths_are_owner_scoped():
assert "def _clear_orphaned_session_endpoint(sess, owner:" in chat_routes
assert "def _recover_empty_session_model(sess, session_id: str, owner:" in chat_routes
assert "q = owner_filter(q, ModelEndpoint, owner)" in chat_routes
assert "resolve_session_auth(sess, session, owner=get_current_user(request))" in chat_routes
assert "resolve_session_auth(sess, session, owner=effective_user(request))" in chat_routes
assert "def resolve_session_auth(sess, session_id: str, owner:" in chat_helpers
assert "update_q = update_q.filter(DBSession.owner == owner)" in chat_helpers
+1 -1
View File
@@ -35,7 +35,7 @@ def _patch_fetch(monkeypatch, text, content_type):
monkeypatch.setattr(
content_mod,
"_get_public_url",
lambda url, headers=None, timeout=5: _FakeResponse(text, content_type),
lambda url, headers=None, timeout=5, **kwargs: _FakeResponse(text, content_type),
)
+206
View File
@@ -0,0 +1,206 @@
"""web_fetch download budgets (#3812).
MAX_OUTPUT_CHARS only trims what the agent sees; these caps bound what the
server downloads, parses, and caches. Soft cap by default with a truncation
notice, per-call override clamped to the hard cap, and a pre-buffer refusal
when Content-Length already exceeds the hard ceiling.
"""
import json
from contextlib import contextmanager
import pytest
from src.constants import WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES
from services.search import content as content_mod
class _FakeStream:
"""Stands in for the httpx.stream(...) context manager."""
def __init__(self, body: bytes, content_type="text/plain", content_length=None,
status_code=200, chunk=8192):
self._body = body
self._chunk = chunk
self.status_code = status_code
self.encoding = "utf-8"
self.url = "https://example.com/x"
self.headers = {"Content-Type": content_type}
if content_length is not None:
self.headers["content-length"] = str(content_length)
self.body_reads = 0
def iter_bytes(self):
for i in range(0, len(self._body), self._chunk):
self.body_reads += 1
yield self._body[i:i + self._chunk]
@pytest.fixture
def no_cache(monkeypatch, tmp_path):
monkeypatch.setattr(content_mod, "CONTENT_CACHE_DIR", tmp_path)
monkeypatch.setattr(content_mod, "_cache_result", lambda *a, **k: None)
monkeypatch.setattr(content_mod, "_public_http_url", lambda u: True)
def _patch_stream(monkeypatch, fake):
@contextmanager
def fake_stream(method, url, **kwargs):
yield fake
monkeypatch.setattr(content_mod.httpx, "stream", fake_stream)
return fake
def test_body_under_cap_is_untouched(monkeypatch, no_cache):
_patch_stream(monkeypatch, _FakeStream(b"hello world"))
r = content_mod.fetch_webpage_content("https://example.com/a.txt")
assert r["success"] is True
assert r["content"] == "hello world"
assert r["truncated"] is False
assert r["fetched_bytes"] == len(b"hello world")
def test_body_over_soft_cap_truncates_with_flags(monkeypatch, no_cache):
body = b"x" * (WEB_FETCH_SOFT_MAX_BYTES + 50_000)
_patch_stream(monkeypatch, _FakeStream(body, content_length=len(body)))
r = content_mod.fetch_webpage_content("https://example.com/big.txt")
assert r["truncated"] is True
assert r["fetched_bytes"] == WEB_FETCH_SOFT_MAX_BYTES
assert r["total_bytes"] == len(body)
assert len(r["content"]) == WEB_FETCH_SOFT_MAX_BYTES
def test_max_bytes_override_raises_budget(monkeypatch, no_cache):
body = b"y" * (WEB_FETCH_SOFT_MAX_BYTES + 50_000)
_patch_stream(monkeypatch, _FakeStream(body))
r = content_mod.fetch_webpage_content(
"https://example.com/big.txt", max_bytes=len(body) + 1
)
assert r["truncated"] is False
assert r["fetched_bytes"] == len(body)
def test_override_is_clamped_to_hard_cap(monkeypatch, no_cache):
# Ask for more than the ceiling; the effective budget must be the ceiling.
fake = _patch_stream(monkeypatch, _FakeStream(b"z" * 10, chunk=4))
r = content_mod.fetch_webpage_content(
"https://example.com/a.txt", max_bytes=WEB_FETCH_HARD_MAX_BYTES * 10
)
assert r["success"] is True
# The clamp itself: effective cap recorded in the cache key path is the
# hard cap, and a declared body over the ceiling is refused regardless.
big = _FakeStream(b"", content_length=WEB_FETCH_HARD_MAX_BYTES + 1)
_patch_stream(monkeypatch, big)
r = content_mod.fetch_webpage_content(
"https://example.com/huge.bin", max_bytes=WEB_FETCH_HARD_MAX_BYTES * 10
)
assert r["success"] is False
assert "TooLarge" in r["error"]
assert big.body_reads == 0 # refused before buffering
def test_declared_over_hard_cap_refused_before_buffering(monkeypatch, no_cache):
fake = _FakeStream(b"irrelevant", content_length=WEB_FETCH_HARD_MAX_BYTES + 1)
_patch_stream(monkeypatch, fake)
r = content_mod.fetch_webpage_content("https://example.com/huge.iso")
assert r["success"] is False
assert "TooLarge" in r["error"]
assert fake.body_reads == 0
def test_truncated_pdf_is_an_error_not_garbage(monkeypatch, no_cache):
body = b"%PDF-1.4 " + b"p" * (WEB_FETCH_SOFT_MAX_BYTES + 10)
_patch_stream(monkeypatch, _FakeStream(body, content_type="application/pdf"))
r = content_mod.fetch_webpage_content("https://example.com/big.pdf")
assert r["success"] is False
assert "TooLarge" in r["error"]
def test_fetch_requests_identity_encoding(monkeypatch, no_cache):
# Compressed responses can decode to far more than Content-Length, so the
# streamed cap and the hard-cap preflight are only honest when we refuse
# transfer compression. Pin that the fetch advertises identity, not gzip.
seen = {}
@contextmanager
def fake_stream(method, url, **kwargs):
seen["headers"] = kwargs.get("headers") or {}
yield _FakeStream(b"hello")
monkeypatch.setattr(content_mod.httpx, "stream", fake_stream)
content_mod.fetch_webpage_content("https://example.com/a.txt")
assert seen["headers"].get("Accept-Encoding") == "identity"
def test_rejects_compressed_response_that_ignored_identity(monkeypatch, no_cache):
# We request Accept-Encoding: identity, but a server can ignore it and send
# gzip anyway. httpx would decode it, so a tiny compressed body could balloon
# past the cap in one decoded chunk. Refuse before reading the body.
fake = _FakeStream(b"x" * 5000, content_length=40)
fake.headers["content-encoding"] = "gzip"
_patch_stream(monkeypatch, fake)
r = content_mod.fetch_webpage_content("https://example.com/a.txt")
assert r["success"] is False
assert "Content-Encoding" in r["error"] or "compressed" in r["error"]
assert fake.body_reads == 0 # refused before decoding any body
def test_oversized_title_does_not_hide_partial_notice(monkeypatch):
# The partial-content notice is the PR's core contract; an untrusted,
# oversized page title must not push it past MAX_OUTPUT_CHARS.
import asyncio
from src.agent_tools.web_tools import WebFetchTool
from src.constants import MAX_OUTPUT_CHARS
def fake_fetch(url, timeout=10, max_bytes=None):
return {
"content": "partial body",
"title": "T" * (MAX_OUTPUT_CHARS + 5_000),
"error": "",
"truncated": True,
"fetched_bytes": WEB_FETCH_SOFT_MAX_BYTES,
"total_bytes": 9_000_000,
}
import src.search.content as alias_mod
monkeypatch.setattr(alias_mod, "fetch_webpage_content", fake_fetch)
out = asyncio.run(WebFetchTool().execute(
json.dumps({"url": "https://example.com/big.txt"}), ctx={}
))
assert out["exit_code"] == 0
assert out["output"].startswith("[partial content:")
assert '"full": true' in out["output"]
def test_tool_layer_emits_partial_notice_and_parses_full(monkeypatch):
import asyncio
from src.agent_tools.web_tools import WebFetchTool
calls = {}
def fake_fetch(url, timeout=10, max_bytes=None):
calls["max_bytes"] = max_bytes
return {
"content": "partial body",
"title": "Big File",
"error": "",
"truncated": True,
"fetched_bytes": WEB_FETCH_SOFT_MAX_BYTES,
"total_bytes": 5_000_000,
}
import src.search.content as alias_mod
monkeypatch.setattr(alias_mod, "fetch_webpage_content", fake_fetch)
out = asyncio.run(WebFetchTool().execute(
json.dumps({"url": "https://example.com/big.txt"}), ctx={}
))
assert out["exit_code"] == 0
assert "[partial content:" in out["output"]
assert '"full": true' in out["output"]
assert calls["max_bytes"] is None
asyncio.run(WebFetchTool().execute(
json.dumps({"url": "https://example.com/big.txt", "full": True}), ctx={}
))
assert calls["max_bytes"] == WEB_FETCH_HARD_MAX_BYTES
+18
View File
@@ -0,0 +1,18 @@
"""The web scraping path routes its User-Agent through one constant.
Guards the dedup: web_fetch / web_search outbound UAs go through
WEB_FETCH_USER_AGENT, so a stale or bare Mozilla string cannot be re-inlined in
the search sources.
"""
from pathlib import Path
_SEARCH = Path(__file__).resolve().parent.parent / "services" / "search"
def test_search_sources_have_no_inline_mozilla_ua():
offenders = [
str(py.relative_to(_SEARCH.parent.parent))
for py in _SEARCH.rglob("*.py")
if "Mozilla/" in py.read_text(encoding="utf-8")
]
assert not offenders, f"inline Mozilla UA found; use WEB_FETCH_USER_AGENT: {offenders}"
@@ -0,0 +1,54 @@
"""Guard: every public webhook emitter goes through the manager.
Public emitters in `routes/` must schedule their fire through
`webhook_manager.fire_and_forget(...)` (or `_spawn_tracked`). A bare
`asyncio.create_task(webhook_manager.fire(...))` escapes
`WebhookManager._bg_tasks`, so asyncio only holds a weak reference to the
delivery task and the GC can collect it before it sends silently dropping
the webhook. Catching this with a scan stops a regression from sneaking
back in via a copy-paste.
"""
import ast
from pathlib import Path
ROUTES_DIR = Path(__file__).resolve().parent.parent / "routes"
def _untracked_fire_calls(tree: ast.AST) -> list[tuple[int, str]]:
"""Return (lineno, snippet) for any asyncio.create_task(webhook_manager.fire(...))."""
hits: list[tuple[int, str]] = []
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
func = node.func
if not (isinstance(func, ast.Attribute) and func.attr == "create_task"):
continue
if not (isinstance(func.value, ast.Name) and func.value.id == "asyncio"):
continue
if not node.args:
continue
inner = node.args[0]
if not isinstance(inner, ast.Call):
continue
inner_func = inner.func
if (
isinstance(inner_func, ast.Attribute)
and inner_func.attr == "fire"
and isinstance(inner_func.value, ast.Name)
and inner_func.value.id == "webhook_manager"
):
hits.append((node.lineno, ast.unparse(node)))
return hits
def test_no_untracked_webhook_fire_in_routes():
offenders: list[str] = []
for path in ROUTES_DIR.rglob("*.py"):
tree = ast.parse(path.read_text(), filename=str(path))
for lineno, snippet in _untracked_fire_calls(tree):
offenders.append(f"{path.relative_to(ROUTES_DIR.parent)}:{lineno}: {snippet}")
assert not offenders, (
"Public webhook emitters must use webhook_manager.fire_and_forget(...) "
"so the delivery task is tracked in WebhookManager._bg_tasks. Found "
"untracked emitter(s):\n " + "\n ".join(offenders)
)