* fix(rag): skip hidden and junk directories when indexing (#5559)
index_personal_documents walked the whole tree with no pruning, so
pointing RAG at a real-world folder silently swept in .obsidian/ plugin
JS, .git/ internals, node_modules/, and __pycache__/ — multiplying
indexing time and polluting retrieval with junk chunks.
Prune hidden directories and well-known junk directories from the walk,
and skip hidden files. The explicitly passed root is exempt, so a user
who deliberately indexes a hidden directory still gets its contents.
* fix(rag): prune hidden/junk dirs in the keyword index too, via a shared helper
The #5559 fix pruned only VectorRAG.index_personal_documents (the vector index).
The parallel keyword index built by PersonalDocsManager.refresh_index ->
load_personal_index walked the same tree unpruned, so .obsidian/, .git/,
node_modules/ etc. still swept into keyword retrieval and the file listing —
the 'end-to-end' guarantee was only half true.
Single-source the pruning policy in src/index_walk (prune_index_dirs +
is_indexable_file) and use it from both walkers so they cannot drift again.
The junk-dir match is now case-insensitive, so a Node_Modules on a
case-insensitive filesystem is pruned too.
Tests: keyword-path regressions covering hidden/junk dirs, hidden files, junk
at depth (not just top level), case-insensitive junk, and the explicit-hidden-
root exemption. The existing vector tests still pass against the shared helper.
* security(url-safety): reject RFC 6598 shared address space in strict mode
Strict mode (block_private=True) is a full SSRF lockdown, but it only
rejected is_private and is_loopback targets. CPython does not classify RFC
6598 shared/CGNAT space (100.64.0.0/10) as is_private (it is "shared", not
"private"), so a public redirect into 100.64.0.1 passed the per-hop guard
and still issued the request to a potentially internal CGNAT service.
not is_global would also exclude it, but only on CPython 3.11.10+/3.12.4+/
3.13+; the CI matrix runs 3.11/3.12, so reject the range explicitly to stay
correct across patch levels and the 3.14 runtime image. Default local-first
mode is unchanged. Adds strict-mode coverage for shared, non-global, and
public targets.
* docs(url-safety): correct CGNAT is_global rationale in strict-mode comment
The prior comment claimed `not is_global` catches 100.64.0.0/10 only on
CPython 3.11.10+/3.12.4+/3.13+. That is inaccurate for CGNAT: is_global
is False for 100.64.0.1 on every supported version (verified 3.10-3.14).
The version-fragility applies to other ranges gh-113171 touched, not CGNAT.
The explicit range reject is still the right choice; restate the reason as
is_private not covering shared space, and not coupling strict mode to
is_global's broader, cross-version definition. No behavior change.
* feat(models): define capability schema and readers
* fix(models): harden Google catalog probing
Restrict native catalog probing to the Gemini host, keep provider keys out of request URLs, filter non-chat model resources, and preserve the manual refresh default in the built-in Google add flow.
* docs: update static/js/MODULE_SUMMARY.md to reflect current ES6 frontend
Rewrite the stale module summary to match the current no-build,
ES6-module frontend architecture. Adds coverage of app.js orchestration,
the chat/SSE pipeline (chat.js, chatStream.js, chatRenderer.js,
streamingRenderer.js), new subsystems (research/, compare/, document
streaming, cookbook*, skills.js), and removes the obsolete <script> load
order assumptions.
* cleanup: remove dead MEMORY_DOC / memory_doc paths (closes#4411)
Removes the unused MEMORY_DOC constant and the matching DataConfig
memory_doc field / set_data_paths entry. No runtime code imports or
references these paths, so this is a no-behavior-change dead-code
cleanup under the storage-architecture tracker #4377.
conn.search() / conn.fetch() operate on volatile positional sequence
numbers that shift whenever messages are deleted or expunged. Three call
sites in the sig-learner (_pull_headers, _fetch_bodies) and morning-brief
email section were storing these as "uid" and reusing them in subsequent
fetches — causing wrong-message returns or NO responses if another client
modified the mailbox concurrently.
Replaced with conn.uid("SEARCH", ...) / conn.uid("FETCH", ...), which use
persistent RFC 3501 UIDs. _scan_one (urgency action) already did this
correctly; these were the remaining callers.
The reproduction window is narrow (requires concurrent deletion between
search and fetch), so the fix is verified by regression tests rather than
manual end-to-end: _SpyImap raises AssertionError if conn.search() or
conn.fetch() are called instead of conn.uid().
stream_agent_loop's per-tool drain loop had no cleanup path for early
generator close. Starlette throws GeneratorExit into the generator at
whatever await point it's suspended on when the SSE client disconnects
(aclose()) - here that's 'await _progress_q.get()' inside the drain
loop, before the final 'await _tool_task' line ever runs. The task,
which wraps execute_tool_block, was left running unawaited and
uncancelled.
For bash/python tools this orphans the underlying subprocess:
subprocess_tools.py already has correct CancelledError handling that
kills the child process, but only runs if the task is actually
cancelled. A client disconnecting mid long-running command left that
subprocess running server-side for its full duration with nothing
left to reap it.
Wrap the drain loop in try/finally: on early exit, cancel _tool_task
(if not already done) and await it so the existing subprocess-kill
path runs.
Adds a regression test that drives the real stream_agent_loop with a
fake tool handler, closes the generator mid tool-call (mirroring what
Starlette does on disconnect), and asserts the handler observed
cancellation immediately - not merely via asyncio.run()'s own
end-of-run task cleanup, which would mask the bug.
Fixes#5105
_sync_blocking (src/caldav_sync.py) and _writeback_blocking
(src/caldav_writeback.py) each open their own caldav.DAVClient via
_build_dav_client, but never close it. The client owns an HTTP session
with a pooled connection; without a close() that connection is held until
process exit.
Previously the fix added explicit client.close() calls before each early
return and at the end of the DB finally block. This still leaked the
client when SessionLocal() raised before the DB try/finally was entered.
Now _sync_blocking wraps the entire post-construction path in an outer
try/finally that calls client.close() unconditionally, covering:
- AuthorizationError / NotFoundError early return
- URL-fallback failure early return
- no-calendars early return
- normal return after sync
- SessionLocal() construction failure (new regression coverage)
_writeback_blocking already used a try/finally (unchanged).
- src/caldav_sync.py: replace scattered client.close() calls with a
single outer try/finally block around the discovery + DB sync path
- tests/test_caldav_client_cleanup.py: add CalendarDeletedEvent to the
database stub; add regression test for SessionLocal() failure path
Closes#4593
_load() returned whatever json.loads() produced without checking it was a
dict; _update() did the same before assigning data[key] = value. If the
oauth_tokens column ever held a JSON array or primitive (DB corruption,
manual edit, migration drift), _load()'s callers crashed with
AttributeError on .get(), and _update() crashed with TypeError trying to
item-assign into a list/string/int.
Validate the parsed value is a dict in both methods, falling back to {}
otherwise - same recovery behavior already used elsewhere in the codebase
for this exact JSON-blob-is-not-a-dict shape (_parse_tool_args,
_is_sensitive_path's siblings).
Adds 3 regression tests for _load, get_tokens, and _update against a
non-dict oauth_tokens value.
Fixes#5082
The edit/delete/pause/run actions of do_manage_tasks gated ownership with
`if owner and task.owner and task.owner != owner`. The middle term made the
check a no-op whenever task.owner was null/empty — the state a scheduled task
sits in when it was created in no-login mode (or via the localhost middleware
bypass) before the periodic legacy-owner sweep reassigns it to the admin user.
Any authenticated user's agent could then edit, delete, pause, or run another
tenant's owner-less task; edit+run lets an attacker rewrite the task prompt and
execute it in the scheduler's agent context.
The sibling `list` action already scopes with an exact `owner == owner` filter,
so the mutators were strictly more permissive than the reader. Drop the middle
term so the guard fails closed on owner-less rows for authenticated callers,
matching `list` and the calendar/notes/gallery/session null-owner gates. Auth
disabled (owner falsy) and same-owner access are unchanged.
request_flags derives (agent, vision) and does last.get("role") after only
a truthy check. A client can send a bare-string message element
("messages": ["hi"]), and the vision loop right below already guards each
element with isinstance — so the .get() on a non-dict last element is an
oversight that raises AttributeError on every Copilot-proxied request with
such a body.
Use isinstance(last, dict) to match the loop's own guard.
Fixes#5273
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
build_user_content derived the data-URL subtype from the file extension
only (image_format = ext[1:]). An extensionless upload (e.g. a pasted
screenshot) has ext == "", producing "data:image/;base64,..." with an
empty subtype (invalid per RFC 2046) that vision/audio endpoints reject,
silently dropping the attachment. Fall back to the resolved MIME subtype
when the extension is missing; present extensions are unchanged.
Align regression tests with the current Odysseus behavior after merging origin/dev into local main.
- keep phone/name-only contacts valid and cover null email without crashes
- pin explicit web-search false form submission in chat.js
- update Cookbook dependency/download completion tests for combined live + persisted output
- expose SGLang OS package repair hints from backend diagnosis
- treat MLX and MLX-community repos as servable on Apple Metal while keeping CUDA behavior unchanged
- keep desktop new-chat coverage on the shared preferred-model helper
- remove a hardcoded crop overlay portal z-index literal
- include the local agent-loop cleanup that removes the old manage_notes reminder repair shim
Verified with: docker run --rm -v /home/pewds/odysseus-cookbook-fresh:/app -w /app odysseus-cookbook-fresh-odysseus python3 -m pytest -q (4515 passed, 4 skipped).
validate_webhook_url resolves the host to accept/reject, but the delivery
connect (httpx.AsyncClient.post) re-resolved independently — a DNS record
flipping between the two lookups (rebinding) could slip an internal IP
(127.0.0.1 / 169.254.169.254 / LAN) past the check and receive the signed
payload. The module docstring already flagged this as only a "partial
defense".
Resolve + validate once via _validated_public_ips, then pin the delivery
TCP connect to that approved IP with an async _PinnedAsyncTransport built
on the public httpcore/httpx APIs (mirrors the sync search-fetch pin from
#704). The URL, Host header, and TLS SNI are unchanged, so certificate
validation and vhost routing still target the original hostname; only the
socket destination is pinned.
Delivery now uses a per-request pinned client instead of one shared client,
so close() is a no-op kept for API compatibility. Adds end-to-end tests that
drive the real transport against loopback servers, proving the connect
follows the pin rather than re-resolving the URL host.
Fixes#5146
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
execute_api_call — reachable by the LLM through the api_call agent
tool — joined the integration's user-configured base_url with an
LLM-controlled path and requested it with no IP validation, so a
base_url (or a hostname resolving) into the metadata range
(169.254.169.254) was fetched server-side with the integration's auth
headers attached.
Run check_outbound_url on the joined URL before connecting, matching
the gallery endpoint, embeddings, CardDAV, and reminder webhook
surfaces. Link-local/metadata is always rejected;
INTEGRATION_API_BLOCK_PRIVATE_IPS=true also blocks RFC-1918/loopback.
Private stays allowed by default because LAN integrations
(Home Assistant, Miniflux, ntfy) are the primary use case.
The truncation-test helpers stub the guard open because their
api.example.com fixture host does not resolve and the guard fails
closed on DNS errors.
Fixes#5143
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The grep tool's ripgrep fast-path excluded deny-listed key files with
`--glob "!*<pat>*"` for each entry in _SENSITIVE_FILE_PATTERNS. ripgrep's
--glob is case-sensitive, so on a case-insensitive filesystem (Windows,
default macOS) a key stored under a case variant of its name (ID_RSA,
Known_Hosts, Authorized_Keys) is the same file on disk but slips past the
lowercase exclusion, and ripgrep returns its contents. Those names are
non-dotfiles, so ripgrep's default hidden-file skipping does not cover them
either. The Python fallback already blocks them via the case-folded
_is_sensitive_path (#5097), so the two paths disagreed.
Switch the sensitive-pattern exclusions to --iglob so they match
case-insensitively, mirroring _is_sensitive_path. Add a regression test
that seeds ID_RSA and Known_Hosts and asserts grep returns ordinary
matches but not the key contents.
send_to_session let an authenticated caller reach a null-owner session.
The owner gate was `if owner and sess.owner and sess.owner != owner`, so a
target whose owner is None (legacy rows, or a session created while auth
was off) skipped the check and was read/written by any authenticated user.
list_sessions (get_sessions_for_user) and manage_session already exclude
null-owner sessions from an authenticated caller via an exact owner match,
so this path was the lone inconsistency — the same class of gap the
calendar owner=None fix closed.
Require an exact owner match: `if owner and sess.owner != owner`. Auth-off
(no owner) is unchanged, an exact-owner match still passes, and both
another user's session and a null-owner session are now not-found. Adds a
regression test that an authenticated caller cannot read the transcript of
or write into a null-owner session while single-user access still works.
_join_integration_url built urljoin(base + '/', '') for a bare '/'
path — the minimum execute_api_call accepts — so every request against
a POST-to-base integration went to base_url + '/'. Discord webhook
URLs 404 ('Unknown Webhook') on the trailing-slash variant, which made
the integration look broken even though the stored base URL was
correct.
Resolve a bare '/' (or empty) path to the base URL itself and keep all
other paths joining exactly as before, including deliberate trailing
slashes inside non-empty paths (linkding /api/tags/, Home Assistant
/api/). The reminder webhook sender and the discord_webhook
connectivity test already posted to the bare base URL; execute_api_call
was the remaining path that re-added the slash.
Fixes#5138
The retrieval-timeout branch hard-coded ALWAYS_AVAILABLE, silently skipping
the deterministic keyword hints whenever the embedding backend was slow
(e.g. a remote endpoint cold-loading its model). Queries that named email
or calendar outright lost those tools and the model concluded the
integrations did not exist. Let the timeout fall through to the existing
keyword fallback instead — same baseline, plus the hints.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When an LLM generates a valid JSON string that parses to a native non-dict
type (like a list, int, or string), _parse_tool_args previously returned
that object. Callers expecting a dictionary would then crash with
AttributeError or KeyError when attempting to look up action keys.
- Update _parse_tool_args in src/tool_utils.py to explicitly type-check
the parsed JSON object and return {} for non-dict objects.
- Add test coverage in tests/test_admin_tools_registry.py for lists,
ints, and strings.
pewdiepie-archdaemon