mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-08-01 19:18:35 -04:00
Alexandre Teixeira
118 lines
5.0 KiB
Markdown
118 lines
5.0 KiB
Markdown
# Attachment References and Upload Storage
|
|
|
|
> [!NOTE]
|
|
> This document records the current attachment-reference and upload-lifecycle
|
|
> contract proposed for maintainer acceptance. Source code, tests, and
|
|
> configuration remain authoritative for implementation-sensitive behaviour.
|
|
|
|
Odysseus stores chat and document attachment bytes under the configured upload
|
|
directory and passes stable references through chat history, document flows, and
|
|
tool context.
|
|
The goal is to avoid duplicating large inline media payloads in
|
|
`chat_messages.content` or the SQLite FTS index.
|
|
|
|
## Reference Shape
|
|
|
|
Attachment references use this minimum shape:
|
|
|
|
```json
|
|
{
|
|
"type": "attachment_ref",
|
|
"attachment_id": "32hex-or-32hex.ext",
|
|
"name": "original-filename.png",
|
|
"mime": "image/png",
|
|
"size": 12345,
|
|
"checksum_sha256": "hex-digest",
|
|
"created_at": "2026-07-09T12:00:00"
|
|
}
|
|
```
|
|
|
|
Optional fields such as `width`, `height`, `vision`, `vision_model`, and
|
|
`gallery_id` may be present when the uploader or preprocessing path knows them.
|
|
|
|
## Persistence
|
|
|
|
The live model call may still receive provider-specific multimodal blocks for
|
|
the current turn. Persistence is different:
|
|
|
|
- `chat_messages.content` stores readable text plus compact attachment reference
|
|
lines, never raw `data:*;base64,...` upload bytes.
|
|
- `chat_messages.metadata.attachments` stores structured attachment reference
|
|
metadata for UI reloads and future processing.
|
|
- The SQLite FTS migration recreates chat-message FTS triggers so new rows do
|
|
not index inline media payloads, and it scrubs legacy rows that were already
|
|
indexed with data URLs.
|
|
|
|
## Tool Access
|
|
|
|
Agent/tool context receives upload entries as `attachment_ref` manifests with an
|
|
`odysseus://attachment/<id>` URI and `read_policy: "owner_checked_upload"`.
|
|
|
|
For compatibility with existing built-in tools, a local `path` may be included
|
|
only after all of these checks pass:
|
|
|
|
- the upload ID resolves through `UploadHandler.resolve_upload`;
|
|
- the requested owner is allowed to read the upload;
|
|
- the file remains inside the configured upload directory;
|
|
- the file path is inside the tool-readable roots.
|
|
|
|
External MCP/custom tools should treat the URI and attachment ID as the stable
|
|
contract and request bytes through an owner-checked server path, not by assuming
|
|
host filesystem layout.
|
|
|
|
## Implementation evidence
|
|
|
|
The current contract is implemented primarily through:
|
|
|
|
- `src/upload_handler.py` for upload metadata, owner-aware resolution,
|
|
reservations, cleanup, and deletion;
|
|
- `src/attachment_refs.py` for compact persisted references and search-index
|
|
sanitization;
|
|
- `src/document_processor.py` for resolving attachments into chat/model context;
|
|
- `src/tool_execution.py` for attachment manifests exposed to tools;
|
|
- `routes/upload_routes.py` and `routes/document_helpers.py` for upload and
|
|
retrieval paths.
|
|
|
|
Focused regression coverage includes:
|
|
|
|
- `tests/test_attachment_refs.py`;
|
|
- `tests/test_upload_handler_cleanup.py`;
|
|
- `tests/test_replace_messages_upload_reservations.py`;
|
|
- `tests/test_resolve_upload_path_nondict.py`;
|
|
- `tests/test_chat_preprocess_tool_policy.py`;
|
|
- the upload, attachment, and PDF-marker cases in
|
|
`tests/test_security_regressions.py`.
|
|
|
|
These tests cover compact persistence, owner isolation, path containment,
|
|
cleanup safety, reservation-before-write behaviour, and traversal resistance.
|
|
|
|
## Retention and Deletion
|
|
|
|
Current retention behavior is conservative:
|
|
|
|
- uploads are indexed in `uploads.json` with owner, checksum, MIME type, size,
|
|
and creation time;
|
|
- admin cleanup first scans persisted chat metadata/content, document versions,
|
|
PDF source markers, gallery hashes, notes, and calendar records for live
|
|
references;
|
|
- cleanup fails closed if that reference scan cannot complete, and the lower-level
|
|
cleanup API removes nothing unless it receives a complete reference snapshot;
|
|
- expired, unreferenced uploads are removed during the completed scan, while
|
|
attachment-bearing writers must first take an owner-checked reservation that
|
|
serializes with deletion and refreshes the upload's access timestamp;
|
|
- deliberate removal atomically drops matching `uploads.json` rows before deleting
|
|
the bytes and restores those rows if filesystem removal fails;
|
|
- deleting a chat removes the chat rows but does not immediately delete shared
|
|
upload bytes, because the same upload may also be referenced by gallery items,
|
|
documents, duplicate-upload rows, or future artifact records.
|
|
|
|
There is no distinct artifact table in the current schema. Artifact-like upload
|
|
references persisted in chat or document text are covered by the canonical
|
|
attachment-ID scan; any future artifact store must be added to reference discovery
|
|
before cleanup is allowed to consider its uploads unreferenced.
|
|
|
|
Cleanup and write reservations share the upload-index lock. This closes the
|
|
scan/write/delete race in the documented single-worker deployment; a future
|
|
multi-process deployment must add an inter-process lock or move lifecycle state
|
|
into the database before enabling destructive cleanup in more than one worker.
|