# Attachment References and Upload Storage

> [!NOTE]
> This document records the current attachment-reference and upload-lifecycle
> contract proposed for maintainer acceptance. Source code, tests, and
> configuration remain authoritative for implementation-sensitive behaviour.

Odysseus stores chat and document attachment bytes under the configured upload
directory and passes stable references through chat history, document flows, and
tool context.
The goal is to avoid duplicating large inline media payloads in
`chat_messages.content` or the SQLite FTS index.

## Reference Shape

Attachment references use this minimum shape:

```json
{
  "type": "attachment_ref",
  "attachment_id": "32hex-or-32hex.ext",
  "name": "original-filename.png",
  "mime": "image/png",
  "size": 12345,
  "checksum_sha256": "hex-digest",
  "created_at": "2026-07-09T12:00:00"
}
```

Optional fields such as `width`, `height`, `vision`, `vision_model`, and
`gallery_id` may be present when the uploader or preprocessing path knows them.

## Persistence

The live model call may still receive provider-specific multimodal blocks for
the current turn. Persistence is different:

- `chat_messages.content` stores readable text plus compact attachment reference
  lines, never raw `data:*;base64,...` upload bytes.
- `chat_messages.metadata.attachments` stores structured attachment reference
  metadata for UI reloads and future processing.
- The SQLite FTS migration recreates chat-message FTS triggers so new rows do
  not index inline media payloads, and it scrubs legacy rows that were already
  indexed with data URLs.

## Tool Access

Agent/tool context receives upload entries as `attachment_ref` manifests with an
`odysseus://attachment/<id>` URI and `read_policy: "owner_checked_upload"`.

For compatibility with existing built-in tools, a local `path` may be included
only after all of these checks pass:

- the upload ID resolves through `UploadHandler.resolve_upload`;
- the requested owner is allowed to read the upload;
- the file remains inside the configured upload directory;
- the file path is inside the tool-readable roots.

External MCP/custom tools should treat the URI and attachment ID as the stable
contract and request bytes through an owner-checked server path, not by assuming
host filesystem layout.

## Implementation evidence

The current contract is implemented primarily through:

- `src/upload_handler.py` for upload metadata, owner-aware resolution,
  reservations, cleanup, and deletion;
- `src/attachment_refs.py` for compact persisted references and search-index
  sanitization;
- `src/document_processor.py` for resolving attachments into chat/model context;
- `src/tool_execution.py` for attachment manifests exposed to tools;
- `routes/upload_routes.py` and `routes/document_helpers.py` for upload and
  retrieval paths.

Focused regression coverage includes:

- `tests/test_attachment_refs.py`;
- `tests/test_upload_handler_cleanup.py`;
- `tests/test_replace_messages_upload_reservations.py`;
- `tests/test_resolve_upload_path_nondict.py`;
- `tests/test_chat_preprocess_tool_policy.py`;
- the upload, attachment, and PDF-marker cases in
  `tests/test_security_regressions.py`.

These tests cover compact persistence, owner isolation, path containment,
cleanup safety, reservation-before-write behaviour, and traversal resistance.

## Retention and Deletion

Current retention behavior is conservative:

- uploads are indexed in `uploads.json` with owner, checksum, MIME type, size,
  and creation time;
- admin cleanup first scans persisted chat metadata/content, document versions,
  PDF source markers, gallery hashes, notes, and calendar records for live
  references;
- cleanup fails closed if that reference scan cannot complete, and the lower-level
  cleanup API removes nothing unless it receives a complete reference snapshot;
- expired, unreferenced uploads are removed during the completed scan, while
  attachment-bearing writers must first take an owner-checked reservation that
  serializes with deletion and refreshes the upload's access timestamp;
- deliberate removal atomically drops matching `uploads.json` rows before deleting
  the bytes and restores those rows if filesystem removal fails;
- deleting a chat removes the chat rows but does not immediately delete shared
  upload bytes, because the same upload may also be referenced by gallery items,
  documents, duplicate-upload rows, or future artifact records.

There is no distinct artifact table in the current schema. Artifact-like upload
references persisted in chat or document text are covered by the canonical
attachment-ID scan; any future artifact store must be added to reference discovery
before cleanup is allowed to consider its uploads unreferenced.

Cleanup and write reservations share the upload-index lock. This closes the
scan/write/delete race in the documented single-worker deployment; a future
multi-process deployment must add an inter-process lock or move lifecycle state
into the database before enabling destructive cleanup in more than one worker.
