Commit Graph

53 Commits

Author SHA1 Message Date
pewdiepie-archdaemon 6d507f8128 Merge remote-tracking branch 'origin/dev' into test-main-dev-merge-20260615
# Conflicts:
#	src/tool_implementations.py
#	static/js/research/panel.js
2026-06-15 21:20:15 +09:00
pewdiepie-archdaemon 2cbd55b8bd Open email context for agent, email search across All Mail, cookbook serve polish
- Agent: pass the open email reader (uid/folder/account/from/subject/body
  preview) on every chat submit so 'reply to this' / 'write email saying
  hi' route to ui_control open_email_reply with the right UID instead of
  inventing a new .md draft. Code-level enforcement (chat_routes strips
  create_document + send_email when active_email is set); cross-session
  active_doc_id is now trusted instead of being silently dropped.
  set_active_email/clear_active_email tool-layer helpers in
  tool_implementations.

- ui_control open_email_reply: optional body argument so the agent can
  open-and-write in one call; envelope now forwards uid/folder/account/
  body/panel through tool_output. Tool description sharpened and the
  parser rejects empty bodies on reply/reply-all (forces the agent to
  write rather than open an empty draft).

- Email library: search now runs against [Gmail]/All Mail when the
  current folder is INBOX (archived emails surface). Whirlpool spinner
  + 'Searching…' placeholder while in flight. Each search result is
  stamped with its source folder so clicks open the right email instead
  of whatever shares its UID in INBOX. Search no longer re-applies the
  same text pill locally (which only checks subject/from/snippet, never
  body) so body-only matches don't get dropped after IMAP returns them.
  Initial inbox load bumped 100→500.

- Email favorites: 'Favorite (pin to top)' / 'Unfavorite' in both the
  card menu and the open-reader more menu, backed by a new
  /api/email/flag/{uid}?on=true|false endpoint. Flagged emails always
  bubble to the top of the grid regardless of active sort.

- AI reply in doc editor: never overwrites existing draft text or the
  quoted history. AI suggestion is prepended; AI-generated 'On …
  wrote:' re-quotes are stripped so the original quote isn't visually
  edited.

- Cookbook serve: pre-launch GPU driver / has_gpu / install / version-
  floor checks (vllm minimax_m2 needs 0.10.0+, deepseek_r1 needs 0.7.0
  etc.) before the launch chain starts. Detect 'another model already
  running on this host' and offer Stop & launch (with graceful then
  force tmux kill helpers, port release wait). Per-vendor deep-link
  buttons (vLLM recipe / SGLang cookbook) with hardware hash. Backend
  picker is now a custom dropdown with accent-coloured logos for vLLM,
  SGLang, llama.cpp, Ollama, Diffusers; same glyphs added next to
  package names in Dependencies. Runtime-readiness note moved inside
  the panel (green when ready, red when missing) with an × dismiss.
  Esc collapses the expanded card; expanded card scrolls when it
  overflows; Trust Remote / Auto Tool / Reasoning Parser / Enforce
  Eager / Prefix Caching / Expert Parallel / Speculative / MoE Env on
  one row (Reasoning Parser auto-detected per model family).
  Dtype→Row 1, GPUs→Row 2 (rightmost). Removed redundant GPU 'auto'
  input — command builders read from the GPU button strip. Default
  cookbook open is Download tab.

- Cookbook hwfit: 'Model (latest)' / 'Model (oldest)' header sorts by
  release_date; release dates can be backfilled with the new
  scripts/backfill_model_release_dates.py and recipe metadata pulled
  with scripts/import_from_vllm_recipes.py against the upstream
  vllm-project/recipes catalog (vllm_recipe + min_vllm_version stamped
  on entries).

- Calendar: Quick add hint cycles a random Odysseus-themed example per
  open (wooden horse Friday, crew muster 10am daily, council on
  Ithaca, …). Typing a time like '11pm' in the event title updates
  the hero clock live.

- Doc editor: email-mode Reply button (sparkle icon, accent) opens the
  same Fast/Full + context popover the email reader uses; Ctrl+Alt+M
  toggles markdown preview.

- Memories panel: custom sort picker with per-option icons, default
  'Latest', visible Enabled/Disabled toggle text matching the section
  description style.
2026-06-15 20:47:51 +09:00
spooky f23e2e6ffb docs: add agent migration manifest helper (#3028)
* docs: add agent migration manifest helper

* fix: use stat+streamed hash for metadata-only archive scans

When include_content is false, skip reading full file content and
only stat+stream-hash for size and sha256. Avoids spurious skipped-
content warnings and keeps large-export previews fast and clean.

Closes review feedback on PR #3028.

* fix: skip symlinked migration inputs

* fix: stream archive traversal warnings

* feat: stage conversation threads in agent migration manifests
2026-06-15 15:57:33 +09:00
Mike ac94885c84 refactor(constants): single source of truth for data dir (#3368)
* refactor(constants): single source of truth for data dir + merge core/src constants

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(contributing): use named src.constants for data paths, drop core/constants references

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 09:58:52 +02:00
horribleCodes 9c90f62657 fix(platform): Improve WSL SSH remote compatibility (#3316)
* fix(platform): add WSL compatibility functions and path translation
fix(cookbook): enhance model scan script to support additional HuggingFace cache paths
fix(hardware): improve cache key generation for remote SSH context
test(tests): add tests for WSL detection and path translation functionality

* fix(cookbook): prefer prebuilt wheels for llama-cpp-python and normalize package aliases

* fix: enable StrictHostKeyChecking in nvidia probe
refactor: consolidate ssh & powershell command execution to utility functions in core module
refactor: consolidate nvidia path candidates in to single variables in core module
tests: add tests for new utility functions

* fix: correct wrong variable name
2026-06-08 00:33:50 +02:00
Alexandre Teixeira 9ad6a2809e test(diffusion-server): exercise security middleware wiring (#3214) 2026-06-07 23:42:11 +02:00
@aaronjmars 108ee1e32b fix(security): close DNS-rebinding hole on diffusion_server (wildcard CORS + missing Host check) (#347)
* fix(security): close DNS-rebinding hole on diffusion_server

scripts/diffusion_server.py used to ship `allow_origins=["*"]` with the
default `--host=127.0.0.1` bind. Combined, that left the OpenAI-compatible
image API reachable from any browser tab via DNS-rebinding: an attacker page
resolves its own domain to 127.0.0.1 mid-fetch, the browser forwards the
request to the loopback server, the server processes it (no Host check), and
the wildcard CORS reply lets the attacker page read the result + drive the
GPU. CWE-346 + CWE-942 + CWE-352 (DNS-rebinding bridge).

Fix:
  - Drop the wildcard CORS at module load (default-deny).
  - Install `TrustedHostMiddleware` with a loopback allowlist so DNS-rebound
    requests are rejected by the middleware before any route runs.
  - Add additive `--allowed-host` / `--allowed-origin` CLI flags so operators
    who need browser access on a specific origin can opt in explicitly without
    re-introducing the wildcard.

Tests: tests/test_diffusion_server_security.py (9 cases) pin the allowlist
helpers, the default-deny CORS behavior, and the live middleware paths via
Starlette's TestClient.

Detected by Aeon + semgrep + manual review.
Severity: medium.
CWE-346 / CWE-942 / CWE-352.

* test(diffusion-server): drive ASGI app via httpx, not TestClient portal

The TrustedHost/CORS integration tests used `with TestClient(app) as
client:`, whose context-manager form spins up an anyio blocking portal to
run the app lifespan. Under the repo's pytest setup (anyio plugin active, a
stray asyncio_mode option, no pytest-asyncio) that portal deadlocks —
`test_trusted_host_middleware_rejects_attacker_host` hung indefinitely in
review before emitting any assertion output.

Replace the TestClient usage with a tiny _asgi_get() helper that drives the
ASGI app over httpx.ASGITransport on a fresh event loop (asyncio.run). No
portal, no lifespan, no dependency on the host project's async test plugins.
Host is taken from the request URL so TrustedHostMiddleware sees the exact
hostname under test; Origin goes through headers. Assertions are unchanged.

Focused test now passes in 0.12s; full file 9 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: aeonframework <aeonframework@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 23:34:39 +01:00
tanmayraut45 17b62a3dba Research CLI: alias --status complete to the stored done value (#2515)
`odysseus-research list --status complete` returns an empty result on
any real corpus. The CLI accepts `complete` as a `--status` choice (the
user-facing label), but the writer in
`services/research/research_handler.py` stores `status="done"` when a
run finishes (and the legacy `src/research_handler.py` copy does the
same). The list filter at `scripts/odysseus-research` was a literal
string compare:

    if args.status and (data.get("status") or "") != args.status:
        continue

so `--status complete` filtered every finished record out, and the user
saw nothing — even though `odysseus-research list` (no filter) listed
them fine and `show RP_ID` worked on the same files. The other
documented choices — `running`, `cancelled`, `error` — are stored
verbatim by the writer, so the surface mismatch is just on `complete`.

Add a small `_STATUS_CLI_TO_STORED = {"complete": "done"}` map and run
`data.get("status")` through `_status_matches(...)` before comparing.
The other CLI choices fall through unchanged, so the filter still
matches them verbatim. A `None` or non-string `status` (corrupt JSON)
is coerced to `""` and never matches `complete`, so a half-written
record can't sneak past the filter.

`tests/test_research_cli_status_filter.py` covers all four documented
choices, the non-string / missing status case, and pins that the
verbatim choices are NOT rewritten — a blanket mapping that turned
every CLI choice into a stored variant would just re-introduce the
empty-result bug on the running/cancelled/error paths.

Part of #2122.
2026-06-05 08:50:33 +01:00
Alexandre Teixeira f2b11ba94e tools: add read-only PR blocker audit helper
Adds a standalone read-only PR blocker audit helper with Markdown, terminal, and JSON output plus focused tests and documentation.
2026-06-04 12:51:48 +01:00
red person 42ef4b6502 Skip invalid research CLI records (#1394) 2026-06-03 14:12:38 +09:00
red person 0e27a574b7 Reject invalid theme CLI prefs (#1396) 2026-06-03 14:12:35 +09:00
red person dfbc94f929 Reject invalid cookbook CLI state (#1531) 2026-06-03 14:11:56 +09:00
red person 2f6d339073 Ignore invalid note CLI items (#1539) 2026-06-03 14:11:53 +09:00
red person 63aac10341 Skip invalid FAISS migration JSON (#1547) 2026-06-03 14:11:49 +09:00
red person 708ac19f28 Skip invalid memory CLI rows (#1552) 2026-06-03 14:11:42 +09:00
red person 83f602e6d1 Skip invalid skills CLI rows (#1553) 2026-06-03 14:11:38 +09:00
red person f549058369 Normalize stored MCP CLI JSON (#1554) 2026-06-03 14:11:35 +09:00
red person ab7145de83 Mask short webhook CLI tokens (#1558) 2026-06-03 14:11:28 +09:00
red person 9e91a172e7 Handle missing gallery album images (#1563) 2026-06-03 14:11:24 +09:00
red person 04e7441d78 Skip invalid contacts CLI rows (#1569) 2026-06-03 14:11:21 +09:00
red person 89b04675e2 Handle missing calendar CLI relation (#1574) 2026-06-03 14:11:17 +09:00
red person ade755b184 Let preset set replace corrupt entries (#1650) 2026-06-03 14:10:58 +09:00
red person 40e1d6e876 Reject non-PNG signature export data (#1651) 2026-06-03 14:10:54 +09:00
red person d8f5c04340 Skip invalid ownerless JSON rows (#1540) 2026-06-03 14:06:57 +09:00
red person 815bdf57d5 Ignore non-string task CLI previews (#1559) 2026-06-03 14:06:49 +09:00
red person 347b193af8 Ignore non-string docs CLI content lengths (#1561) 2026-06-03 14:06:46 +09:00
red person 3b9c601498 Skip invalid personal CLI index rows (#1571) 2026-06-03 14:06:42 +09:00
Afonso Coutinho d9e6071528 fix: odysseus-mail read crashes on an empty IMAP fetch payload (#1730) 2026-06-03 13:31:10 +09:00
Afonso Coutinho 6b2618dab4 fix: logs CLI _resolve crashes on a non-string name (#1631) 2026-06-03 08:59:30 +09:00
red person df3864bd15 Normalize session CLI counters (#1578)
* Normalize session CLI counters

* Keep sessions CLI test imports isolated
2026-06-03 08:57:41 +09:00
red person ffeb7d8c97 Reject invalid preset CLI entries (#1579)
* Reject invalid preset CLI entries

* Use modern preset CLI test loader
2026-06-03 08:57:35 +09:00
red person a6b7a7bc60 Validate signature CLI PNG data (#1580)
* Validate signature CLI PNG data

* Keep signature CLI test imports isolated
2026-06-03 08:57:28 +09:00
red person 0cc1814658 Reject empty mail CLI recipients (#1581)
* Reject empty mail CLI recipients

* Keep mail CLI test imports isolated
2026-06-03 08:57:23 +09:00
red person 953305a5af Remove duplicate update database body (#1584) 2026-06-03 08:57:03 +09:00
red person 15a3b71802 Require runnable dispatcher subcommands (#1585)
* Require runnable dispatcher subcommands

* Use modern dispatcher test loader
2026-06-03 08:56:56 +09:00
red person e68d0448b8 Parse all AMD GPU check args (#1586) 2026-06-03 08:56:48 +09:00
red person db3a5c17b0 Reject backup output inside data dir (#1587) 2026-06-03 08:38:27 +09:00
Afonso Coutinho 19b6cbac12 fix: skills CLI summary crashes on a non-string description (#1595) 2026-06-03 08:37:05 +09:00
Afonso Coutinho 258fe455eb fix: research CLI summary crashes on a non-string query (#1596) 2026-06-03 08:36:57 +09:00
Afonso Coutinho 667c663668 fix: gallery CLI image serialization crashes on a non-string prompt (#1598) 2026-06-03 08:36:51 +09:00
Afonso Coutinho 0d88c9989e fix: mcp CLI _serialize crashes when stored env JSON is a list (#1609) 2026-06-03 08:35:09 +09:00
red person abbc073429 Reject invalid preset CLI stores (#1395) 2026-06-03 03:59:05 +09:00
Afonso Coutinho 8852c7ea4a fix: claim_ownerless actually claims ownerless documents (was a no-op self-update) (#1288) 2026-06-03 01:38:38 +09:00
spooky 18a445ba22 docs: add AMD Docker GPU preflight (#1168) 2026-06-02 22:54:08 +09:00
tanmayraut45 4e440a9fd5 Hwfit: estimate params from config.json fallback
`add_hwfit_models.py` infers `parameter_count` and `parameters_raw` by
regexing the HF repo name for a `<num>B` token, optionally with an
`-A<num>B` MoE active-param suffix. Repos that don't encode a size in
their name at all (e.g. `zai-org/GLM-4.5`, where the "4.5" is a version
not a parameter count) fall through to the safetensors element-count
path. That path works for unquantized FP16 / BF16 repos but is brittle
in two cases the catalog hits often:

1. Author-bulk runs (`AUTHORS = ["cyankiwi"]`) pull pre-quantized AWQ /
   GPTQ / MLX repos. The safetensors metadata stores the packed I32
   tensors and a per-dtype `parameters` map, which the script unpacks
   via a per-quant pack factor. When the upload doesn't populate that
   map (older repos, custom shards), `st.total` is used raw and the
   parameter count is off by 4-8x.
2. Repos where the safetensors block is absent from `model_info()`
   entirely. The current code returns `None` and silently drops the
   model, which then has to be added to `EXTRA_REPOS` by hand with a
   literal `parameter_count` string.

Both are exactly what the issue calls out — the regex / safetensors
combo can't size GLM-4.5 by itself because the name has no `<num>B`
and the upstream repo's safetensors block doesn't carry a usable param
total either.

Add a config.json fallback in front of the safetensors path:

- `_fetch_config_json(repo_id)` downloads `config.json` via
  `hf_hub_download` (so the standard HF on-disk cache handles
  deduplication across runs, no extra cache layer needed). Network /
  404 / gated-repo errors return `None` and the caller proceeds to the
  safetensors fallback. An in-process `_CONFIG_CACHE` dedupes the
  base-model vs. source-repo lookups within a single run.
- `_params_from_config(cfg)` first honours explicit `num_parameters` /
  `n_params` / `total_params` fields when present. Otherwise it sums
  embeddings + attention (GQA-aware via `num_key_value_heads` and
  `head_dim`) + dense MLP (`3 * hidden_size * intermediate_size`,
  covering SwiGLU / GeGLU). For MoE configs it picks up both naming
  conventions in the wild — `num_experts` / `num_experts_per_tok`
  (Qwen3-MoE) and `n_routed_experts` / `n_shared_experts` (GLM-4-MoE,
  DeepSeek-V3) — uses `moe_intermediate_size`, and respects
  `first_k_dense_replace` so the first N layers stay dense. Active
  parameters come out as `num_experts_per_tok + n_shared_experts` of
  the routed experts, which matches how each architecture reports its
  active count.
- In `_entry_from_modelinfo`, try config.json on the source repo first
  (works for unquantized models) and then on the `base_model:` parent
  (covers AWQ / GPTQ children whose own config is just a quantization
  manifest). Both lookups run only when regex + override + base_model
  tag all failed, so the normal author-bulk run still resolves sizes
  from names without touching the Hub.

Spot-checks against the three architecture families this script
actually pulls — within ~5% of the documented param counts, which is
well inside the `parameter_count` rounding (one decimal of "B") and
the `min_vram_gb` downstream bucket:

  Qwen2.5-7B-Instruct      7.62B   (HF card: 7.6B)
  Qwen3-30B-A3B            30.5B / 3.34B active   (card: 30.5B / 3.3B)
  GLM-4.5                  352.7B / 33.6B active  (card: 355B / 32B)

The safetensors path is unchanged and remains the last resort, so
repos with neither a parsable name nor a fetchable config.json behave
exactly as before.

Closes #955.
2026-06-02 20:33:25 +09:00
spooky cd4f496cb4 Fix native Cookbook quant classification 2026-06-02 13:07:20 +09:00
Alexandre Teixeira 8455b88643 Improve Docker GPU setup diagnostics (#705)
* Improve Docker GPU setup diagnostics

Add a Docker GPU preflight script for NVIDIA users. The script is
read-only by default, checks host NVIDIA drivers, Docker availability,
and container GPU passthrough, and prints actionable next steps.

Add explicit opt-in modes to print install commands, install NVIDIA
Container Toolkit on Ubuntu/Debian, and enable the NVIDIA Compose overlay
in .env after passthrough is verified.

Document common NVIDIA Docker failure modes, ignore generated .env
backups, and clarify that Cookbook can only detect GPUs exposed to the
Odysseus container.

* Clarify Docker GPU diagnostic limits
2026-06-02 12:30:40 +09:00
pewdiepie-archdaemon 966b53df77 Improve Cookbook serve diagnostics and recommendations 2026-06-02 12:15:47 +09:00
ghreprimand 491a8a5480 Harden backup restore tar extraction
Co-authored-by: ghreprimand <203024559+ghreprimand@users.noreply.github.com>
2026-06-02 05:55:03 +09:00
Sirsyorrz 9955f5bc95 Fix VRAM estimates for pre-quantized HF repos
The Cookbook fit scanner was reporting impossibly low VRAM requirements
for some pre-quantized models — e.g. cyankiwi/Qwen3-Coder-Next-REAM-AWQ-4bit
shown as 7.1 GB ('perfect' on a 12 GB card) when the real load is ~40 GB.

Root cause is in the catalog builder. When _entry_from_modelinfo falls
back to safetensors metadata for the parameter count, it stored
safetensors.total directly. For pre-quantized repos that figure reflects
*packed* element counts: AWQ/GPTQ-Int4 pack 8x 4-bit weights into one
I32, AWQ-8bit/GPTQ-Int8/FP8 pack 4x. The catalog therefore recorded
~1/8 of the real parameter count, and min_vram_gb = packed * bpp
double-applied the quantization.

Fix the safetensors fallback:

* prefer the per-dtype parameters dict when available and unpack only the
  I32/I64 entries (the F16/BF16 scale/zero tensors and embeddings are
  already at their real element counts)
* fall back to total * pack_factor when only total is exposed

Patch the catalog entries that were affected by the old fallback so the
fit ratings reflect reality without waiting for a full catalog rebuild:

* cyankiwi/Qwen3-Coder-Next-REAM-AWQ-4bit  11.4B -> 79.7B (40.8 GB VRAM)
* stelterlab/Qwen3-Coder-30B-A3B-Instruct-AWQ  4.6B -> 30.5B
* stelterlab/NVIDIA-Nemotron-3-Nano-30B-A3B-AWQ  5.1B -> 30.5B
* warshanks/Qwen3-8B-abliterated-AWQ  2.2B -> 8.2B
* QuantTrio/sarvam-30b-AWQ  7B -> 30B
* QuantTrio/sarvam-105b-AWQ  19B -> 105B

Closes #377.
2026-06-01 18:32:58 +09:00