mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-06-22 04:35:29 -04:00
fix(llm): stop sending llama.cpp slot-affinity fields to cloud providers (#3945)
* fix(llm): stop sending llama.cpp slot-affinity fields to cloud providers _apply_local_cache_affinity adds session_id + cache_prompt for llama.cpp KV-cache slot affinity (#2927), gated on _is_self_hosted_openai_compatible, which treated any unknown OpenAI-compatible host as self-hosted. Strict cloud providers added as custom endpoints (Mistral at api.mistral.ai) reject unknown body fields, so every request failed with 422 extra_forbidden. Self-hosted now also requires the endpoint to resolve as local via model_context.is_local_endpoint: loopback/private/tailscale host, or endpoint kind explicitly configured as "local" (the escape hatch for tunneled self-hosted servers). is_local_endpoint is promoted to a public name since llm_core now shares it. Fixes #3793 * test(llm): sweep cloud OpenAI-compatible hosts in affinity gating Parametrized cases adapted from #3839 (credit: Shabablinchikow): deepseek, x.ai, together, fireworks, and the Gemini OpenAI-compat endpoint must all stay free of the llama.cpp extras, not just the Mistral host from #3793. * fix(llm): narrow the Tailscale range to 100.64.0.0/10 in is_local_endpoint Review finding on #3945: _PRIVATE_PREFIXES carried a bare "100." prefix, treating all of 100.0.0.0/8 as local while Tailscale only uses the CGNAT block 100.64.0.0/10. Public 100.x hosts (e.g. AWS ranges outside the block) were classified local and still received the llama.cpp extras this PR exists to keep away from strict providers. Match the narrowed classification routes/model_routes.py already uses, with boundary tests just below, inside, and just above the range.
This commit is contained in:
committed by
GitHub
parent
f941db29d3
commit
263d41c58a
@@ -11,7 +11,7 @@ import src.model_context as mc
|
||||
|
||||
def _setup(monkeypatch, windows):
|
||||
"""windows: {endpoint_url: context_length}. Force the remote path."""
|
||||
monkeypatch.setattr(mc, "_is_local_endpoint", lambda url: False)
|
||||
monkeypatch.setattr(mc, "is_local_endpoint", lambda url: False)
|
||||
monkeypatch.setattr(mc, "_configured_endpoint_kind", lambda url: "api")
|
||||
monkeypatch.setattr(mc, "_query_context_length", lambda url, model: windows[url])
|
||||
mc._context_cache.clear()
|
||||
|
||||
Reference in New Issue
Block a user