Claude
ee585ea65c
Ollama was defaulting to a 4096-token context (vs. the model's 32768 training context) and silently truncating any oversized prompt by dropping content from the middle, with no error surfaced anywhere — observed losing ~53% of a merge-cluster prompt in production. Two prompt builders (buildPrompt/buildRecapPrompt) concatenated all source summaries/article bodies with no size cap, so a cluster with enough sources (or a recap spanning enough articles) could easily exceed the window. Fix: OllamaProvider.generate() now always sends explicit num_ctx/ num_predict options (sized for CPU-only inference — i5-6600K, no GPU, ~17 tok/s prompt processing) instead of leaving Ollama to pick a default. synthesis.ts now caps prompt size itself before it ever reaches Ollama, giving each source/article an equal character budget and trimming individual entries rather than dropping whole ones off the end — every source stays at least partially represented and attributable. Trims are logged via the existing admin log stream instead of failing silently.
7 lines
301 B
TypeScript
7 lines
301 B
TypeScript
export interface InferenceProvider {
|
|
generate(prompt: string, opts?: { model?: string; system?: string; numCtx?: number; numPredict?: number }): Promise<string>;
|
|
embed(text: string, opts?: { model?: string }): Promise<number[]>;
|
|
listModels(): Promise<string[]>;
|
|
isReachable(): Promise<boolean>;
|
|
}
|