Initial Commit

backend/.env.example

@@ -0,0 +1,10 @@
+PORT=4000
+FRONTEND_ORIGIN=http://localhost:5173
+DB_PATH=./data/homefeed.db
+MEDIA_DIR=./data/media
+
+# Seeded once on first run — change the password after logging in.
+ADMIN_USERNAME=admin
+ADMIN_PASSWORD=change-me-immediately
+
+NODE_ENV=development

backend/.gitignore

@@ -0,0 +1,4 @@
+node_modules
+dist
+data
+.env

backend/README.md

@@ -0,0 +1,89 @@
+# Homefeed backend
+
+Node.js + TypeScript, per `project-structure.md`. Ingests RSS/API/Telegram sources,
+clusters and synthesizes them into merged articles via a self-hosted Ollama instance,
+and serves the same `/api/feed`, `/api/article/:id`, `/api/tags`, `/api/events`
+contract the frontend already consumes from the mock backend — plus the full
+`/api/admin/*` surface behind session auth.
+
+## Setup
+
+```bash
+cp .env.example .env
+# edit .env — at minimum set ADMIN_PASSWORD to something real
+npm install
+npm run dev
+```
+
+Runs on `:4000` by default. Point the frontend's `VITE_BACKEND_URL` at it instead of
+the mock backend and everything else keeps working unchanged — same API contract.
+
+You'll also need a running Ollama instance (see `AI_SERVICE_HOST`/`AI_SERVICE_PORT` in
+the admin panel's Connections tab, or `PATCH /api/admin/settings` directly) with at
+minimum an embedding model (e.g. `nomic-embed-text`) and a generation model (e.g.
+`qwen2.5:7b-instruct-q4_K_M`) pulled. Without it reachable, ingestion still runs, but
+the synthesis tick quietly skips each cycle until the AI service comes back — nothing
+crashes, articles just don't get published.
+
+## Behavior before Ollama is set up
+
+Per the "assume Ollama arrives after the backend launches" requirement: ingestion and
+publishing don't wait for it.
+
+- **Adding a source polls it immediately**, not on the next scheduler tick — you see
+  results right away instead of waiting up to a minute.
+- **With no AI service reachable**, the synthesis tick falls back to publishing each
+  item directly once it clears the hold-before-publish window — no rewriting, no
+  cross-source merging, no tags (there's no LLM to extract them yet), but the page
+  populates instead of staying empty. Media (images) still downloads normally, since
+  that never needed AI in the first place.
+- **Once Ollama becomes reachable**, the real pipeline (embed → cluster → synthesize →
+  tag) takes back over for anything ingested from that point on. Articles already
+  published via the passthrough path aren't retroactively rewritten or merged — they
+  stand as they are, but new related coverage can still link to them as a follow-up
+  via the normal tag-based thread detection.
+- **Tracked-event recaps are the one thing that still waits for AI** — summarizing a
+  day's worth of Telegram messages isn't something worth faking with a passthrough;
+  those simply don't fire until the AI service is available.
+
+## What's real vs. stubbed
+
+Built and verified end-to-end (see the test run in this conversation — real SQLite,
+real RSS parsing, real HTTP calls to a stub Ollama server, real media download to
+disk, real tag dedup across separate synthesis calls):
+
+- SQLite schema + repository layer for every entity in `homefeed-data-schema.md`
+- Session auth (scrypt password hashing, httpOnly cookie, CORS locked to the
+  configured frontend origin) protecting all `/api/admin/*` routes
+- RSS adapter (real parsing, images/video extraction) and a generic JSON API adapter
+  (configurable field mapping)
+- Poller respecting per-source poll intervals
+- Embedding → clustering → synthesis → tag extraction/dedup → image selection →
+  media download → publish pipeline, talking to Ollama over HTTP
+- Priority queue ordering by admin-ranked category before clustering
+- Retention sweep (age-based for both raw items and published articles, size-based
+  FIFO cap, tag expiry) running hourly
+- Full admin API: settings, sources, tracked events, live model catalog from Ollama,
+  AI service status
+
+**Stubbed / simplified, worth knowing before relying on it:**
+
+- **Telegram adapter** (`ingestion/adapters/telegram.ts`) is a stub — logs a warning
+  and returns nothing. Needs a real implementation (grammY, per the architecture doc)
+  keyed off `source.config.telegramChannelId`.
+- **Image selection** picks the highest-resolution candidate rather than using a
+  vision model to judge relevance — cheap and deterministic, but doesn't catch a
+  technically-large image that's actually a poor choice (watermarked, wrong subject).
+  A vision-model pass is a natural upgrade once this needs to be smarter.
+- **Follow-up thread detection** matches on shared tags within a 3-day lookback rather
+  than comparing article-level embeddings directly. Works, but a cluster with no tags
+  extracted can't be linked to a prior thread at all.
+- **Storage-cap FIFO eviction** deletes oldest articles but doesn't yet cascade-delete
+  their associated media rows/files in the same pass — there's a comment in
+  `queue/retention.ts` explaining the gap and the backstop that partially covers it.
+- **`node:sqlite`** is still an experimental Node API. If a future Node version changes
+  its behavior, `better-sqlite3` is a drop-in-shaped alternative (same synchronous
+  `.prepare().run()/.get()/.all()` style).
+
+None of these block running the backend end-to-end — they're the specific spots worth
+hardening next, roughly in the order listed.

backend/package.json

@@ -0,0 +1,25 @@
+{
+  "name": "homefeed-backend",
+  "version": "1.0.0",
+  "description": "Homefeed backend — ingestion, clustering, synthesis, and the public/admin API",
+  "type": "module",
+  "main": "dist/index.js",
+  "scripts": {
+    "dev": "tsx watch --env-file=.env src/index.ts",
+    "build": "tsc",
+    "start": "node --experimental-sqlite dist/index.js",
+    "typecheck": "tsc --noEmit"
+  },
+  "license": "UNLICENSED",
+  "dependencies": {
+    "@fastify/cookie": "^11.1.1",
+    "@fastify/cors": "^11.3.0",
+    "fastify": "^5.10.0",
+    "rss-parser": "^3.13.0"
+  },
+  "devDependencies": {
+    "@types/node": "^26.1.1",
+    "tsx": "^4.23.0",
+    "typescript": "^7.0.2"
+  }
+}

backend/src/api/admin.ts

@@ -0,0 +1,112 @@
+import type { FastifyInstance } from 'fastify';
+import * as settingsDb from '../storage/db/settings.js';
+import * as sourcesDb from '../storage/db/sources.js';
+import * as eventsDb from '../storage/db/events.js';
+import * as categoriesDb from '../storage/db/categories.js';
+import { OllamaProvider } from '../inference/ollama-provider.js';
+import { pollSourceNow } from '../ingestion/poller.js';
+import { logger, listLogs } from '../storage/db/logs.js';
+
+export async function registerAdminRoutes(app: FastifyInstance) {
+	// --- Settings ---
+	app.get('/api/admin/settings', async () => {
+		const settings = settingsDb.getSettings();
+		return { ...settings, categoryPriority: categoriesDb.listCategories() };
+	});
+
+	app.patch('/api/admin/settings', async (req) => {
+		const body = req.body as any;
+		if (body.categoryPriority) {
+			categoriesDb.setCategoryOrder(body.categoryPriority);
+			delete body.categoryPriority;
+		}
+		const settings = settingsDb.updateSettings(body);
+		return { ...settings, categoryPriority: categoriesDb.listCategories() };
+	});
+
+	// --- Sources ---
+	app.get('/api/admin/sources', async () => sourcesDb.listSources());
+
+	app.post('/api/admin/sources', async (req, reply) => {
+		const created = sourcesDb.createSource(req.body as any);
+		// Poll immediately rather than waiting for the next scheduler tick (up to 1 minute)
+		// — the admin adding a feed expects to see it start working right away.
+		pollSourceNow(created).catch((err) => logger.error('poller', `Immediate poll failed for "${created.name}": ${err.message}`));
+		return reply.code(201).send(created);
+	});
+
+	app.patch('/api/admin/sources/:id', async (req, reply) => {
+		const { id } = req.params as { id: string };
+		const updated = sourcesDb.updateSource(id, req.body as any);
+		if (!updated) return reply.code(404).send({ error: 'not found' });
+		return updated;
+	});
+
+	app.delete('/api/admin/sources/:id', async (req, reply) => {
+		const { id } = req.params as { id: string };
+		sourcesDb.deleteSource(id);
+		return reply.code(204).send();
+	});
+
+	// Manual "poll now" — the refresh icon on each source in the admin panel.
+	app.post('/api/admin/sources/:id/poll', async (req, reply) => {
+		const { id } = req.params as { id: string };
+		const source = sourcesDb.getSource(id);
+		if (!source) return reply.code(404).send({ error: 'not found' });
+
+		const ingested = await pollSourceNow(source);
+		logger.info('poller', `Manual poll of "${source.name}" — ${ingested} new item(s)`);
+		return { ingested, source: sourcesDb.getSource(id) };
+	});
+
+	// --- Tracked events ---
+	app.get('/api/admin/events', async () => eventsDb.listEvents());
+
+	app.post('/api/admin/events', async (req, reply) => {
+		const created = eventsDb.createEvent(req.body as any);
+		return reply.code(201).send(created);
+	});
+
+	app.patch('/api/admin/events/:id', async (req, reply) => {
+		const { id } = req.params as { id: string };
+		const updated = eventsDb.updateEvent(id, req.body as any);
+		if (!updated) return reply.code(404).send({ error: 'not found' });
+		return updated;
+	});
+
+	app.delete('/api/admin/events/:id', async (req, reply) => {
+		const { id } = req.params as { id: string };
+		eventsDb.deleteEvent(id);
+		return reply.code(204).send();
+	});
+
+	// --- Models / AI service (fetched live from the configured Ollama host) ---
+	app.get('/api/admin/models', async (_req, reply) => {
+		const settings = settingsDb.getSettings();
+		const provider = new OllamaProvider(settings.aiServiceHost, settings.aiServicePort);
+		try {
+			const models = await provider.listModels();
+			// Ollama doesn't distinguish task type, so the catalog surfaces the full list
+			// for each dropdown — the admin picks which installed model to use for what.
+			return { embedding: models, image: models, synthesis: models };
+		} catch (err) {
+			return reply.code(502).send({ error: `AI service unreachable: ${(err as Error).message}` });
+		}
+	});
+
+	app.get('/api/admin/ai-status', async () => {
+		const settings = settingsDb.getSettings();
+		const provider = new OllamaProvider(settings.aiServiceHost, settings.aiServicePort);
+		const connected = await provider.isReachable();
+		return { connected, host: settings.aiServiceHost, port: settings.aiServicePort, ramGB: null, gpu: null };
+	});
+
+	// --- Logs ---
+	app.get('/api/admin/logs', async (req) => {
+		const { level, limit } = req.query as { level?: string; limit?: string };
+		return listLogs({
+			level: level === 'info' || level === 'warn' || level === 'error' ? level : undefined,
+			limit: limit ? Number(limit) : undefined
+		});
+	});
+}

backend/src/api/auth.ts

@@ -0,0 +1,45 @@
+import type { FastifyInstance } from 'fastify';
+import { getAdminUserByUsername, createSession, isSessionValid, deleteSession } from '../storage/db/auth.js';
+import { verifyPassword } from './password.js';
+
+const SESSION_COOKIE = 'homefeed_session';
+
+export async function registerAuth(app: FastifyInstance) {
+	app.post('/api/admin/login', async (req, reply) => {
+		const { username, password } = req.body as { username?: string; password?: string };
+		if (!username || !password) return reply.code(400).send({ error: 'username and password required' });
+
+		const user = getAdminUserByUsername(username);
+		if (!user || !verifyPassword(password, user.password_hash)) {
+			// Deliberately generic — doesn't reveal whether the username exists.
+			return reply.code(401).send({ error: 'invalid credentials' });
+		}
+
+		const session = createSession(req.ip ?? null);
+		reply.setCookie(SESSION_COOKIE, session.id, {
+			httpOnly: true,
+			secure: process.env.NODE_ENV === 'production',
+			sameSite: process.env.NODE_ENV === 'production' ? 'none' : 'lax',
+			path: '/',
+			expires: new Date(session.expiresAt)
+		});
+		return { ok: true };
+	});
+
+	app.post('/api/admin/logout', async (req, reply) => {
+		const sessionId = req.cookies[SESSION_COOKIE];
+		if (sessionId) deleteSession(sessionId);
+		reply.clearCookie(SESSION_COOKIE, { path: '/' });
+		return { ok: true };
+	});
+
+	// Guards every /api/admin/* route except login itself.
+	app.addHook('preHandler', async (req, reply) => {
+		if (!req.url.startsWith('/api/admin/') || req.url === '/api/admin/login') return;
+
+		const sessionId = req.cookies[SESSION_COOKIE];
+		if (!sessionId || !isSessionValid(sessionId)) {
+			return reply.code(401).send({ error: 'unauthorized' });
+		}
+	});
+}

backend/src/api/password.ts

@@ -0,0 +1,19 @@
+import { randomBytes, scryptSync, timingSafeEqual } from 'node:crypto';
+
+const KEY_LEN = 64;
+
+export function hashPassword(password: string): string {
+	const salt = randomBytes(16);
+	const hash = scryptSync(password, salt, KEY_LEN);
+	return `${salt.toString('hex')}:${hash.toString('hex')}`;
+}
+
+export function verifyPassword(password: string, stored: string): boolean {
+	const [saltHex, hashHex] = stored.split(':');
+	if (!saltHex || !hashHex) return false;
+	const salt = Buffer.from(saltHex, 'hex');
+	const expected = Buffer.from(hashHex, 'hex');
+	const actual = scryptSync(password, salt, KEY_LEN);
+	if (actual.length !== expected.length) return false;
+	return timingSafeEqual(actual, expected);
+}

backend/src/api/public.ts

@@ -0,0 +1,27 @@
+import type { FastifyInstance } from 'fastify';
+import * as articlesDb from '../storage/db/articles.js';
+import * as tagsDb from '../storage/db/tags.js';
+import * as eventsDb from '../storage/db/events.js';
+
+export async function registerPublicRoutes(app: FastifyInstance) {
+	app.get('/api/feed', async (req) => {
+		const { category, geo, eventId, tag } = req.query as Record<string, string | undefined>;
+		return articlesDb.queryFeed({ category, geo, eventId, tag });
+	});
+
+	app.get('/api/article/:id', async (req, reply) => {
+		const { id } = req.params as { id: string };
+		const article = articlesDb.getArticle(id);
+		if (!article) return reply.code(404).send({ error: 'not found' });
+		return article;
+	});
+
+	app.get('/api/tags', async () => {
+		return tagsDb.listActiveTags();
+	});
+
+	app.get('/api/events', async () => {
+		// Public fields only — sourceIds, cadenceTime etc. stay admin-only.
+		return eventsDb.listEvents().map((e) => ({ id: e.id, name: e.name, active: e.active, cadence: e.cadence }));
+	});
+}

backend/src/index.ts

@@ -0,0 +1,57 @@
+import Fastify from 'fastify';
+import cors from '@fastify/cors';
+import cookie from '@fastify/cookie';
+import fs from 'node:fs';
+import path from 'node:path';
+import { migrate } from './storage/db/index.js';
+import { ensureAdminUserSeeded } from './storage/db/auth.js';
+import { registerAuth } from './api/auth.js';
+import { registerPublicRoutes } from './api/public.js';
+import { registerAdminRoutes } from './api/admin.js';
+import { startScheduler } from './queue/scheduler.js';
+import { logger } from './storage/db/logs.js';
+
+const PORT = Number(process.env.PORT) || 4000;
+const FRONTEND_ORIGIN = process.env.FRONTEND_ORIGIN || 'http://localhost:5173';
+const MEDIA_DIR = process.env.MEDIA_DIR || './data/media';
+
+async function main() {
+	migrate();
+	ensureAdminUserSeeded(
+		process.env.ADMIN_USERNAME || 'admin',
+		process.env.ADMIN_PASSWORD || 'change-me-immediately'
+	);
+
+	const app = Fastify({ logger: false });
+
+	// Cross-origin is expected — see project-structure.md "Cross-origin and security
+	// implications". Not a wildcard: only the configured frontend origin is allowed.
+	await app.register(cors, { origin: FRONTEND_ORIGIN, credentials: true });
+	await app.register(cookie);
+
+	await registerAuth(app);
+	await registerPublicRoutes(app);
+	await registerAdminRoutes(app);
+
+	// Locally hosted media (see storage/media) — served directly rather than via a
+	// heavier static-file plugin, since this is a small, flat directory.
+	app.get('/media/:filename', async (req, reply) => {
+		const { filename } = req.params as { filename: string };
+		if (filename.includes('..') || filename.includes('/')) return reply.code(400).send();
+		const filePath = path.join(MEDIA_DIR, filename);
+		if (!fs.existsSync(filePath)) return reply.code(404).send();
+		return reply.send(fs.createReadStream(filePath));
+	});
+
+	app.get('/health', async () => ({ ok: true }));
+
+	await app.listen({ port: PORT, host: '0.0.0.0' });
+	logger.info('server', `Listening on :${PORT} (frontend origin: ${FRONTEND_ORIGIN})`);
+
+	startScheduler();
+}
+
+main().catch((err) => {
+	console.error('[server] Fatal startup error:', err);
+	process.exit(1);
+});

backend/src/inference/ollama-provider.ts

@@ -0,0 +1,60 @@
+import type { InferenceProvider } from './provider.js';
+
+/**
+ * Talks to a self-hosted Ollama instance over HTTP. Address is a normal backend
+ * setting (GlobalSettings.aiServiceHost/Port), editable via the admin panel —
+ * see the Connections tab and "AI service connection" in the schema doc.
+ */
+export class OllamaProvider implements InferenceProvider {
+	constructor(
+		private host: string,
+		private port: number
+	) {}
+
+	private base(): string {
+		return `${this.host}:${this.port}`;
+	}
+
+	async generate(prompt: string, opts: { model?: string; system?: string } = {}): Promise<string> {
+		const res = await fetch(`${this.base()}/api/generate`, {
+			method: 'POST',
+			headers: { 'Content-Type': 'application/json' },
+			body: JSON.stringify({
+				model: opts.model,
+				prompt,
+				system: opts.system,
+				stream: false
+			})
+		});
+		if (!res.ok) throw new Error(`Ollama generate failed: ${res.status} ${await res.text()}`);
+		const data = (await res.json()) as { response: string };
+		return data.response;
+	}
+
+	async embed(text: string, opts: { model?: string } = {}): Promise<number[]> {
+		const res = await fetch(`${this.base()}/api/embeddings`, {
+			method: 'POST',
+			headers: { 'Content-Type': 'application/json' },
+			body: JSON.stringify({ model: opts.model, prompt: text })
+		});
+		if (!res.ok) throw new Error(`Ollama embed failed: ${res.status} ${await res.text()}`);
+		const data = (await res.json()) as { embedding: number[] };
+		return data.embedding;
+	}
+
+	async listModels(): Promise<string[]> {
+		const res = await fetch(`${this.base()}/api/tags`);
+		if (!res.ok) throw new Error(`Ollama listModels failed: ${res.status}`);
+		const data = (await res.json()) as { models: { name: string }[] };
+		return data.models.map((m) => m.name);
+	}
+
+	async isReachable(): Promise<boolean> {
+		try {
+			const res = await fetch(`${this.base()}/api/tags`, { signal: AbortSignal.timeout(3000) });
+			return res.ok;
+		} catch {
+			return false;
+		}
+	}
+}

backend/src/inference/provider.ts

@@ -0,0 +1,6 @@
+export interface InferenceProvider {
+	generate(prompt: string, opts?: { model?: string; system?: string }): Promise<string>;
+	embed(text: string, opts?: { model?: string }): Promise<number[]>;
+	listModels(): Promise<string[]>;
+	isReachable(): Promise<boolean>;
+}

backend/src/ingestion/adapters/api.ts

@@ -0,0 +1,45 @@
+import type { Source } from '../../storage/db/types.js';
+import type { SourceAdapter, FetchedItem } from './base.js';
+
+function getPath(obj: unknown, path: string): unknown {
+	return path.split('.').reduce<unknown>((acc, key) => (acc && typeof acc === 'object' ? (acc as any)[key] : undefined), obj);
+}
+
+/**
+ * For arbitrary JSON APIs. `source.config.fieldMap` tells the adapter where to find
+ * the array of items and each field within an item, e.g.:
+ *   { itemsPath: "articles", title: "headline", link: "url", summary: "description", publishedAt: "date" }
+ */
+export const apiAdapter: SourceAdapter = {
+	async fetch(source: Source): Promise<FetchedItem[]> {
+		if (!source.url) return [];
+		const fieldMap = (source.config.fieldMap as Record<string, string>) ?? {};
+		const headers = (source.config.authHeaders as Record<string, string>) ?? {};
+
+		const res = await fetch(source.url, { headers });
+		if (!res.ok) throw new Error(`API fetch failed: ${res.status}`);
+		const data = await res.json();
+
+		const rawItems = fieldMap.itemsPath ? getPath(data, fieldMap.itemsPath) : data;
+		if (!Array.isArray(rawItems)) return [];
+
+		const items: FetchedItem[] = [];
+		for (const raw of rawItems) {
+			const title = getPath(raw, fieldMap.title ?? 'title');
+			const link = getPath(raw, fieldMap.link ?? 'link');
+			if (typeof title !== 'string' || typeof link !== 'string') continue;
+
+			items.push({
+				title,
+				summary: String(getPath(raw, fieldMap.summary ?? 'summary') ?? '').slice(0, 500),
+				body: null,
+				images: [],
+				videos: [],
+				link,
+				publishedAt: String(getPath(raw, fieldMap.publishedAt ?? 'publishedAt') ?? new Date().toISOString()),
+				raw
+			});
+		}
+		return items;
+	}
+};

backend/src/ingestion/adapters/base.ts

@@ -0,0 +1,48 @@
+import type { Source, ContentItem } from '../../storage/db/types.js';
+
+export interface FetchedItem {
+	title: string;
+	summary: string;
+	body: string | null;
+	images: { url: string; caption?: string; width?: number; height?: number }[];
+	videos: { url: string; provider?: string; embedHtml?: string }[];
+	link: string;
+	publishedAt: string;
+	raw: unknown;
+}
+
+export interface SourceAdapter {
+	/** Fetches and normalizes new items from this source. Should not throw on individual bad items. */
+	fetch(source: Source): Promise<FetchedItem[]>;
+}
+
+export function toContentItem(source: Source, item: FetchedItem): Omit<ContentItem, 'id'> {
+	return {
+		sourceId: source.id,
+		type: 'article',
+		title: item.title,
+		summary: item.summary,
+		body: item.body,
+		images: item.images,
+		videos: item.videos,
+		link: item.link,
+		publishedAt: item.publishedAt,
+		fetchedAt: new Date().toISOString(),
+		tags: [],
+		geo: source.category.some((c) => c.toLowerCase().startsWith('local')) ? extractGeoTag(source) : null,
+		embedding: null,
+		eventId: null,
+		clusterId: null,
+		raw: item.raw
+	};
+}
+
+function extractGeoTag(source: Source): string | null {
+	// Sources assigned to a LocalRegion carry a "Local: X" category by convention in the
+	// admin panel (see SourcesTab). Real geo-tagging comes from LocalRegion.sourceIds
+	// membership; this is a lightweight fallback derived from the category label.
+	const match = source.category.find((c) => c.toLowerCase().startsWith('local'));
+	if (!match) return null;
+	const parts = match.split(':');
+	return parts[1]?.trim().toLowerCase().replace(/\s+/g, '-') ?? null;
+}

backend/src/ingestion/adapters/rss.ts

@@ -0,0 +1,70 @@
+import Parser from 'rss-parser';
+import type { Source } from '../../storage/db/types.js';
+import type { SourceAdapter, FetchedItem } from './base.js';
+
+type RssItem = Parser.Item & {
+	'media:content'?: any;
+	enclosure?: { url?: string; type?: string };
+};
+
+const parser = new Parser<Record<string, unknown>, RssItem>({
+	customFields: {
+		item: [['media:content', 'media:content']]
+	}
+});
+
+function extractImages(item: RssItem): FetchedItem['images'] {
+	const images: FetchedItem['images'] = [];
+
+	const media = item['media:content'];
+	if (media) {
+		const entries = Array.isArray(media) ? media : [media];
+		for (const m of entries) {
+			if (m?.$?.url) {
+				images.push({
+					url: m.$.url,
+					width: m.$.width ? Number(m.$.width) : undefined,
+					height: m.$.height ? Number(m.$.height) : undefined
+				});
+			}
+		}
+	}
+
+	if (item.enclosure?.url && item.enclosure.type?.startsWith('image/')) {
+		images.push({ url: item.enclosure.url });
+	}
+
+	return images;
+}
+
+function extractVideos(item: RssItem): FetchedItem['videos'] {
+	if (item.enclosure?.url && item.enclosure.type?.startsWith('video/')) {
+		return [{ url: item.enclosure.url }];
+	}
+	return [];
+}
+
+export const rssAdapter: SourceAdapter = {
+	async fetch(source: Source): Promise<FetchedItem[]> {
+		if (!source.url) return [];
+
+		const feed = await parser.parseURL(source.url);
+		const items: FetchedItem[] = [];
+
+		for (const item of feed.items) {
+			if (!item.link || !item.title) continue;
+			items.push({
+				title: item.title,
+				summary: (item.contentSnippet || item.summary || '').slice(0, 500),
+				body: item.content ?? null,
+				images: extractImages(item),
+				videos: extractVideos(item),
+				link: item.link,
+				publishedAt: item.isoDate ?? item.pubDate ?? new Date().toISOString(),
+				raw: item
+			});
+		}
+
+		return items;
+	}
+};

backend/src/ingestion/adapters/telegram.ts

@@ -0,0 +1,17 @@
+import type { Source } from '../../storage/db/types.js';
+import type { SourceAdapter, FetchedItem } from './base.js';
+import { logger } from '../../storage/db/logs.js';
+
+/**
+ * TODO: real implementation. Telegram channels are read via the Bot API (grammY,
+ * per project-structure.md) using `getUpdates` or a channel-history fetch, keyed by
+ * `source.config.telegramChannelId`. Left as a stub — satisfies the SourceAdapter
+ * interface so the poller and admin panel can already list/configure Telegram sources
+ * (e.g. for the "Iran war" tracked event example) before this is filled in.
+ */
+export const telegramAdapter: SourceAdapter = {
+	async fetch(source: Source): Promise<FetchedItem[]> {
+		logger.warn('telegram', `Adapter not yet implemented — skipping source "${source.name}"`);
+		return [];
+	}
+};

backend/src/ingestion/poller.ts

@@ -0,0 +1,49 @@
+import * as sourcesDb from '../storage/db/sources.js';
+import * as contentItemsDb from '../storage/db/contentItems.js';
+import { logger } from '../storage/db/logs.js';
+import { rssAdapter } from './adapters/rss.js';
+import { telegramAdapter } from './adapters/telegram.js';
+import { apiAdapter } from './adapters/api.js';
+import { toContentItem, type SourceAdapter } from './adapters/base.js';
+import type { Source } from '../storage/db/types.js';
+
+const adapters: Record<Source['type'], SourceAdapter> = {
+	rss: rssAdapter,
+	telegram: telegramAdapter,
+	api: apiAdapter,
+	custom: apiAdapter
+};
+
+export async function pollDueSources(defaultIntervalMinutes: number): Promise<number> {
+	const due = sourcesDb.sourcesDueForPoll(defaultIntervalMinutes);
+	let ingested = 0;
+	for (const source of due) {
+		ingested += await pollOne(source);
+	}
+	return ingested;
+}
+
+/** Polls a single source immediately, bypassing its schedule — used right after a source is created. */
+export async function pollSourceNow(source: Source): Promise<number> {
+	return pollOne(source);
+}
+
+async function pollOne(source: Source): Promise<number> {
+	const adapter = adapters[source.type];
+	let ingested = 0;
+	try {
+		const fetched = await adapter.fetch(source);
+		for (const item of fetched) {
+			if (contentItemsDb.existsByLink(item.link)) continue;
+			contentItemsDb.insertContentItem(toContentItem(source, item));
+			ingested++;
+		}
+		sourcesDb.markPolled(source.id, null);
+		logger.info('poller', `Polled "${source.name}" (${source.type}) — ${ingested} new item(s)`);
+	} catch (err) {
+		const message = (err as Error).message;
+		logger.error('poller', `Source "${source.name}" failed: ${message}`);
+		sourcesDb.markPolled(source.id, message);
+	}
+	return ingested;
+}

backend/src/pipeline/clustering.ts

@@ -0,0 +1,56 @@
+import { randomUUID } from 'node:crypto';
+import { cosineSimilarity } from './embedding.js';
+import type { ContentItem } from '../storage/db/types.js';
+
+/**
+ * Maps the admin-facing 1-5 strictness slider to a cosine similarity threshold.
+ * 1 (loose) merges more readily; 5 (strict) requires near-duplicate stories.
+ */
+export function strictnessToThreshold(strictness: number): number {
+	const table: Record<number, number> = { 1: 0.55, 2: 0.65, 3: 0.75, 4: 0.85, 5: 0.92 };
+	return table[strictness] ?? 0.75;
+}
+
+export interface Cluster {
+	id: string;
+	items: ContentItem[];
+	centroid: number[];
+}
+
+function average(vectors: number[][]): number[] {
+	const len = vectors[0].length;
+	const sum = new Array(len).fill(0);
+	for (const v of vectors) for (let i = 0; i < len; i++) sum[i] += v[i];
+	return sum.map((s) => s / vectors.length);
+}
+
+/**
+ * Greedy single-pass clustering: items already events-only (event-assigned items are
+ * excluded upstream — see TrackedEvent handling) get grouped against existing cluster
+ * centroids, or start a new cluster if nothing is similar enough.
+ */
+export function clusterItems(items: ContentItem[], strictness: number): Cluster[] {
+	const threshold = strictnessToThreshold(strictness);
+	const clusters: Cluster[] = [];
+
+	for (const item of items) {
+		if (!item.embedding) continue;
+
+		let best: { cluster: Cluster; score: number } | null = null;
+		for (const cluster of clusters) {
+			const score = cosineSimilarity(item.embedding, cluster.centroid);
+			if (score >= threshold && (!best || score > best.score)) {
+				best = { cluster, score };
+			}
+		}
+
+		if (best) {
+			best.cluster.items.push(item);
+			best.cluster.centroid = average(best.cluster.items.map((i) => i.embedding!));
+		} else {
+			clusters.push({ id: randomUUID(), items: [item], centroid: item.embedding });
+		}
+	}
+
+	return clusters;
+}

backend/src/pipeline/embedding.ts

@@ -0,0 +1,41 @@
+import type { InferenceProvider } from '../inference/provider.js';
+import { logger } from '../storage/db/logs.js';
+import * as contentItems from '../storage/db/contentItems.js';
+import type { ContentItem } from '../storage/db/types.js';
+
+/** Computes and stores an embedding for any content item that doesn't have one yet. */
+export async function embedPendingItems(
+	provider: InferenceProvider,
+	model: string,
+	items: ContentItem[]
+): Promise<ContentItem[]> {
+	const withEmbeddings: ContentItem[] = [];
+	for (const item of items) {
+		if (item.embedding) {
+			withEmbeddings.push(item);
+			continue;
+		}
+		try {
+			const embedding = await provider.embed(`${item.title}\n${item.summary}`, { model });
+			contentItems.setEmbedding(item.id, embedding);
+			withEmbeddings.push({ ...item, embedding });
+		} catch (err) {
+			logger.error('synthesis', `Embedding failed for item ${item.id}: ${(err as Error).message}`);
+		}
+	}
+	return withEmbeddings;
+}
+
+export function cosineSimilarity(a: number[], b: number[]): number {
+	if (a.length === 0 || b.length === 0 || a.length !== b.length) return 0;
+	let dot = 0,
+		normA = 0,
+		normB = 0;
+	for (let i = 0; i < a.length; i++) {
+		dot += a[i] * b[i];
+		normA += a[i] ** 2;
+		normB += b[i] ** 2;
+	}
+	if (normA === 0 || normB === 0) return 0;
+	return dot / (Math.sqrt(normA) * Math.sqrt(normB));
+}

backend/src/pipeline/image-selection.ts

@@ -0,0 +1,37 @@
+import type { ContentItem } from '../storage/db/types.js';
+
+export interface SelectedImage {
+	url: string;
+	sourceItemId: string;
+	selectionReason: string;
+}
+
+/**
+ * Picks the best candidate image across a cluster's items. Currently a fast heuristic
+ * (largest reported resolution, first-seen as tiebreak) rather than a vision-model call —
+ * cheap, deterministic, and good enough for most cases. A vision model (via the same
+ * Ollama connection) could replace this later for cases where resolution alone picks
+ * a poor crop or an image with a watermark; that requires downloading image bytes to
+ * pass to the model, which is a larger change than swapping this function's internals.
+ */
+export function selectBestImage(items: ContentItem[]): SelectedImage | null {
+	let best: { image: ContentItem['images'][number]; item: ContentItem; area: number } | null = null;
+
+	for (const item of items) {
+		for (const image of item.images) {
+			const area = (image.width ?? 0) * (image.height ?? 0);
+			if (!best || area > best.area) {
+				best = { image, item, area };
+			}
+		}
+	}
+
+	if (!best) return null;
+
+	const reason =
+		best.area > 0
+			? `Highest-resolution candidate (${best.image.width}x${best.image.height}) among ${items.length} source(s)`
+			: `First available image among ${items.length} source(s)`;
+
+	return { url: best.image.url, sourceItemId: best.item.id, selectionReason: reason };
+}

backend/src/pipeline/publish.ts

@@ -0,0 +1,179 @@
+import { randomUUID } from 'node:crypto';
+import type { InferenceProvider } from '../inference/provider.js';
+import type { Cluster } from './clustering.js';
+import { synthesizeArticle } from './synthesis.js';
+import { selectBestImage } from './image-selection.js';
+import { downloadAndStore, promoteToPublished } from '../storage/media/index.js';
+import { logger } from '../storage/db/logs.js';
+import * as articles from '../storage/db/articles.js';
+import * as tags from '../storage/db/tags.js';
+import * as sources from '../storage/db/sources.js';
+import type { GlobalSettings, MergedArticle, ContentItem } from '../storage/db/types.js';
+
+const FOLLOW_UP_LOOKBACK_DAYS = 3;
+
+function uniqueCategories(items: ContentItem[]): string[] {
+	const cats = new Set<string>();
+	for (const item of items) {
+		const source = sources.getSource(item.sourceId);
+		for (const cat of source?.category ?? []) cats.add(cat);
+	}
+	return [...cats];
+}
+
+/** Takes the first line of the synthesized body as a working title until a dedicated title-generation step exists. */
+function deriveTitle(body: string): string {
+	const firstLine = body.split('\n')[0];
+	return firstLine.length > 100 ? firstLine.slice(0, 97) + '…' : firstLine;
+}
+
+/**
+ * Publishes a single item as-is, with no AI calls at all — used when the AI service
+ * isn't reachable (e.g. Ollama hasn't been set up yet, per the "assume it arrives
+ * after the backend launches" requirement). No rewriting, no tag extraction, no
+ * embedding. This is deliberately a lesser version of the real pipeline: once Ollama
+ * is available, newly-ingested items get the full embed/cluster/synthesize treatment
+ * and can be linked as follow-ups to these passthrough articles via the normal
+ * tag-based thread detection — but these earlier articles aren't retroactively
+ * rewritten or merged with anything after the fact.
+ */
+export async function publishDirect(item: ContentItem): Promise<MergedArticle> {
+	const category = uniqueCategories([item]);
+	const now = new Date().toISOString();
+
+	let heroImage: MergedArticle['heroImage'] = null;
+	if (item.images.length > 0) {
+		const stored = await downloadAndStore(item.images[0].url, 'published', {});
+		heroImage = stored
+			? { url: stored.servedPath, sourceItemId: item.id, selectionReason: 'Only available image (no AI service configured yet)' }
+			: { url: item.images[0].url, sourceItemId: item.id, selectionReason: 'Only available image (no AI service configured yet)' };
+	}
+
+	const video = item.videos[0] ? { url: item.videos[0].url, provider: item.videos[0].provider, sourceItemId: item.id } : null;
+
+	return articles.insertArticle({
+		title: item.title,
+		body: item.body || item.summary,
+		heroImage,
+		video,
+		category,
+		geo: item.geo,
+		eventId: item.eventId,
+		sourceCount: 1,
+		sources: [
+			{
+				itemId: item.id,
+				sourceName: sources.getSource(item.sourceId)?.name ?? 'Unknown source',
+				link: item.link,
+				publishedAt: item.publishedAt
+			}
+		],
+		publishedAt: now,
+		updatedAt: now,
+		mergeConfidence: 1.0,
+		tags: [], // no LLM available to extract tags yet — backfilling these later is a reasonable future improvement
+		threadId: randomUUID(),
+		previousArticleId: null,
+		nextArticleId: null
+	});
+}
+
+/**
+ * Publishing is always automatic — there's no draft/review state (see schema doc).
+ * A cluster of size 1 publishes as-is via the same path; synthesizeArticle lightly
+ * rewrites rather than merges when there's only one source.
+ */
+export async function publishCluster(
+	provider: InferenceProvider,
+	settings: GlobalSettings,
+	cluster: Cluster,
+	opts: { eventId?: string } = {}
+): Promise<MergedArticle> {
+	const items = cluster.items;
+
+	const { body, tagLabels } = await synthesizeArticle(provider, settings.selectedModels.synthesis, items);
+
+	const resolvedTags = [];
+	for (const label of tagLabels) {
+		try {
+			const embedding = await provider.embed(label, { model: settings.selectedModels.embedding });
+			resolvedTags.push(tags.resolveOrCreateTag(label, embedding, settings.tagDedupThreshold));
+		} catch (err) {
+			logger.error('synthesis', `Tag embedding failed for "${label}": ${(err as Error).message}`);
+		}
+	}
+	const tagIds = resolvedTags.map((t) => t.id);
+
+	const selectedImage = selectBestImage(items);
+	let heroImage: MergedArticle['heroImage'] = null;
+	let storedMediaId: string | null = null;
+	if (selectedImage) {
+		const stored = await downloadAndStore(selectedImage.url, 'published', {});
+		if (stored) {
+			storedMediaId = stored.id;
+			heroImage = { url: stored.servedPath, sourceItemId: selectedImage.sourceItemId, selectionReason: selectedImage.selectionReason };
+		} else {
+			heroImage = selectedImage; // fall back to the hotlinked URL if the download failed, rather than losing the image entirely
+		}
+	}
+	const videoItem = items.find((i) => i.videos.length > 0);
+	const video = videoItem
+		? { url: videoItem.videos[0].url, provider: videoItem.videos[0].provider, sourceItemId: videoItem.id }
+		: null;
+
+	const category = uniqueCategories(items);
+	const geo = items.find((i) => i.geo)?.geo ?? null;
+
+	const itemSources = items.map((item) => ({
+		itemId: item.id,
+		sourceName: sources.getSource(item.sourceId)?.name ?? 'Unknown source',
+		link: item.link,
+		publishedAt: item.publishedAt
+	}));
+
+	// Follow-up detection: only links to a previous article if enough time AND enough
+	// new corroborating sources have passed the admin's thresholds (see MergeTab).
+	// Otherwise this cluster just becomes its own independent thread — holding items
+	// in a "pending, not yet enough to follow up" state is a reasonable future
+	// improvement but adds a queue state this MVP doesn't have yet.
+	const previous = tagIds.length > 0 ? articles.findRecentArticleByTags(tagIds, FOLLOW_UP_LOOKBACK_DAYS) : null;
+	let threadId: string = randomUUID();
+	let previousArticleId: string | null = null;
+
+	if (previous) {
+		const hoursSince = (Date.now() - new Date(previous.publishedAt).getTime()) / 3600_000;
+		const previousLinks = new Set(previous.sources.map((s) => s.link));
+		const newSourceCount = itemSources.filter((s) => !previousLinks.has(s.link)).length;
+
+		if (hoursSince >= settings.followUpMinHoursSinceLast && newSourceCount >= settings.followUpMinNewSources) {
+			threadId = previous.threadId;
+			previousArticleId = previous.id;
+		}
+	}
+
+	const now = new Date().toISOString();
+	const article = articles.insertArticle({
+		title: deriveTitle(body),
+		body,
+		heroImage,
+		video,
+		category,
+		geo,
+		eventId: opts.eventId ?? items[0]?.eventId ?? null,
+		sourceCount: items.length,
+		sources: itemSources,
+		publishedAt: now,
+		updatedAt: now,
+		mergeConfidence: items.length > 1 ? 0.8 : 1.0,
+		tags: tagIds,
+		threadId,
+		previousArticleId,
+		nextArticleId: null
+	});
+
+	if (storedMediaId) {
+		promoteToPublished(storedMediaId, article.id);
+	}
+
+	return article;
+}

backend/src/pipeline/synthesis.ts

@@ -0,0 +1,42 @@
+import type { InferenceProvider } from '../inference/provider.js';
+import type { ContentItem } from '../storage/db/types.js';
+
+const TAG_DELIMITER = '---TAGS---';
+
+const SYSTEM_PROMPT = `You are a neutral news synthesis assistant. Given summaries from multiple news sources describing the same event, write a single original article that:
+- Attributes specific claims to the outlet that reported them (e.g. "Reuters reported...", "AP notes...")
+- Does not copy phrasing verbatim from any source
+- Stays neutral and factual, without editorializing
+- Is 2-4 short paragraphs
+
+If only one source is provided, lightly rewrite it in your own words rather than merging.
+
+After the article, on a new line, write exactly "${TAG_DELIMITER}" followed by 2-4 short comma-separated topic/entity tags (e.g. proper nouns, named events) that this article is about. If nothing salient qualifies, leave the tag line empty.`;
+
+export interface SynthesisResult {
+	body: string;
+	tagLabels: string[];
+}
+
+function buildPrompt(items: ContentItem[]): string {
+	return items
+		.map((item, i) => `Source ${i + 1} (${item.sourceId}):\nTitle: ${item.title}\nSummary: ${item.summary}`)
+		.join('\n\n');
+}
+
+export async function synthesizeArticle(
+	provider: InferenceProvider,
+	model: string,
+	items: ContentItem[]
+): Promise<SynthesisResult> {
+	const prompt = buildPrompt(items);
+	const raw = await provider.generate(prompt, { model, system: SYSTEM_PROMPT });
+
+	const [body, tagSection] = raw.split(TAG_DELIMITER);
+	const tagLabels = (tagSection ?? '')
+		.split(',')
+		.map((t) => t.trim())
+		.filter((t) => t.length > 0 && t.length < 60);
+
+	return { body: body.trim(), tagLabels };
+}

backend/src/queue/eventsRecap.ts

@@ -0,0 +1,72 @@
+import type { InferenceProvider } from '../inference/provider.js';
+import * as eventsDb from '../storage/db/events.js';
+import * as contentItemsDb from '../storage/db/contentItems.js';
+import { embedPendingItems } from '../pipeline/embedding.js';
+import { publishCluster } from '../pipeline/publish.js';
+import { randomUUID } from 'node:crypto';
+import type { GlobalSettings } from '../storage/db/types.js';
+import { logger } from '../storage/db/logs.js';
+
+function isDue(event: ReturnType<typeof eventsDb.listActiveEvents>[number]): boolean {
+	if (event.cadence === 'continuous') return true; // handled every cycle like normal clustering, just scoped to its sources
+
+	const now = new Date();
+	const last = event.lastRecapAt ? new Date(event.lastRecapAt) : null;
+
+	if (event.cadence === 'hourly') {
+		return !last || now.getTime() - last.getTime() >= 3600_000;
+	}
+
+	if (event.cadence === 'daily') {
+		if (!event.cadenceTime) return false;
+		const [h, m] = event.cadenceTime.split(':').map(Number);
+		const scheduledToday = new Date(now);
+		scheduledToday.setHours(h, m, 0, 0);
+		const alreadyRecappedToday = last && last.toDateString() === now.toDateString();
+		return now >= scheduledToday && !alreadyRecappedToday;
+	}
+
+	return false;
+}
+
+export async function runEventRecaps(provider: InferenceProvider, settings: GlobalSettings): Promise<number> {
+	const events = eventsDb.listActiveEvents();
+	let published = 0;
+
+	for (const event of events) {
+		if (event.sourceIds.length === 0 || !isDue(event)) continue;
+
+		const since = event.lastRecapAt ?? new Date(Date.now() - 24 * 3600_000).toISOString();
+		const items = contentItemsDb.unclusteredItemsForSources(event.sourceIds, since);
+		if (items.length === 0) continue;
+
+		const embedded = await embedPendingItems(provider, settings.selectedModels.embedding, items);
+		const withEmbeddings = embedded.filter((i) => i.embedding);
+		if (withEmbeddings.length === 0) continue;
+
+		try {
+			const article = await publishCluster(
+				provider,
+				settings,
+				{
+					id: randomUUID(),
+					items: withEmbeddings,
+					centroid: withEmbeddings[0].embedding!
+				},
+				{ eventId: event.id }
+			);
+
+			contentItemsDb.assignCluster(
+				withEmbeddings.map((i) => i.id),
+				article.id
+			);
+			eventsDb.markRecapped(event.id);
+			published++;
+			logger.info('events', `Published recap for "${event.name}" from ${withEmbeddings.length} item(s)`);
+		} catch (err) {
+			logger.error('events', `Recap failed for "${event.name}": ${(err as Error).message}`);
+		}
+	}
+
+	return published;
+}

backend/src/queue/priorityQueue.ts

@@ -0,0 +1,111 @@
+import type { InferenceProvider } from '../inference/provider.js';
+import * as contentItemsDb from '../storage/db/contentItems.js';
+import * as sourcesDb from '../storage/db/sources.js';
+import * as categoriesDb from '../storage/db/categories.js';
+import * as eventsDb from '../storage/db/events.js';
+import { embedPendingItems } from '../pipeline/embedding.js';
+import { clusterItems } from '../pipeline/clustering.js';
+import { publishCluster, publishDirect } from '../pipeline/publish.js';
+import { logger } from '../storage/db/logs.js';
+import type { GlobalSettings, ContentItem } from '../storage/db/types.js';
+
+function primaryCategoryRank(item: ContentItem, rankByName: Map<string, number>): number {
+	const source = sourcesDb.getSource(item.sourceId);
+	const cats = source?.category ?? [];
+	let best = Infinity;
+	for (const cat of cats) {
+		// Category names on sources can be free text (e.g. "Local: Philadelphia") —
+		// match on the leading segment against the admin-ranked category list.
+		const leading = cat.split(':')[0].trim();
+		const rank = rankByName.get(leading.toLowerCase());
+		if (rank !== undefined && rank < best) best = rank;
+	}
+	return best;
+}
+
+/**
+ * Fallback for when the AI service isn't reachable yet — publishes eligible items
+ * directly (no clustering across sources, no rewriting) rather than leaving pages
+ * empty until Ollama is set up. Still respects category priority and the
+ * hold-before-publish window; just skips everything that requires an AI call.
+ */
+export async function runPassthroughCycle(settings: GlobalSettings): Promise<number> {
+	const eventSourceIds = eventsDb.listActiveEvents().flatMap((e) => e.sourceIds);
+	const items = contentItemsDb.unclusteredItemsExcludingSources(eventSourceIds);
+	if (items.length === 0) return 0;
+
+	const categories = categoriesDb.listCategories();
+	const rankByName = new Map(categories.map((c) => [c.name.toLowerCase(), c.priorityRank]));
+	const ranked = items
+		.map((item) => ({ item, rank: primaryCategoryRank(item, rankByName) }))
+		.sort((a, b) => a.rank - b.rank)
+		.map((r) => r.item);
+
+	const holdMs = settings.holdBeforePublishMinutes * 60_000;
+	let published = 0;
+
+	for (const item of ranked) {
+		const ready = Date.now() - new Date(item.fetchedAt).getTime() >= holdMs;
+		if (!ready) continue;
+
+		try {
+			const article = await publishDirect(item);
+			contentItemsDb.assignCluster([item.id], article.id);
+			published++;
+			logger.info('synthesis', `Published "${article.title}" directly (no AI available)`);
+		} catch (err) {
+			logger.error('synthesis', `Passthrough publish failed for "${item.title}": ${(err as Error).message}`);
+		}
+	}
+
+	return published;
+}
+
+/**
+ * One pass of the synthesis queue: cluster whatever's unclustered (excluding items
+ * belonging to tracked-event sources, which are handled by eventsRecap.ts instead),
+ * ordered by admin-defined category priority, and publish clusters that have cleared
+ * the hold-before-publish window.
+ */
+export async function runSynthesisCycle(provider: InferenceProvider, settings: GlobalSettings): Promise<number> {
+	const eventSourceIds = eventsDb.listActiveEvents().flatMap((e) => e.sourceIds);
+	const items = contentItemsDb.unclusteredItemsExcludingSources(eventSourceIds);
+	if (items.length === 0) return 0;
+
+	const categories = categoriesDb.listCategories();
+	const rankByName = new Map(categories.map((c) => [c.name.toLowerCase(), c.priorityRank]));
+
+	const ranked = items
+		.map((item) => ({ item, rank: primaryCategoryRank(item, rankByName) }))
+		.sort((a, b) => a.rank - b.rank)
+		.map((r) => r.item);
+
+	const embedded = await embedPendingItems(provider, settings.selectedModels.embedding, ranked);
+	const clusters = clusterItems(embedded, settings.mergeStrictness);
+
+	const holdMs = settings.holdBeforePublishMinutes * 60_000;
+	let published = 0;
+
+	for (const cluster of clusters) {
+		const earliestFetch = Math.min(...cluster.items.map((i) => new Date(i.fetchedAt).getTime()));
+		const ready = Date.now() - earliestFetch >= holdMs;
+		if (!ready) continue; // left unclustered — reconsidered next cycle, possibly with more corroborating items
+
+		try {
+			const article = await publishCluster(provider, settings, cluster);
+			contentItemsDb.assignCluster(
+				cluster.items.map((i) => i.id),
+				cluster.id
+			);
+			published++;
+			logger.info(
+				'synthesis',
+				`Published "${article.title}" from ${cluster.items.length} source(s)`
+			);
+		} catch (err) {
+			logger.error('synthesis', `Failed to publish cluster: ${(err as Error).message}`);
+		}
+	}
+
+	return published;
+}

backend/src/queue/retention.ts

@@ -0,0 +1,64 @@
+import * as articlesDb from '../storage/db/articles.js';
+import * as contentItemsDb from '../storage/db/contentItems.js';
+import * as tagsDb from '../storage/db/tags.js';
+import * as eventsDb from '../storage/db/events.js';
+import { deleteCandidateMediaOlderThan, totalStorageBytes } from '../storage/media/index.js';
+import { logger } from '../storage/db/logs.js';
+import type { GlobalSettings } from '../storage/db/types.js';
+
+/**
+ * One sweep, evaluating both dimensions from the schema doc: age-based limits and the
+ * size-based FIFO cap. Whichever fires, oldest-first. Tracked events can override the
+ * published-article age limit (e.g. "Iran war" recaps kept forever) — see EventsTab.
+ */
+export function runRetentionSweep(settings: GlobalSettings) {
+	const { retention } = settings;
+
+	if (retention.rawItemMaxAgeDays != null) {
+		const stale = contentItemsDb.itemsOlderThan(retention.rawItemMaxAgeDays);
+		if (stale.length > 0) {
+			contentItemsDb.deleteContentItems(stale.map((i) => i.id));
+			logger.info('retention', `Pruned ${stale.length} raw item(s) older than ${retention.rawItemMaxAgeDays}d`);
+		}
+	}
+
+	if (retention.publishedArticleMaxAgeDays != null) {
+		const eventOverrides = new Map(eventsDb.listEvents().map((e) => [e.id, e.retentionOverrideDays]));
+		const stale = articlesDb.articlesOlderThan(retention.publishedArticleMaxAgeDays);
+		for (const article of stale) {
+			const override = article.eventId ? eventOverrides.get(article.eventId) : undefined;
+			if (override === null) continue; // explicit "Forever" override
+			if (override !== undefined) {
+				const overrideCutoff = Date.now() - override * 86_400_000;
+				if (new Date(article.publishedAt).getTime() > overrideCutoff) continue;
+			}
+			articlesDb.deleteArticle(article.id);
+		}
+	}
+
+	deleteCandidateMediaOlderThan(retention.rawItemMaxAgeDays ?? 7);
+
+	if (retention.storageCapEnabled) {
+		enforceStorageCap(retention.storageCapValue, retention.storageCapUnit);
+	}
+
+	const expiredTags = tagsDb.expireStaleTags(settings.tagExpiryDays);
+	if (expiredTags > 0) logger.info('retention', `Expired ${expiredTags} stale tag(s)`);
+}
+
+function enforceStorageCap(capValue: number, unit: 'MB' | 'GB') {
+	const capBytes = capValue * (unit === 'GB' ? 1024 * 1024 * 1024 : 1024 * 1024);
+	let used = totalStorageBytes();
+	if (used <= capBytes) return;
+
+	// FIFO: delete oldest published articles (and their media, via cascade at the DB
+	// level being absent here — media rows are cleaned up by the candidate sweep above;
+	// published-tier media tied to a deleted article becomes orphaned and is swept next
+	// cycle once its downloaded_at also ages past raw-item retention as a backstop).
+	const oldest = articlesDb.queryFeed({}).reverse(); // oldest first
+	for (const article of oldest) {
+		if (used <= capBytes) break;
+		articlesDb.deleteArticle(article.id);
+		used = totalStorageBytes();
+	}
+}

backend/src/queue/scheduler.ts

@@ -0,0 +1,67 @@
+import { pollDueSources } from '../ingestion/poller.js';
+import { runSynthesisCycle, runPassthroughCycle } from './priorityQueue.js';
+import { runEventRecaps } from './eventsRecap.js';
+import { runRetentionSweep } from './retention.js';
+import { OllamaProvider } from '../inference/ollama-provider.js';
+import * as settingsDb from '../storage/db/settings.js';
+import { pruneExpiredSessions } from '../storage/db/auth.js';
+import { logger } from '../storage/db/logs.js';
+
+const POLL_TICK_MS = 60_000; // checks which sources are due every minute; each source's own interval governs actual fetch frequency
+const SYNTHESIS_TICK_MS = 60_000;
+const RETENTION_TICK_MS = 60 * 60_000; // hourly
+
+export function startScheduler() {
+	const provider = () => {
+		const s = settingsDb.getSettings();
+		return new OllamaProvider(s.aiServiceHost, s.aiServicePort);
+	};
+
+	setInterval(async () => {
+		try {
+			const settings = settingsDb.getSettings();
+			const ingested = await pollDueSources(settings.defaultPollIntervalMinutes);
+			if (ingested > 0) logger.info('scheduler', `Poll tick: ingested ${ingested} new item(s)`);
+		} catch (err) {
+			logger.error('scheduler', `Poll tick failed: ${(err as Error).message}`);
+		}
+	}, POLL_TICK_MS);
+
+	setInterval(async () => {
+		try {
+			const settings = settingsDb.getSettings();
+			const p = provider();
+
+			if (!(await p.isReachable())) {
+				// Ollama isn't set up yet — publish what we can directly rather than
+				// leaving the site empty. Tracked-event recaps genuinely need the AI
+				// (summarizing many messages isn't something to fake), so those still wait.
+				const published = await runPassthroughCycle(settings);
+				if (published > 0) {
+					logger.warn('scheduler', `AI service unreachable — published ${published} article(s) directly (no rewriting/merging)`);
+				}
+				return;
+			}
+
+			const published = await runSynthesisCycle(p, settings);
+			const recapped = await runEventRecaps(p, settings);
+			if (published > 0 || recapped > 0) {
+				logger.info('scheduler', `Synthesis tick: published ${published} article(s), ${recapped} event recap(s)`);
+			}
+		} catch (err) {
+			logger.error('scheduler', `Synthesis tick failed: ${(err as Error).message}`);
+		}
+	}, SYNTHESIS_TICK_MS);
+
+	setInterval(() => {
+		try {
+			runRetentionSweep(settingsDb.getSettings());
+			pruneExpiredSessions();
+			logger.info('retention', 'Retention sweep completed');
+		} catch (err) {
+			logger.error('retention', `Retention tick failed: ${(err as Error).message}`);
+		}
+	}, RETENTION_TICK_MS);
+
+	logger.info('scheduler', 'Started: poll every 1m, synthesis every 1m, retention every 1h');
+}

backend/src/storage/db/articles.ts

@@ -0,0 +1,115 @@
+import { randomUUID } from 'node:crypto';
+import { db } from './index.js';
+import type { MergedArticle } from './types.js';
+
+function rowToArticle(row: any): MergedArticle {
+	return {
+		id: row.id,
+		title: row.title,
+		body: row.body,
+		heroImage: row.hero_image ? JSON.parse(row.hero_image) : null,
+		video: row.video ? JSON.parse(row.video) : null,
+		category: JSON.parse(row.category),
+		geo: row.geo,
+		eventId: row.event_id,
+		sourceCount: row.source_count,
+		sources: JSON.parse(row.sources),
+		publishedAt: row.published_at,
+		updatedAt: row.updated_at,
+		mergeConfidence: row.merge_confidence,
+		tags: JSON.parse(row.tags),
+		threadId: row.thread_id,
+		previousArticleId: row.previous_article_id,
+		nextArticleId: row.next_article_id
+	};
+}
+
+export function insertArticle(article: Omit<MergedArticle, 'id'>): MergedArticle {
+	const id = `art-${randomUUID()}`;
+	db.prepare(
+		`INSERT INTO merged_articles
+		 (id, title, body, hero_image, video, category, geo, event_id, source_count, sources, published_at, updated_at, merge_confidence, tags, thread_id, previous_article_id, next_article_id)
+		 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
+	).run(
+		id,
+		article.title,
+		article.body,
+		article.heroImage ? JSON.stringify(article.heroImage) : null,
+		article.video ? JSON.stringify(article.video) : null,
+		JSON.stringify(article.category),
+		article.geo,
+		article.eventId,
+		article.sourceCount,
+		JSON.stringify(article.sources),
+		article.publishedAt,
+		article.updatedAt,
+		article.mergeConfidence,
+		JSON.stringify(article.tags),
+		article.threadId,
+		article.previousArticleId,
+		article.nextArticleId
+	);
+	if (article.previousArticleId) {
+		db.prepare('UPDATE merged_articles SET next_article_id = ? WHERE id = ?').run(id, article.previousArticleId);
+	}
+	return { ...article, id };
+}
+
+export function getArticle(id: string): MergedArticle | null {
+	const row = db.prepare('SELECT * FROM merged_articles WHERE id = ?').get(id);
+	return row ? rowToArticle(row) : null;
+}
+
+export function queryFeed(filters: { category?: string; geo?: string; eventId?: string; tag?: string }): MergedArticle[] {
+	let sql = 'SELECT * FROM merged_articles WHERE 1=1';
+	const params: unknown[] = [];
+	if (filters.category) {
+		sql += ' AND category LIKE ?';
+		params.push(`%"${filters.category}"%`);
+	}
+	if (filters.geo) {
+		sql += ' AND geo = ?';
+		params.push(filters.geo);
+	}
+	if (filters.eventId) {
+		sql += ' AND event_id = ?';
+		params.push(filters.eventId);
+	}
+	if (filters.tag) {
+		sql += ' AND tags LIKE ?';
+		params.push(`%"${filters.tag}"%`);
+	}
+	sql += ' ORDER BY published_at DESC';
+	const rows = db.prepare(sql).all(...(params as any[]));
+	return rows.map(rowToArticle);
+}
+
+export function latestArticleInThread(threadId: string): MergedArticle | null {
+	const row = db
+		.prepare('SELECT * FROM merged_articles WHERE thread_id = ? ORDER BY published_at DESC LIMIT 1')
+		.get(threadId);
+	return row ? rowToArticle(row) : null;
+}
+
+export function articlesOlderThan(days: number): MergedArticle[] {
+	const cutoff = new Date(Date.now() - days * 86_400_000).toISOString();
+	const rows = db.prepare('SELECT * FROM merged_articles WHERE published_at < ?').all(cutoff);
+	return rows.map(rowToArticle);
+}
+
+export function findRecentArticleByTags(tagIds: string[], sinceDays: number): MergedArticle | null {
+	if (tagIds.length === 0) return null;
+	const cutoff = new Date(Date.now() - sinceDays * 86_400_000).toISOString();
+	const rows = db
+		.prepare('SELECT * FROM merged_articles WHERE published_at > ? ORDER BY published_at DESC')
+		.all(cutoff);
+	for (const row of rows) {
+		const article = rowToArticle(row);
+		if (article.tags.some((t) => tagIds.includes(t))) return article;
+	}
+	return null;
+}
+
+export function deleteArticle(id: string) {
+	db.prepare('DELETE FROM merged_articles WHERE id = ?').run(id);
+}

backend/src/storage/db/auth.ts

@@ -0,0 +1,53 @@
+import { randomUUID } from 'node:crypto';
+import { db } from './index.js';
+import { hashPassword } from '../../api/password.js';
+import { logger } from './logs.js';
+
+const SESSION_TTL_HOURS = 24;
+
+export function ensureAdminUserSeeded(defaultUsername: string, defaultPassword: string) {
+	const existing = db.prepare('SELECT id FROM admin_users LIMIT 1').get();
+	if (existing) return;
+	db.prepare('INSERT INTO admin_users (id, username, password_hash, created_at) VALUES (?, ?, ?, ?)').run(
+		randomUUID(),
+		defaultUsername,
+		hashPassword(defaultPassword),
+		new Date().toISOString()
+	);
+	logger.warn('auth', `Seeded initial admin user "${defaultUsername}". Change this password after first login.`);
+}
+
+export function getAdminUserByUsername(username: string) {
+	return db.prepare('SELECT * FROM admin_users WHERE username = ?').get(username) as
+		| { id: string; username: string; password_hash: string }
+		| undefined;
+}
+
+export function createSession(ip: string | null): { id: string; expiresAt: string } {
+	const id = randomUUID();
+	const now = new Date();
+	const expiresAt = new Date(now.getTime() + SESSION_TTL_HOURS * 3600_000).toISOString();
+	db.prepare('INSERT INTO sessions (id, created_at, expires_at, ip) VALUES (?, ?, ?, ?)').run(
+		id,
+		now.toISOString(),
+		expiresAt,
+		ip
+	);
+	return { id, expiresAt };
+}
+
+export function isSessionValid(id: string): boolean {
+	const row = db.prepare('SELECT expires_at FROM sessions WHERE id = ?').get(id) as
+		| { expires_at: string }
+		| undefined;
+	if (!row) return false;
+	return new Date(row.expires_at).getTime() > Date.now();
+}
+
+export function deleteSession(id: string) {
+	db.prepare('DELETE FROM sessions WHERE id = ?').run(id);
+}
+
+export function pruneExpiredSessions() {
+	db.prepare('DELETE FROM sessions WHERE expires_at < ?').run(new Date().toISOString());
+}

backend/src/storage/db/categories.ts

@@ -0,0 +1,16 @@
+import { db } from './index.js';
+import type { Category } from './types.js';
+
+function rowToCategory(row: any): Category {
+	return { id: row.id, name: row.name, priorityRank: row.priority_rank, isDefault: !!row.is_default };
+}
+
+export function listCategories(): Category[] {
+	const rows = db.prepare('SELECT * FROM categories ORDER BY priority_rank').all();
+	return rows.map(rowToCategory);
+}
+
+export function setCategoryOrder(order: { id: string; priorityRank: number }[]) {
+	const stmt = db.prepare('UPDATE categories SET priority_rank = ? WHERE id = ?');
+	for (const c of order) stmt.run(c.priorityRank, c.id);
+}

backend/src/storage/db/contentItems.ts

@@ -0,0 +1,100 @@
+import { randomUUID } from 'node:crypto';
+import { db } from './index.js';
+import type { ContentItem } from './types.js';
+
+function rowToItem(row: any): ContentItem {
+	return {
+		id: row.id,
+		sourceId: row.source_id,
+		type: row.type,
+		title: row.title,
+		summary: row.summary,
+		body: row.body,
+		images: JSON.parse(row.images),
+		videos: JSON.parse(row.videos),
+		link: row.link,
+		publishedAt: row.published_at,
+		fetchedAt: row.fetched_at,
+		tags: JSON.parse(row.tags),
+		geo: row.geo,
+		embedding: row.embedding ? JSON.parse(row.embedding) : null,
+		eventId: row.event_id,
+		clusterId: row.cluster_id,
+		raw: row.raw ? JSON.parse(row.raw) : null
+	};
+}
+
+export function insertContentItem(item: Omit<ContentItem, 'id'>): ContentItem {
+	const id = `ci-${randomUUID()}`;
+	db.prepare(
+		`INSERT INTO content_items
+		 (id, source_id, type, title, summary, body, images, videos, link, published_at, fetched_at, tags, geo, embedding, event_id, cluster_id, raw)
+		 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
+	).run(
+		id,
+		item.sourceId,
+		item.type,
+		item.title,
+		item.summary,
+		item.body,
+		JSON.stringify(item.images),
+		JSON.stringify(item.videos),
+		item.link,
+		item.publishedAt,
+		item.fetchedAt,
+		JSON.stringify(item.tags),
+		item.geo,
+		item.embedding ? JSON.stringify(item.embedding) : null,
+		item.eventId,
+		item.clusterId,
+		item.raw ? JSON.stringify(item.raw) : null
+	);
+	return { ...item, id };
+}
+
+/** Avoids re-ingesting the same story twice on repeated polls. */
+export function existsByLink(link: string): boolean {
+	const row = db.prepare('SELECT id FROM content_items WHERE link = ?').get(link);
+	return !!row;
+}
+
+export function unclusteredItemsExcludingSources(excludeSourceIds: string[]): ContentItem[] {
+	const rows = db.prepare('SELECT * FROM content_items WHERE cluster_id IS NULL').all();
+	const items = rows.map(rowToItem);
+	if (excludeSourceIds.length === 0) return items;
+	return items.filter((i) => !excludeSourceIds.includes(i.sourceId));
+}
+
+export function unclusteredItemsForSources(sourceIds: string[], sinceISO: string): ContentItem[] {
+	if (sourceIds.length === 0) return [];
+	const placeholders = sourceIds.map(() => '?').join(',');
+	const rows = db
+		.prepare(`SELECT * FROM content_items WHERE cluster_id IS NULL AND source_id IN (${placeholders}) AND fetched_at > ?`)
+		.all(...sourceIds, sinceISO);
+	return rows.map(rowToItem);
+}
+
+export function setEmbedding(id: string, embedding: number[]) {
+	db.prepare('UPDATE content_items SET embedding = ? WHERE id = ?').run(JSON.stringify(embedding), id);
+}
+
+export function assignCluster(ids: string[], clusterId: string) {
+	const stmt = db.prepare('UPDATE content_items SET cluster_id = ? WHERE id = ?');
+	for (const id of ids) stmt.run(clusterId, id);
+}
+
+export function itemsByCluster(clusterId: string): ContentItem[] {
+	const rows = db.prepare('SELECT * FROM content_items WHERE cluster_id = ?').all(clusterId);
+	return rows.map(rowToItem);
+}
+
+export function itemsOlderThan(days: number): ContentItem[] {
+	const cutoff = new Date(Date.now() - days * 86_400_000).toISOString();
+	const rows = db.prepare('SELECT * FROM content_items WHERE fetched_at < ?').all(cutoff);
+	return rows.map(rowToItem);
+}
+
+export function deleteContentItems(ids: string[]) {
+	const stmt = db.prepare('DELETE FROM content_items WHERE id = ?');
+	for (const id of ids) stmt.run(id);
+}

backend/src/storage/db/events.ts

@@ -0,0 +1,81 @@
+import { randomUUID } from 'node:crypto';
+import { db } from './index.js';
+import type { TrackedEvent } from './types.js';
+
+function rowToEvent(row: any): TrackedEvent {
+	return {
+		id: row.id,
+		name: row.name,
+		description: row.description,
+		sourceIds: JSON.parse(row.source_ids),
+		cadence: row.cadence,
+		cadenceTime: row.cadence_time,
+		active: !!row.active,
+		retentionOverrideDays: row.retention_override_days,
+		lastRecapAt: row.last_recap_at,
+		createdAt: row.created_at
+	};
+}
+
+export function listEvents(): TrackedEvent[] {
+	const rows = db.prepare('SELECT * FROM tracked_events ORDER BY created_at').all();
+	return rows.map(rowToEvent);
+}
+
+export function listActiveEvents(): TrackedEvent[] {
+	const rows = db.prepare('SELECT * FROM tracked_events WHERE active = 1').all();
+	return rows.map(rowToEvent);
+}
+
+export function getEvent(id: string): TrackedEvent | null {
+	const row = db.prepare('SELECT * FROM tracked_events WHERE id = ?').get(id);
+	return row ? rowToEvent(row) : null;
+}
+
+export function createEvent(input: Partial<TrackedEvent>): TrackedEvent {
+	const id = `evt-${randomUUID()}`;
+	const now = new Date().toISOString();
+	db.prepare(
+		`INSERT INTO tracked_events (id, name, description, source_ids, cadence, cadence_time, active, retention_override_days, last_recap_at, created_at)
+		 VALUES (?, ?, ?, ?, ?, ?, ?, ?, NULL, ?)`
+	).run(
+		id,
+		input.name ?? 'Untitled event',
+		input.description ?? '',
+		JSON.stringify(input.sourceIds ?? []),
+		input.cadence ?? 'continuous',
+		input.cadenceTime ?? null,
+		input.active === false ? 0 : 1,
+		input.retentionOverrideDays ?? null,
+		now
+	);
+	return getEvent(id)!;
+}
+
+export function updateEvent(id: string, patch: Partial<TrackedEvent>): TrackedEvent | null {
+	const existing = getEvent(id);
+	if (!existing) return null;
+	const merged = { ...existing, ...patch };
+	db.prepare(
+		`UPDATE tracked_events SET name=?, description=?, source_ids=?, cadence=?, cadence_time=?, active=?, retention_override_days=?, last_recap_at=? WHERE id=?`
+	).run(
+		merged.name,
+		merged.description,
+		JSON.stringify(merged.sourceIds),
+		merged.cadence,
+		merged.cadenceTime,
+		merged.active ? 1 : 0,
+		merged.retentionOverrideDays,
+		merged.lastRecapAt,
+		id
+	);
+	return getEvent(id);
+}
+
+export function deleteEvent(id: string) {
+	db.prepare('DELETE FROM tracked_events WHERE id = ?').run(id);
+}
+
+export function markRecapped(id: string) {
+	db.prepare('UPDATE tracked_events SET last_recap_at = ? WHERE id = ?').run(new Date().toISOString(), id);
+}

backend/src/storage/db/index.ts

@@ -0,0 +1,196 @@
+// Uses Node's built-in node:sqlite (experimental as of Node 22) — no native dependency
+// to compile, which matters on the target hardware (i5-6600K, no GPU) as much as it
+// matters for keeping the backend lightweight in general.
+//
+// If node:sqlite's API changes in a future Node version, `better-sqlite3` is a drop-in
+// alternative with a near-identical synchronous API — see project-structure.md.
+
+import { DatabaseSync } from 'node:sqlite';
+import path from 'node:path';
+import fs from 'node:fs';
+
+const DB_PATH = process.env.DB_PATH || './data/homefeed.db';
+
+fs.mkdirSync(path.dirname(DB_PATH), { recursive: true });
+
+export const db = new DatabaseSync(DB_PATH);
+db.exec('PRAGMA journal_mode = WAL;');
+db.exec('PRAGMA foreign_keys = ON;');
+
+export function migrate() {
+	db.exec(`
+		CREATE TABLE IF NOT EXISTS sources (
+			id TEXT PRIMARY KEY,
+			name TEXT NOT NULL,
+			type TEXT NOT NULL, -- rss | api | telegram | custom
+			category TEXT NOT NULL DEFAULT '[]', -- JSON array
+			url TEXT,
+			config TEXT NOT NULL DEFAULT '{}', -- JSON: apiKey, telegramChannelId, authHeaders
+			poll_interval_minutes INTEGER NOT NULL DEFAULT 15,
+			enabled INTEGER NOT NULL DEFAULT 1,
+			last_polled_at TEXT,
+			last_error TEXT,
+			created_at TEXT NOT NULL
+		);
+
+		CREATE TABLE IF NOT EXISTS content_items (
+			id TEXT PRIMARY KEY,
+			source_id TEXT NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
+			type TEXT NOT NULL DEFAULT 'article',
+			title TEXT NOT NULL,
+			summary TEXT NOT NULL DEFAULT '',
+			body TEXT,
+			images TEXT NOT NULL DEFAULT '[]', -- JSON
+			videos TEXT NOT NULL DEFAULT '[]', -- JSON
+			link TEXT NOT NULL,
+			published_at TEXT NOT NULL,
+			fetched_at TEXT NOT NULL,
+			tags TEXT NOT NULL DEFAULT '[]', -- JSON, raw extracted tag labels pre-dedup
+			geo TEXT,
+			embedding TEXT, -- JSON float array
+			event_id TEXT,
+			cluster_id TEXT, -- set once assigned to a cluster awaiting synthesis
+			raw TEXT -- JSON, original payload
+		);
+		CREATE INDEX IF NOT EXISTS idx_content_items_source ON content_items(source_id);
+		CREATE INDEX IF NOT EXISTS idx_content_items_cluster ON content_items(cluster_id);
+		CREATE INDEX IF NOT EXISTS idx_content_items_published ON content_items(published_at);
+
+		CREATE TABLE IF NOT EXISTS merged_articles (
+			id TEXT PRIMARY KEY,
+			title TEXT NOT NULL,
+			body TEXT NOT NULL,
+			hero_image TEXT, -- JSON {url, sourceItemId, selectionReason}
+			video TEXT, -- JSON
+			category TEXT NOT NULL DEFAULT '[]', -- JSON
+			geo TEXT,
+			event_id TEXT,
+			source_count INTEGER NOT NULL DEFAULT 1,
+			sources TEXT NOT NULL DEFAULT '[]', -- JSON, copied at publish time (see schema doc)
+			published_at TEXT NOT NULL,
+			updated_at TEXT NOT NULL,
+			merge_confidence REAL NOT NULL DEFAULT 1.0,
+			tags TEXT NOT NULL DEFAULT '[]', -- JSON tag ids
+			thread_id TEXT NOT NULL,
+			previous_article_id TEXT,
+			next_article_id TEXT
+		);
+		CREATE INDEX IF NOT EXISTS idx_articles_published ON merged_articles(published_at);
+		CREATE INDEX IF NOT EXISTS idx_articles_thread ON merged_articles(thread_id);
+
+		CREATE TABLE IF NOT EXISTS media_assets (
+			id TEXT PRIMARY KEY,
+			source_url TEXT NOT NULL,
+			local_path TEXT NOT NULL,
+			size_bytes INTEGER NOT NULL,
+			downloaded_at TEXT NOT NULL,
+			tier TEXT NOT NULL, -- candidate | published
+			content_item_id TEXT,
+			article_id TEXT
+		);
+		CREATE INDEX IF NOT EXISTS idx_media_tier ON media_assets(tier);
+
+		CREATE TABLE IF NOT EXISTS tags (
+			id TEXT PRIMARY KEY,
+			label TEXT NOT NULL,
+			aliases TEXT NOT NULL DEFAULT '[]', -- JSON
+			slug TEXT NOT NULL UNIQUE,
+			embedding TEXT NOT NULL DEFAULT '[]', -- JSON
+			first_seen_at TEXT NOT NULL,
+			last_seen_at TEXT NOT NULL,
+			article_count INTEGER NOT NULL DEFAULT 0,
+			status TEXT NOT NULL DEFAULT 'active' -- active | expired
+		);
+
+		CREATE TABLE IF NOT EXISTS tracked_events (
+			id TEXT PRIMARY KEY,
+			name TEXT NOT NULL,
+			description TEXT NOT NULL DEFAULT '',
+			source_ids TEXT NOT NULL DEFAULT '[]', -- JSON
+			cadence TEXT NOT NULL DEFAULT 'continuous',
+			cadence_time TEXT,
+			active INTEGER NOT NULL DEFAULT 1,
+			retention_override_days INTEGER,
+			last_recap_at TEXT,
+			created_at TEXT NOT NULL
+		);
+
+		CREATE TABLE IF NOT EXISTS local_regions (
+			id TEXT PRIMARY KEY,
+			name TEXT NOT NULL,
+			source_ids TEXT NOT NULL DEFAULT '[]', -- JSON
+			seasonal INTEGER NOT NULL DEFAULT 0,
+			active_months TEXT, -- JSON int array or null
+			created_at TEXT NOT NULL
+		);
+
+		CREATE TABLE IF NOT EXISTS categories (
+			id TEXT PRIMARY KEY,
+			name TEXT NOT NULL,
+			priority_rank INTEGER NOT NULL,
+			is_default INTEGER NOT NULL DEFAULT 0
+		);
+
+		CREATE TABLE IF NOT EXISTS admin_users (
+			id TEXT PRIMARY KEY,
+			username TEXT NOT NULL UNIQUE,
+			password_hash TEXT NOT NULL,
+			created_at TEXT NOT NULL
+		);
+
+		CREATE TABLE IF NOT EXISTS sessions (
+			id TEXT PRIMARY KEY,
+			created_at TEXT NOT NULL,
+			expires_at TEXT NOT NULL,
+			ip TEXT
+		);
+
+		CREATE TABLE IF NOT EXISTS logs (
+			id INTEGER PRIMARY KEY AUTOINCREMENT,
+			timestamp TEXT NOT NULL,
+			level TEXT NOT NULL, -- info | warn | error
+			source TEXT NOT NULL, -- e.g. 'poller', 'scheduler', 'synthesis', 'retention'
+			message TEXT NOT NULL
+		);
+		CREATE INDEX IF NOT EXISTS idx_logs_timestamp ON logs(timestamp);
+
+		CREATE TABLE IF NOT EXISTS global_settings (
+			id INTEGER PRIMARY KEY CHECK (id = 1), -- singleton row
+			merge_strictness INTEGER NOT NULL DEFAULT 3,
+			default_poll_interval_minutes INTEGER NOT NULL DEFAULT 15,
+			hold_before_publish_minutes INTEGER NOT NULL DEFAULT 30,
+			tag_dedup_threshold REAL NOT NULL DEFAULT 0.82,
+			tag_expiry_days INTEGER NOT NULL DEFAULT 21,
+			follow_up_min_hours_since_last INTEGER NOT NULL DEFAULT 6,
+			follow_up_min_new_sources INTEGER NOT NULL DEFAULT 2,
+			ai_service_host TEXT NOT NULL DEFAULT 'http://localhost',
+			ai_service_port INTEGER NOT NULL DEFAULT 11434,
+			selected_models TEXT NOT NULL DEFAULT '{}', -- JSON {embedding, image, synthesis}
+			published_article_max_age_days INTEGER,
+			raw_item_max_age_days INTEGER DEFAULT 7,
+			storage_cap_enabled INTEGER NOT NULL DEFAULT 1,
+			storage_cap_value INTEGER NOT NULL DEFAULT 500,
+			storage_cap_unit TEXT NOT NULL DEFAULT 'GB'
+		);
+	`);
+
+	// Seed the singleton settings row if it doesn't exist yet.
+	const existing = db.prepare('SELECT id FROM global_settings WHERE id = 1').get();
+	if (!existing) {
+		db.prepare(
+			`INSERT INTO global_settings (id, selected_models) VALUES (1, '{"embedding":"nomic-embed-text","image":"","synthesis":"qwen2.5:7b-instruct-q4_K_M"}')`
+		).run();
+	}
+
+	// Seed default categories if none exist yet.
+	const catCount = db.prepare('SELECT COUNT(*) as c FROM categories').get() as { c: number };
+	if (catCount.c === 0) {
+		const defaults = ['Top stories', 'Local', 'World', 'Business', 'Tech', 'Culture'];
+		const stmt = db.prepare(
+			'INSERT INTO categories (id, name, priority_rank, is_default) VALUES (?, ?, ?, 1)'
+		);
+		defaults.forEach((name, i) => {
+			stmt.run(`cat-${name.toLowerCase().replace(/\s+/g, '-')}`, name, i + 1);
+		});
+	}
+}

backend/src/storage/db/logs.ts

@@ -0,0 +1,62 @@
+import { db } from './index.js';
+
+export type LogLevel = 'info' | 'warn' | 'error';
+
+export interface LogEntry {
+	id: number;
+	timestamp: string;
+	level: LogLevel;
+	source: string;
+	message: string;
+}
+
+const MAX_LOGS = 2000;
+let sinceLastPrune = 0;
+
+function insertLog(level: LogLevel, source: string, message: string) {
+	db.prepare('INSERT INTO logs (timestamp, level, source, message) VALUES (?, ?, ?, ?)').run(
+		new Date().toISOString(),
+		level,
+		source,
+		message
+	);
+
+	// Cheap periodic prune rather than checking row count on every insert.
+	sinceLastPrune++;
+	if (sinceLastPrune >= 100) {
+		sinceLastPrune = 0;
+		db.prepare(
+			'DELETE FROM logs WHERE id NOT IN (SELECT id FROM logs ORDER BY id DESC LIMIT ?)'
+		).run(MAX_LOGS);
+	}
+}
+
+/** Writes to both the logs table (for the admin panel) and the console (for the terminal). */
+export const logger = {
+	info(source: string, message: string) {
+		console.log(`[${source}] ${message}`);
+		insertLog('info', source, message);
+	},
+	warn(source: string, message: string) {
+		console.warn(`[${source}] ${message}`);
+		insertLog('warn', source, message);
+	},
+	error(source: string, message: string) {
+		console.error(`[${source}] ${message}`);
+		insertLog('error', source, message);
+	}
+};
+
+export function listLogs(filters: { level?: LogLevel; limit?: number } = {}): LogEntry[] {
+	const limit = Math.min(filters.limit ?? 200, 1000);
+	let sql = 'SELECT * FROM logs';
+	const params: unknown[] = [];
+	if (filters.level) {
+		sql += ' WHERE level = ?';
+		params.push(filters.level);
+	}
+	sql += ' ORDER BY id DESC LIMIT ?';
+	params.push(limit);
+	const rows = db.prepare(sql).all(...(params as any[])) as any[];
+	return rows.map((r) => ({ id: r.id, timestamp: r.timestamp, level: r.level, source: r.source, message: r.message }));
+}

backend/src/storage/db/settings.ts

@@ -0,0 +1,65 @@
+import { db } from './index.js';
+import type { GlobalSettings } from './types.js';
+
+function rowToSettings(row: any): GlobalSettings {
+	return {
+		mergeStrictness: row.merge_strictness,
+		defaultPollIntervalMinutes: row.default_poll_interval_minutes,
+		holdBeforePublishMinutes: row.hold_before_publish_minutes,
+		tagDedupThreshold: row.tag_dedup_threshold,
+		tagExpiryDays: row.tag_expiry_days,
+		followUpMinHoursSinceLast: row.follow_up_min_hours_since_last,
+		followUpMinNewSources: row.follow_up_min_new_sources,
+		aiServiceHost: row.ai_service_host,
+		aiServicePort: row.ai_service_port,
+		selectedModels: JSON.parse(row.selected_models),
+		retention: {
+			publishedArticleMaxAgeDays: row.published_article_max_age_days,
+			rawItemMaxAgeDays: row.raw_item_max_age_days,
+			storageCapEnabled: !!row.storage_cap_enabled,
+			storageCapValue: row.storage_cap_value,
+			storageCapUnit: row.storage_cap_unit
+		}
+	};
+}
+
+export function getSettings(): GlobalSettings {
+	const row = db.prepare('SELECT * FROM global_settings WHERE id = 1').get();
+	return rowToSettings(row);
+}
+
+export function updateSettings(patch: Partial<GlobalSettings>): GlobalSettings {
+	const current = getSettings();
+	const merged: GlobalSettings = {
+		...current,
+		...patch,
+		retention: { ...current.retention, ...(patch.retention ?? {}) },
+		selectedModels: { ...current.selectedModels, ...(patch.selectedModels ?? {}) }
+	};
+	db.prepare(
+		`UPDATE global_settings SET
+			merge_strictness=?, default_poll_interval_minutes=?, hold_before_publish_minutes=?,
+			tag_dedup_threshold=?, tag_expiry_days=?, follow_up_min_hours_since_last=?, follow_up_min_new_sources=?,
+			ai_service_host=?, ai_service_port=?, selected_models=?,
+			published_article_max_age_days=?, raw_item_max_age_days=?,
+			storage_cap_enabled=?, storage_cap_value=?, storage_cap_unit=?
+		 WHERE id = 1`
+	).run(
+		merged.mergeStrictness,
+		merged.defaultPollIntervalMinutes,
+		merged.holdBeforePublishMinutes,
+		merged.tagDedupThreshold,
+		merged.tagExpiryDays,
+		merged.followUpMinHoursSinceLast,
+		merged.followUpMinNewSources,
+		merged.aiServiceHost,
+		merged.aiServicePort,
+		JSON.stringify(merged.selectedModels),
+		merged.retention.publishedArticleMaxAgeDays,
+		merged.retention.rawItemMaxAgeDays,
+		merged.retention.storageCapEnabled ? 1 : 0,
+		merged.retention.storageCapValue,
+		merged.retention.storageCapUnit
+	);
+	return getSettings();
+}

backend/src/storage/db/sources.ts

@@ -0,0 +1,97 @@
+import { randomUUID } from 'node:crypto';
+import { db } from './index.js';
+import type { Source } from './types.js';
+
+function rowToSource(row: any): Source {
+	return {
+		id: row.id,
+		name: row.name,
+		type: row.type,
+		category: JSON.parse(row.category),
+		url: row.url,
+		config: JSON.parse(row.config),
+		pollIntervalMinutes: row.poll_interval_minutes,
+		enabled: !!row.enabled,
+		lastPolledAt: row.last_polled_at,
+		lastError: row.last_error,
+		createdAt: row.created_at
+	};
+}
+
+export function listSources(): Source[] {
+	const rows = db.prepare('SELECT * FROM sources ORDER BY created_at').all();
+	return rows.map(rowToSource);
+}
+
+export function listEnabledSources(): Source[] {
+	const rows = db.prepare('SELECT * FROM sources WHERE enabled = 1').all();
+	return rows.map(rowToSource);
+}
+
+export function getSource(id: string): Source | null {
+	const row = db.prepare('SELECT * FROM sources WHERE id = ?').get(id);
+	return row ? rowToSource(row) : null;
+}
+
+export function createSource(input: Partial<Source>): Source {
+	const id = `src-${randomUUID()}`;
+	const now = new Date().toISOString();
+	db.prepare(
+		`INSERT INTO sources (id, name, type, category, url, config, poll_interval_minutes, enabled, last_polled_at, last_error, created_at)
+		 VALUES (?, ?, ?, ?, ?, ?, ?, ?, NULL, NULL, ?)`
+	).run(
+		id,
+		input.name ?? 'Untitled source',
+		input.type ?? 'rss',
+		JSON.stringify(input.category ?? []),
+		input.url ?? null,
+		JSON.stringify(input.config ?? {}),
+		input.pollIntervalMinutes ?? 15,
+		input.enabled === false ? 0 : 1,
+		now
+	);
+	return getSource(id)!;
+}
+
+export function updateSource(id: string, patch: Partial<Source>): Source | null {
+	const existing = getSource(id);
+	if (!existing) return null;
+	const merged = { ...existing, ...patch };
+	db.prepare(
+		`UPDATE sources SET name=?, type=?, category=?, url=?, config=?, poll_interval_minutes=?, enabled=?, last_polled_at=?, last_error=? WHERE id=?`
+	).run(
+		merged.name,
+		merged.type,
+		JSON.stringify(merged.category),
+		merged.url,
+		JSON.stringify(merged.config),
+		merged.pollIntervalMinutes,
+		merged.enabled ? 1 : 0,
+		merged.lastPolledAt,
+		merged.lastError,
+		id
+	);
+	return getSource(id);
+}
+
+export function deleteSource(id: string): void {
+	db.prepare('DELETE FROM sources WHERE id = ?').run(id);
+}
+
+export function markPolled(id: string, error: string | null) {
+	db.prepare('UPDATE sources SET last_polled_at = ?, last_error = ? WHERE id = ?').run(
+		new Date().toISOString(),
+		error,
+		id
+	);
+}
+
+/** Sources due for polling right now, based on their own interval (or the global default). */
+export function sourcesDueForPoll(defaultIntervalMinutes: number): Source[] {
+	const now = Date.now();
+	return listEnabledSources().filter((s) => {
+		if (!s.lastPolledAt) return true;
+		const interval = (s.pollIntervalMinutes || defaultIntervalMinutes) * 60_000;
+		return now - new Date(s.lastPolledAt).getTime() >= interval;
+	});
+}

backend/src/storage/db/tags.ts

@@ -0,0 +1,95 @@
+import { randomUUID } from 'node:crypto';
+import { db } from './index.js';
+import type { Tag } from './types.js';
+
+function rowToTag(row: any): Tag {
+	return {
+		id: row.id,
+		label: row.label,
+		aliases: JSON.parse(row.aliases),
+		slug: row.slug,
+		embedding: JSON.parse(row.embedding),
+		firstSeenAt: row.first_seen_at,
+		lastSeenAt: row.last_seen_at,
+		articleCount: row.article_count,
+		status: row.status
+	};
+}
+
+function slugify(label: string): string {
+	return label
+		.toLowerCase()
+		.replace(/[^a-z0-9]+/g, '-')
+		.replace(/(^-|-$)/g, '');
+}
+
+export function listActiveTags(): Tag[] {
+	const rows = db.prepare("SELECT * FROM tags WHERE status = 'active'").all();
+	return rows.map(rowToTag);
+}
+
+export function getTagsByIds(ids: string[]): Tag[] {
+	if (ids.length === 0) return [];
+	const placeholders = ids.map(() => '?').join(',');
+	const rows = db.prepare(`SELECT * FROM tags WHERE id IN (${placeholders})`).all(...ids);
+	return rows.map(rowToTag);
+}
+
+function cosineSimilarity(a: number[], b: number[]): number {
+	if (a.length === 0 || b.length === 0 || a.length !== b.length) return 0;
+	let dot = 0,
+		normA = 0,
+		normB = 0;
+	for (let i = 0; i < a.length; i++) {
+		dot += a[i] * b[i];
+		normA += a[i] ** 2;
+		normB += b[i] ** 2;
+	}
+	if (normA === 0 || normB === 0) return 0;
+	return dot / (Math.sqrt(normA) * Math.sqrt(normB));
+}
+
+/**
+ * Resolves a candidate tag label to an existing tag (adding it as an alias) or creates
+ * a new one, based on embedding similarity against active tags. See "Tag" and its
+ * dedup flow in homefeed-data-schema.md.
+ */
+export function resolveOrCreateTag(label: string, embedding: number[], dedupThreshold: number): Tag {
+	const active = listActiveTags();
+	let best: { tag: Tag; score: number } | null = null;
+	for (const tag of active) {
+		const score = cosineSimilarity(embedding, tag.embedding);
+		if (!best || score > best.score) best = { tag, score };
+	}
+
+	const now = new Date().toISOString();
+
+	if (best && best.score >= dedupThreshold) {
+		const tag = best.tag;
+		const aliases = tag.aliases.includes(label) || tag.label === label ? tag.aliases : [...tag.aliases, label];
+		db.prepare('UPDATE tags SET aliases = ?, last_seen_at = ?, article_count = article_count + 1 WHERE id = ?').run(
+			JSON.stringify(aliases),
+			now,
+			tag.id
+		);
+		return { ...tag, aliases, lastSeenAt: now, articleCount: tag.articleCount + 1 };
+	}
+
+	const id = `tag-${randomUUID()}`;
+	const slug = slugify(label);
+	db.prepare(
+		`INSERT INTO tags (id, label, aliases, slug, embedding, first_seen_at, last_seen_at, article_count, status)
+		 VALUES (?, ?, '[]', ?, ?, ?, ?, 1, 'active')`
+	).run(id, label, slug, JSON.stringify(embedding), now, now);
+
+	return { id, label, aliases: [], slug, embedding, firstSeenAt: now, lastSeenAt: now, articleCount: 1, status: 'active' };
+}
+
+/** Scheduled sweep: flips tags with no new articles in `expiryDays` to expired. */
+export function expireStaleTags(expiryDays: number): number {
+	const cutoff = new Date(Date.now() - expiryDays * 86_400_000).toISOString();
+	const result = db
+		.prepare("UPDATE tags SET status = 'expired' WHERE status = 'active' AND last_seen_at < ?")
+		.run(cutoff);
+	return Number(result.changes);
+}

backend/src/storage/db/types.ts

@@ -0,0 +1,121 @@
+export interface Source {
+	id: string;
+	name: string;
+	type: 'rss' | 'api' | 'telegram' | 'custom';
+	category: string[];
+	url: string | null;
+	config: Record<string, unknown>;
+	pollIntervalMinutes: number;
+	enabled: boolean;
+	lastPolledAt: string | null;
+	lastError: string | null;
+	createdAt: string;
+}
+
+export interface ContentItem {
+	id: string;
+	sourceId: string;
+	type: 'article' | 'social_post' | 'alert' | 'other';
+	title: string;
+	summary: string;
+	body: string | null;
+	images: { url: string; caption?: string; width?: number; height?: number }[];
+	videos: { url: string; provider?: string; embedHtml?: string }[];
+	link: string;
+	publishedAt: string;
+	fetchedAt: string;
+	tags: string[];
+	geo: string | null;
+	embedding: number[] | null;
+	eventId: string | null;
+	clusterId: string | null;
+	raw: unknown;
+}
+
+export interface ArticleSource {
+	itemId: string;
+	sourceName: string;
+	link: string;
+	publishedAt: string;
+}
+
+export interface MergedArticle {
+	id: string;
+	title: string;
+	body: string;
+	heroImage: { url: string; sourceItemId: string; selectionReason: string } | null;
+	video: { url: string; provider?: string; sourceItemId: string } | null;
+	category: string[];
+	geo: string | null;
+	eventId: string | null;
+	sourceCount: number;
+	sources: ArticleSource[];
+	publishedAt: string;
+	updatedAt: string;
+	mergeConfidence: number;
+	tags: string[];
+	threadId: string;
+	previousArticleId: string | null;
+	nextArticleId: string | null;
+}
+
+export interface Tag {
+	id: string;
+	label: string;
+	aliases: string[];
+	slug: string;
+	embedding: number[];
+	firstSeenAt: string;
+	lastSeenAt: string;
+	articleCount: number;
+	status: 'active' | 'expired';
+}
+
+export interface TrackedEvent {
+	id: string;
+	name: string;
+	description: string;
+	sourceIds: string[];
+	cadence: 'continuous' | 'daily' | 'hourly' | 'custom';
+	cadenceTime: string | null;
+	active: boolean;
+	retentionOverrideDays: number | null;
+	lastRecapAt: string | null;
+	createdAt: string;
+}
+
+export interface LocalRegion {
+	id: string;
+	name: string;
+	sourceIds: string[];
+	seasonal: boolean;
+	activeMonths: number[] | null;
+	createdAt: string;
+}
+
+export interface Category {
+	id: string;
+	name: string;
+	priorityRank: number;
+	isDefault: boolean;
+}
+
+export interface GlobalSettings {
+	mergeStrictness: 1 | 2 | 3 | 4 | 5;
+	defaultPollIntervalMinutes: number;
+	holdBeforePublishMinutes: number;
+	tagDedupThreshold: number;
+	tagExpiryDays: number;
+	followUpMinHoursSinceLast: number;
+	followUpMinNewSources: number;
+	aiServiceHost: string;
+	aiServicePort: number;
+	selectedModels: { embedding: string; image: string; synthesis: string };
+	retention: {
+		publishedArticleMaxAgeDays: number | null;
+		rawItemMaxAgeDays: number | null;
+		storageCapEnabled: boolean;
+		storageCapValue: number;
+		storageCapUnit: 'MB' | 'GB';
+	};
+}

backend/src/storage/media/index.ts

@@ -0,0 +1,80 @@
+import { randomUUID } from 'node:crypto';
+import fs from 'node:fs';
+import path from 'node:path';
+import { db } from '../db/index.js';
+import { logger } from '../db/logs.js';
+
+const MEDIA_DIR = process.env.MEDIA_DIR || './data/media';
+fs.mkdirSync(MEDIA_DIR, { recursive: true });
+
+export interface StoredMedia {
+	id: string;
+	localPath: string;
+	servedPath: string; // what the API returns to the frontend
+	sizeBytes: number;
+}
+
+/**
+ * Downloads and locally hosts an image (or other media) rather than hotlinking the
+ * original — see the retention design doc. `tier` is 'candidate' for raw source
+ * images (pruned with raw-item retention) or 'published' for an article's chosen
+ * image (kept for the article's own retention lifetime, independent of the source).
+ */
+export async function downloadAndStore(
+	sourceUrl: string,
+	tier: 'candidate' | 'published',
+	refs: { contentItemId?: string; articleId?: string }
+): Promise<StoredMedia | null> {
+	try {
+		const res = await fetch(sourceUrl, { signal: AbortSignal.timeout(10_000) });
+		if (!res.ok) return null;
+		const buffer = Buffer.from(await res.arrayBuffer());
+
+		const ext = guessExtension(res.headers.get('content-type') ?? '', sourceUrl);
+		const id = randomUUID();
+		const filename = `${id}${ext}`;
+		const localPath = path.join(MEDIA_DIR, filename);
+		fs.writeFileSync(localPath, buffer);
+
+		db.prepare(
+			`INSERT INTO media_assets (id, source_url, local_path, size_bytes, downloaded_at, tier, content_item_id, article_id)
+			 VALUES (?, ?, ?, ?, ?, ?, ?, ?)`
+		).run(id, sourceUrl, localPath, buffer.length, new Date().toISOString(), tier, refs.contentItemId ?? null, refs.articleId ?? null);
+
+		return { id, localPath, servedPath: `/media/${filename}`, sizeBytes: buffer.length };
+	} catch (err) {
+		logger.error('media', `Failed to download ${sourceUrl}: ${(err as Error).message}`);
+		return null;
+	}
+}
+
+/** Promotes a candidate-tier asset to published-tier so raw-item retention can no longer prune it. */
+export function promoteToPublished(mediaId: string, articleId: string) {
+	db.prepare("UPDATE media_assets SET tier = 'published', article_id = ? WHERE id = ?").run(articleId, mediaId);
+}
+
+export function totalStorageBytes(): number {
+	const row = db.prepare('SELECT COALESCE(SUM(size_bytes), 0) as total FROM media_assets').get() as { total: number };
+	return row.total;
+}
+
+export function deleteCandidateMediaOlderThan(days: number): number {
+	const cutoff = new Date(Date.now() - days * 86_400_000).toISOString();
+	const rows = db
+		.prepare("SELECT id, local_path FROM media_assets WHERE tier = 'candidate' AND downloaded_at < ?")
+		.all(cutoff) as { id: string; local_path: string }[];
+	for (const row of rows) {
+		fs.rm(row.local_path, () => {});
+	}
+	db.prepare("DELETE FROM media_assets WHERE tier = 'candidate' AND downloaded_at < ?").run(cutoff);
+	return rows.length;
+}
+
+function guessExtension(contentType: string, url: string): string {
+	if (contentType.includes('jpeg')) return '.jpg';
+	if (contentType.includes('png')) return '.png';
+	if (contentType.includes('webp')) return '.webp';
+	if (contentType.includes('gif')) return '.gif';
+	const match = url.match(/\.(jpg|jpeg|png|webp|gif)(\?|$)/i);
+	return match ? `.${match[1].toLowerCase()}` : '.jpg';
+}

backend/tsconfig.json

@@ -0,0 +1,16 @@
+{
+	"compilerOptions": {
+		"target": "ES2022",
+		"module": "NodeNext",
+		"moduleResolution": "NodeNext",
+		"outDir": "dist",
+		"rootDir": "src",
+		"strict": true,
+		"esModuleInterop": true,
+		"skipLibCheck": true,
+		"resolveJsonModule": true,
+		"declaration": false,
+		"sourceMap": true
+	},
+	"include": ["src/**/*.ts"]
+}