Many Improvements

2026-07-21 12:51:31 -04:00
parent d16cd508b5
commit b742320108
33 changed files with 1348 additions and 241 deletions
@@ -25,6 +25,28 @@ minimum an embedding model (e.g. `nomic-embed-text`) and a generation model (e.g
 the synthesis tick quietly skips each cycle until the AI service comes back — nothing
 crashes, articles just don't get published.

+## Full-article capture
+
+RSS descriptions are frequently truncated teasers or ad-mangled snippets, not the
+actual article. For RSS and API sources, every newly-ingested item's link gets
+followed and the real page content extracted via Mozilla's Readability (the same
+approach behind Firefox Reader View) — `ingestion/articleFetcher.ts`. When extraction
+succeeds, the feed's own title/summary/body/image get replaced with what was actually
+on the page; nav, ads, sidebars, comments, and footers are excluded, not just stripped
+of tags. Also pulls the page's `og:image`/`twitter:image` meta tag as a hero image when
+the feed itself didn't provide one.
+
+Extraction fails constantly in the real world — paywalls, bot detection, JS-rendered
+pages, odd markup — so this is deliberately non-fatal: on any failure it falls back to
+whatever the feed itself provided (title/description), logged at `warn` level so it's
+visible in the admin panel's Logs tab rather than silent. Telegram sources skip this
+entirely, since a Telegram message *is* the content — there's no separate page to follow.
+
+This adds two dependencies (`jsdom`, `@mozilla/readability`) — the one deliberate
+exception to the "no heavy dependencies" preference elsewhere in this backend, since a
+hand-rolled content-extraction heuristic would be meaningfully less reliable across the
+range of real-world site markup this needs to handle.
+
 ## Behavior before Ollama is set up

 Per the "assume Ollama arrives after the backend launches" requirement: ingestion and
@@ -32,11 +54,24 @@ publishing don't wait for it.

 - **Adding a source polls it immediately**, not on the next scheduler tick — you see
  results right away instead of waiting up to a minute.
- **With no AI service reachable**, the synthesis tick falls back to publishing each
-  item directly once it clears the hold-before-publish window — no rewriting, no
-  cross-source merging, no tags (there's no LLM to extract them yet), but the page
-  populates instead of staying empty. Media (images) still downloads normally, since
-  that never needed AI in the first place.
+- **With no AI service reachable**, every eligible item publishes directly and
+  immediately — no hold-before-publish wait (that window exists to let corroborating
+  sources arrive before an AI merge locks in, which doesn't apply when nothing's being
+  merged). No rewriting, no cross-source merging, no tags (there's no LLM to extract
+  them yet), but the page populates right away instead of staying empty.
+- **Passthrough articles show the original feed publish date** (`pubDate`/`isoDate`
+  from the source), not when they were ingested or published locally. Only
+  AI-synthesized/merged articles get a "modified" timeframe reflecting when the merge
+  actually happened — see `publishDirect` vs `publishCluster` in `pipeline/publish.ts`.
+- **Feed content gets cleaned before storage** — RSS/API content is frequently raw (or
+  double-escaped) HTML; `ingestion/clean.ts` strips tags and decodes entities into
+  readable plain text with paragraph breaks preserved, applied centrally in
+  `adapters/base.ts` so every adapter benefits without duplicating the logic.
+- **No image or video in the feed item?** Falls back to the site's favicon
+  (`{origin}/favicon.ico`, downloaded and locally hosted like any other image) rather
+  than leaving the article with no art at all. Doesn't parse the page's `<head>` for a
+  proper `<link rel="icon">` — just the conventional path, which covers most sites
+  without an extra HTTP round trip. See `faviconUrlFor` in `pipeline/image-selection.ts`.
 - **Once Ollama becomes reachable**, the real pipeline (embed → cluster → synthesize →
  tag) takes back over for anything ingested from that point on. Articles already
  published via the passthrough path aren't retroactively rewritten or merged — they