Nothing published for hours, every cluster failing with "fetch failed".
Ollama's own log showed the real story: requests being cancelled at
exactly 5m0s with a 500, not a model or server error. Node's global
fetch (undici) defaults to a 5-minute headers/body timeout, and CPU-only
prompt processing on the reference hardware (i5-6600K, no GPU, ~17
tok/s) legitimately takes longer than that once prompts carry full
article bodies instead of short blurbs (the previous fix in this same
line of work) — every generate() call past a few thousand tokens got
killed client-side before Ollama could finish.
OllamaProvider.generate() now passes a dedicated undici Agent with
headersTimeout/bodyTimeout disabled as the fetch dispatcher, so the
request runs as long as it actually needs to. Verified the failure
mode and the fix directly: a short-timeout dispatcher against a
deliberately slow server reproduces the exact same "fetch failed" /
UND_ERR_HEADERS_TIMEOUT error seen in production, and a zero-timeout
dispatcher completes the same slow request without issue.
undici was already a transitive dependency (via jsdom); added directly
since ollama-provider.ts now imports from it.
Category priority admin pane gains a "No AI" checkbox alongside
Private/More. When set, items whose source falls under that category
skip embedding, clustering, and LLM synthesis entirely — each
publishes on its own, verbatim from its source (title + body/summary),
the same direct-publish path YouTube/Nitter/Telegram items always use.
Backend: new categories.disable_ai column (default off, migrated in
for existing installs), threaded through categories.ts CRUD and the
POST /api/admin/categories + PATCH /api/admin/settings routes.
priorityQueue.ts's runSynthesisCycle now partitions items three ways
before clustering: source-type direct (youtube/nitter/telegram),
category-disabled direct (new), then whatever's left goes through the
normal embed/cluster/synthesize pipeline.
Tracked-event recaps are a separate, already-existing per-event
toggle (TrackedEvent.recapIntervalHours) since events aren't tied to
a single category — unaffected by this change.
buildPrompt() only ever sent ContentItem.summary (a ~500-char RSS
description) to the model, never .body (the full article text when
the feed provides <content:encoded>) — even though publishDirect
already preferred body over summary for the no-AI-merge path. A
single-source cluster was effectively asking the model to "lightly
rewrite" a one-paragraph blurb, which it did almost verbatim,
producing a short repeated synopsis instead of an actual article.
Now mirrors publishDirect's item.body || item.summary fallback. Body
is already HTML-stripped at ingestion (ingestion/adapters/base.ts),
so no new sanitization needed. The per-entry character budget added
in the previous truncation fix now does real work here, since full
bodies can be much longer than summaries.
Ollama was defaulting to a 4096-token context (vs. the model's 32768
training context) and silently truncating any oversized prompt by
dropping content from the middle, with no error surfaced anywhere —
observed losing ~53% of a merge-cluster prompt in production. Two
prompt builders (buildPrompt/buildRecapPrompt) concatenated all
source summaries/article bodies with no size cap, so a cluster with
enough sources (or a recap spanning enough articles) could easily
exceed the window.
Fix: OllamaProvider.generate() now always sends explicit num_ctx/
num_predict options (sized for CPU-only inference — i5-6600K, no GPU,
~17 tok/s prompt processing) instead of leaving Ollama to pick a
default. synthesis.ts now caps prompt size itself before it ever
reaches Ollama, giving each source/article an equal character budget
and trimming individual entries rather than dropping whole ones off
the end — every source stays at least partially represented and
attributable. Trims are logged via the existing admin log stream
instead of failing silently.
A widget's own custom routes previously required a full process restart to
register — impractical in practice, since the admin API key regenerates on
every restart and would log the admin out of the panel they were just using
to install the widget.
New backend/src/server.ts owns building and swapping the Fastify instance,
split into two steps: validateRoutesBuildable() builds a candidate app and
listens on a throwaway ephemeral port to catch a broken widget's route
registration (e.g. a path collision) before anything live is touched, and
swapLiveServer() does the real close-old/build-new/listen-new cycle on the
actual port. Only the HTTP server and its router are rebuilt — the DB
connection, in-memory widget registry, scheduler intervals, Telegram
session, and admin API key all stay untouched in the same running process.
The split exists because of a real bug hit in testing: the admin routes that
trigger install/delete are themselves served by the live Fastify instance, so
awaiting the full swap inline closed the connection before the response could
be sent (a DELETE that should have returned 204 came back as a bare
connection reset). Now install/uninstall only awaiit the safe ephemeral-port
validation inline (letting a broken widget be rejected and rolled back within
its own request), and the admin routes schedule the actual swap via
setImmediate after their response is already on the wire.
Verified live: uploaded a widget with a custom route, confirmed a clean 201
and the route working moments later with no restart (same PID, same admin
API key); deleted it and confirmed a clean 204, the route gone, and core
widget routes unaffected; and uploaded a deliberately broken widget whose
route collided with /health, confirming it was rejected with a 400, fully
rolled back, and /health kept responding normally throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Weather, Stocks, and Bookmarks move into backend/src/widgets/<id>/ alongside
PoE2, each a full WidgetPlugin rather than a thin wrapper: their tables are
renamed to the widget_<id>_ convention (stock_tickers, bookmarks) or moved off
global_settings entirely into the generic widget_kv store (weather's
location/unit config and forecast cache). Every widget's routes move under a
consistent /api/widget/<id> (public) and /api/admin/widget/<id> (admin)
namespace, replacing the previous flat /api/weather, /api/admin/poe2/browse,
etc. The registry-management routes (list/upload/enable/delete any widget)
stay at /api/admin/widgets since they address the collection, not one
widget's own data.
This also closes the gap where an uploaded widget had backend data plumbing
but no visible presence anywhere: a widget's poll() can now publish to a
generic GET /api/widget/:id/report feed (registered once, works for any id
with zero per-widget route registration, so it's live immediately after
upload with no restart); the sidebar renders it via a new GenericWidgetCard,
or via a new DynamicWidgetSlot that dynamic-imports an optional pre-built
vanilla-JS frontend bundle the widget can ship (served from a new
/widget-assets/:id/* route) — plain browser import(), not blocked by
SvelteKit's ahead-of-time Svelte compilation the way raw .svelte source would
be. A widget's own custom API routes still need a restart to register
(a hard Fastify limitation), but its data/poll/report and any custom frontend
UI now work fully live. The admin Widgets tab gained an upload form and a
list of installed pluggable widgets with enable/delete.
Verified against a copy of the real dev DB (all three renames + the weather
config/cache migration fire once and are idempotent on a second boot) and
through a real browser: uploaded a widget with both a report-driven poll and
a custom frontend bundle, confirmed it renders live in the sidebar with no
backend restart, then deleted it and confirmed it disappears along with all
of its data (table/kv rows/on-disk files).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Widgets are no longer hand-wired per-feature across scheduler.ts, settings.ts,
and the API routes. A new WidgetPlugin interface (backend/src/widgets/types.ts)
lets a widget declare its own schema migration, poll interval, routes, and
uninstall hook; an installed_widgets registry table replaces the closed
widgets/widgetOrder unions on global_settings, and the scheduler/route
registration now iterate loaded widgets generically instead of one hardcoded
block per widget.
PoE2 moves into backend/src/widgets/poe2/ as the first real plugin (its tables
renamed to the widget_poe2_ convention, league cache moved into a new generic
widget_kv store). Weather/Stocks/Bookmarks get thin wrapper plugins so they
share the same dispatch loop without migrating their schema.
New admin routes let a widget be uploaded live (POST /api/admin/widgets, JSON
body with inline file contents — no archive dependency needed) and removed
with full data pruning (DELETE /api/admin/widgets/:id): a host-side safety-net
sweep drops any widget_<id>_* table and widget_kv rows regardless of whether
the widget's own uninstall() hook runs, so deleted widgets don't leave dead
data behind. Built-in widgets can't be deleted through this route.
Verified against a copy of the real dev DB: old poe2_watchlist data survives
the rename, the migration is idempotent on a second boot, and a live-uploaded
test widget was polled, queried, and fully deleted (table/kv/on-disk directory
all gone) without a restart.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
An empty default meant every fresh install started with no Nitter
instance configured at all, so the first Nitter source's prefill had
nothing to draw from. Default to nitter.net instead, a widely-used
public instance, while leaving the field fully editable for anyone who
wants to point at a self-hosted mirror or another public instance.
Every Nitter source previously required the admin to know and paste a
full RSS URL, with no central place showing or setting which instance
they were actually using. Add a nitterInstanceUrl setting (Connections
tab) with a disclaimer against hammering public instances, and prefill
new Nitter sources' feed URL from it — each source's own URL remains
fully editable and is still what's actually polled.
Also relocates the existing tweet-media-mode and fxtwitter base URL
settings from Retention into this same Connections panel, so all
Nitter-related configuration lives in one place.
Drops the checkbox/tooltip/hint-paragraph combo for a single click-to-toggle
Active/Disabled badge (same pattern as EventsTab's Active/Paused). Widget
order is now admin-sortable via up/down arrows and persisted as
widgetOrder, with Sidebar.svelte rendering widgets in that exact sequence
instead of a fixed hardcoded order.
Disabling Weather/Stocks/PoE2 in the Widgets tab previously only hid the
sidebar box — the backend kept polling on schedule regardless. scheduler.ts
now skips both the interval tick and the startup-immediate poll for any
widget currently disabled, so there's no outbound call for something nobody's
displaying. Re-enabling a widget (via PATCH /api/admin/settings) triggers an
immediate poll instead of waiting out its normal cadence, mirroring the
existing weather-location-change behavior.
Verified live: booted with poe2 disabled and confirmed no poll fired at
startup (previously always immediate), then re-enabled via PATCH and
confirmed the poll fired immediately.
Undoes the previous lock — some tracked items (a commit feed, a torrent feed)
are just organizing sources under a nav entry and never want an AI recap at
all, so the control needs to be both live and able to express "never."
Replaces the old cadence ('continuous'/'daily'/'hourly'/'custom') + cadenceTime
model with a single recapIntervalHours: 1 | 3 | 6 | 12 | 24 | null field (null
= off), which also simplifies eventsRecap.ts's isDue() down to one comparison.
Off by default for new items. The "Show in More »" toggle is now available
both inline on the row and inside the edit panel.
- Tracked items get the same isSpillover flag as categories: a per-item "More"
checkbox collapses it into the "More »" nav tab instead of its own top-level
tab, mirroring Category.isSpillover end to end (schema, CRUD, public API,
nav partitioning, /more page).
- Renamed user-facing "Tracked events" text to "Tracked items" (admin tab,
toolbar, buttons, placeholders, /event/:id subtitle) — the underlying
TrackedEvent/events data model and routes are unchanged.
- Locked the per-item recap cadence dropdown (previously a live
Continuous/Daily/Hourly picker) and moved it to its own labeled section with
an explanatory hint: reading eventsRecap.ts confirmed Continuous and Hourly
currently behave identically, so the live choice was more confusing than
useful. New items are fixed to Continuous; existing items show their real
cadence read-only.
Confirmed dead: every source's pollIntervalMinutes is NOT NULL DEFAULT 15 and
the admin form only ever submits 5/15/60, so sourcesDueForPoll()'s fallback to
the global default could never actually trigger. Removes the setting end to
end (schema, settings.ts, scheduler/poller/sources signatures, admin types,
mock backend fixture) and drops the now-single-item "Poll interval" panel from
the Sources & Merge tab.
Each widget now has an independent enable/disable checkbox that gates its
visibility in the sidebar (new global_settings.widgets columns + GET /api/widgets),
and is collapsed by default behind a WidgetSection wrapper so the tab stays
manageable as more widgets get added. The four existing tab components
(WeatherTab/StocksTab/BookmarksTab/Poe2Tab) are reused unmodified as each
section's expanded content.
- Delete dead ArticleCard.svelte and five zero-call-site backend exports
(getTagsByIds, unclusteredItemsForSources, itemsByCluster, deleteAllContentItems,
latestArticleInThread)
- Drop poe2_primary_currency_name from the schema entirely (dev DB, no migration
concerns)
- settings.ts: bind named params instead of 33 positional ?s to remove the
reorder-and-silently-corrupt risk
- Remove the long-dead stooqToYahooSymbols unconditional startup rewrite
- Share direct-publish logic between runPassthroughCycle/runSynthesisCycle via a new
publishItemsDirect helper, and cache sourcesDb.listSources() once per synthesis tick
instead of a per-item getSource() call
- Remove the 'custom' source type (had no adapter of its own, aliased to api)
Simplify to 24h-only change per pair (both directions) since that's
all that's needed. Icons weren't adding anything to the display, so
they're gone from the schema, API, and both components. The sidebar
widget and admin tab's underlying data model both got smaller as a
result — fewer columns, fewer fields, less to render.
The sidebar panel now links out to poe.ninja's own currency page for
the currently tracked league (https://poe.ninja/poe2/economy/{league
slug}/currency), matching the existing pattern of Weather's widget
linking to its own detail page.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
PoE2's economy is inherently pairwise (Exalted vs Chaos, Divine vs
Exalted), not everything quoted in one reference currency, so the
watchlist now tracks admin-picked currency pairs and shows both
directions with 1h/24h/7d change. poe.ninja doesn't expose per-pair
rates or multiple change windows, so both are self-computed: any
pair's rate comes from dividing the two currencies' primaryValue
(same reference currency cancels out), and change% is derived from
our own poll-history snapshots rather than poe.ninja's fixed 7-day
sparkline. The inverse direction's change is exact closed-form math
from the forward change, not a sign-flip approximation.
The old single-currency watchlist schema can't be mapped onto pairs,
so migrate() drops and rebuilds poe2_watchlist when it detects the
old shape.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Tracks Path of Exile 2 currency values via poe.ninja's public economy
API, mirroring the existing Weather/Stocks sidebar modules. Always
follows the current challenge league (auto-detected, no admin picker).
Admin browses and picks currencies from a live list rather than typing
symbols, since currency ids are opaque. Change % is a 7-day window,
labeled accordingly to avoid the same interval ambiguity Stocks had.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Stooq's public quote endpoint now gates every request behind a
client-side proof-of-work challenge (confirmed by manual curl
testing), which a plain server-side fetch can't pass and isn't
worth running a headless browser to solve. Yahoo's unofficial
/v8/finance/chart endpoint still works with just a browser-like
User-Agent header (also confirmed manually — bare curl/fetch UAs
get rate-limited immediately).
One request per ticker instead of one batched request (Yahoo's
batch quote endpoint needs a cookie+crumb handshake this one
doesn't), and change % is now computed against the real previous
close instead of the open-vs-close approximation Stooq's format
forced. Existing installs get their three default tickers
(Dow/S&P/Bitcoin) rewritten from Stooq to Yahoo symbol syntax
automatically; any ticker an admin added themselves is left alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Sidebar widget now reads "Weather - <Location>" and shows a "Feels
like" line (Open-Meteo's own apparent_temperature, which already
blends heat index and wind chill as appropriate rather than needing
season-specific logic here).
The /weather page gains a current-conditions grid (humidity,
precipitation chance, wind direction/speed, pressure, sunrise,
sunset — wind and pressure units independently configurable in the
admin Weather tab) and an alerts section sourced from the US National
Weather Service (free, no key, US-only — fails safe to no alerts
elsewhere) shown between current conditions and the hourly strip.
Also fixes the hourly strip, which is now a fixed 12-column grid (two
rows of 12) instead of one overflowing horizontal-scroll row that
extended into the sidebar's column.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
A persistent right-hand sidebar (hidden only on /admin/**) with three
independent widgets: current-conditions weather (Open-Meteo, no API
key) linking to a new /weather forecast page; Dow/S&P/crypto/stock
tickers (Stooq, polled every 15 minutes); and admin-curated bookmark
links reusing the existing private-access lock per entry.
Weather and stocks are each self-contained modules (client + poller)
under backend/src/weather and backend/src/stocks, mirroring how
telegram/ is separated from the rest of the ingestion pipeline, so
either can be modified or removed independently. Bookmarks has no
external service, so it follows the plainer categories/events
DB-module + CRUD-route pattern instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Categories can now be flagged isSpillover in the admin priority list,
collapsing them out of the main nav into a single "More »" tab that
links to a new /more page listing each spillover category's newest
articles, with the category name linking through to its full page.
Keeps the nav from growing unbounded or word-wrapping as categories
are added.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Previously a tracked event withheld all matching items from ever
publishing individually — they sat unclustered until the recap job fired,
which then bundled them into one article and discarded the raw items.
Visitors saw nothing from an event until a recap happened, and never saw
the underlying pieces at all.
Matching items now publish through the exact same pipeline as everything
else (individually or merged with same-story coverage via the normal
clustering pipeline), just tagged with the event's id via a new
publishDirect/publishCluster opts.eventId. An item whose source is
assigned to an event but doesn't match its keyword filter still
publishes normally, just without the tag, instead of being dropped.
The recap is now a periodic *additional* AI-written summary of everything
already published under the event since the last recap (new
synthesizeRecap prompt + publishEventRecap, reading MergedArticle bodies
rather than raw feed items) — never a replacement. Added a new
MergedArticle.isRecap flag so the two are visually distinguishable.
Frontend: a tracked event is now a real displayed category — new
/event/[id] page (mirrors /category/[name]), active events appended to
the site nav, and a "🧵 AI Recap" marker on recap articles.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Retention panel's "currently using" line and usage bar were always blank
— totalStorageBytes() existed but was never wired into the settings
response, so retention.storageUsedMB was undefined on every load. Now
computed fresh on every GET/PATCH /api/admin/settings.
Tracked events gain a keywords field: only items whose title/summary/
body contain at least one of them (word, phrase, or emoji — e.g. 🇮🇷 for
an "Iran war" event) qualify for that event's recap, instead of every
item from its assigned sources. An item from an event-linked source that
doesn't match now falls through to normal synthesis rather than being
silently dropped. Also added the source-assignment + keyword-filter edit
UI to EventsTab.svelte, which had no way to populate sourceIds at all
before this (the "assign from the Sources tab" comment referenced a
feature that was never built).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Was backwards — the card displayed the polled channel's own name/avatar
with a "Forwarded from @origin" line. Now mirrors the tweet retweet
pattern exactly: the card's author identity (name/handle/avatar) is
always the original channel/user, forward or not, same as tweet.authorName
never being the retweeter. A new repostedByHandle field (replacing
forwardedFrom) carries the polled channel's own handle for the
"Forwarded by @X" line above the card.
Media resolution still keys off the polled channel specifically (a new
sourceChannelUsername field) since attached media lives on the polled
channel's own copy of the message regardless of who originally posted it
— only the avatar now resolves against the displayed (possibly origin,
possibly null) channel identity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Mirrors the repost-line treatment tweets already get. GramJS resolves a
forward's origin channel/user from entities Telegram sends alongside the
same getMessages response (message.forward.chat/.sender), falling back
to fwdFrom.fromName for the rarer case where the origin hid its identity.
TelegramCard shows "↪️ Forwarded from @username" (or just the name if no
public handle) above the meta row.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Telegram has no public hotlinkable media URL the way Twitter's CDN does,
so there's no "direct" option: self-host downloads via the logged-in
account and stores locally; proxy re-fetches live through that same
account on each view via a new /media/telegram-proxy route (small
in-memory cache to absorb repeat views), without persisting anything.
Adapter no longer downloads media eagerly at ingestion — it only records
lightweight refs (message id, kind, mime type, dimensions); publish.ts
resolves those into a servable url per the admin's chosen mode, same
timing as Nitter's tweet media resolution. New "Telegram (message media)"
panel added to the admin Retention tab alongside the existing Nitter one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Logs into a real Telegram account via MTProto (GramJS) rather than a bot,
so it can read any public channel's history. API ID/hash and the login
session are entered through the admin Connections panel and stored
encrypted at rest (new storage/crypto.ts AES-256-GCM helper) rather than
via .env. Messages render as their own TelegramCard (same treatment as
tweets) and open the original message on Telegram instead of an internal
article page; attached media/albums are downloaded and self-hosted at
ingestion time since Telegram has no public hotlinkable media URL.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Unlike "Clear content" (which deletes both the raw ingested items and
the published articles), reissue keeps the raw content_items and only
deletes the articles built from them, resetting cluster_id so those
items get picked up and republished by the very next scheduler tick.
This is for picking up pipeline/rendering changes on already-ingested
content without depending on the source feed to serve the same items
again — Nitter/Twitter in particular won't reliably resurface an old
tweet on a fresh poll. Same multi-source protection as clearing: an
article merged from this source's items together with another
source's is left alone entirely, since undoing just one contributor's
share of a merge isn't supported.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
extractQuotedTweet() was capping the quoted tweet's text at 240
characters via toSummary(); it now keeps the full cleaned text,
matching the outer tweet's own body which was never truncated.
Dropped the matching -webkit-line-clamp: 3 on .quoted-text in
TweetCard.svelte so the full text actually renders instead of being
clipped after 3 lines. Media sizing (.quoted-img's 140px cap, the
media grid's fixed cell heights) is unchanged — only text was meant
to stay unconstrained.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Nitter's RSS marks a bare retweet with a "RT by @handle:" prefix on
the item's <title> (dc:creator is already the original author, not
the retweeter — confirmed against a real sample). TweetCard now shows
a "🔁 Reposted by @handle" line above an otherwise-unchanged card.
A quote-tweet's RSS description carries the embedded tweet fully
inline in a <blockquote> (author, text, one image, permalink) — no
extra fxtwitter call needed. TweetCard renders it as a smaller frame
nested inside the same outer card, below the quoting tweet's own
text and media, labeled "↩️ Replying to @handle" per how this reads
to a visitor even though it's technically Nitter's quote-tweet
representation. Clicking it opens that tweet's own permalink,
independent of the outer card's link.
Both parsers verified against the real sample RSS (Polymarket/
rawsalerts retweet, Goldman/zerohedge quote-tweet) and against the
live publishDirect pipeline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
authorName came from fxtwitter's enrichment (keyed by the tweet ID, which
for a retweet/reply resolves to the original tweet) while authorHandle was
hardcoded to the RSS feed's dc:creator — which is actually whichever list
member's retweet or reply surfaced the item, not the original author. The
card ended up showing one person's display name next to another person's
@handle. Both fields now come from the same enrichment response, falling
back together to the RSS-derived handle only when enrichment fails.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Categories can now be marked "Private" in the admin panel's Category
priority list. Private categories (and every article tagged with
one, even if it's also tagged with a public category) are hidden
from /api/categories, /api/feed, and /api/article/:id for anyone
without a valid login — a plain visitor's browser, not the admin API
key, since that's a header-based credential for the admin SPA only.
Login is a single shared password set via PRIVATE_ACCESS_PASSWORD in
the backend's .env (unset by default, which disables the feature
entirely). On success the backend sets a stateless httpOnly cookie —
its value is a deterministic hash of the password, checked with a
timing-safe comparison on every request, so there's no session table
to maintain. The cookie is requested at the ~400-day cap browsers
enforce on persistent cookies, the closest a cookie can get to
"retained indefinitely."
On the frontend, an always-visible lock icon in the masthead (shown
whenever the feature is configured, independent of the admin panel's
own enabled/disabled toggle) opens a password prompt and reflects
locked/unlocked state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Categories can now be marked "Private" in the admin panel's Category
priority list. Private categories (and every article tagged with
one, even if it's also tagged with a public category) are hidden
from /api/categories, /api/feed, and /api/article/:id for anyone
without a valid login — a plain visitor's browser, not the admin API
key, since that's a header-based credential for the admin SPA only.
Login is a single shared password set via PRIVATE_ACCESS_PASSWORD in
the backend's .env (unset by default, which disables the feature
entirely). On success the backend sets a stateless httpOnly cookie —
its value is a deterministic hash of the password, checked with a
timing-safe comparison on every request, so there's no session table
to maintain. The cookie is requested at the ~400-day cap browsers
enforce on persistent cookies, the closest a cookie can get to
"retained indefinitely."
On the frontend, an always-visible lock icon in the masthead (shown
whenever the feature is configured, independent of the admin panel's
own enabled/disabled toggle) opens a password prompt and reflects
locked/unlocked state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Tweets can carry up to 4 photos/videos/gifs; fxtwitter's media.all
preserves their original order and, for videos, gives a real playable
.mp4 plus a poster thumbnail. TweetCard now renders these as a
1/2/3/4-item grid (Twitter's own layout shapes) with fixed cell
heights so a tall portrait image no longer dictates the whole card's
height in the column view, and video/gif items play back with native
controls instead of showing a static frame. Each item's url (and a
video's thumbnail) still resolves through the configured Nitter media
mode (self-host/proxy/direct) individually.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
Adds nitterMediaMode (self-host/proxy/direct, default proxy) and
fxtwitterBaseUrl to global settings with a new Retention tab panel.
Tweet images and avatars now resolve through the chosen mode instead
of always being downloaded — proxy mode streams media through a new
SSRF-hardened /media/proxy route (hostname allowlist + DNS-rebinding
defense) so the origin server's IP is never exposed to Twitter's CDN,
direct hotlinks the original URL, and self-host keeps the prior
always-download behavior. fxtwitterBaseUrl lets the enrichment call
target a self-hosted FixTweet mirror instead of the public instance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014c1L8ghNBFjfiH64UMViP8
resolveHeroImage's favicon fallback exists so regular articles never look
entirely bare, but for a tweet it meant an image-less tweet showed the
Nitter instance's own favicon slapped on as if it were the tweet's photo.
publishDirect now skips that fallback specifically for tweet items —
TweetCard.svelte already renders cleanly with no image at all.
Was calling /2/status/<id> (no username) based on the originally-given
example; the actual working endpoint is /<handle>/status/<id> (no version
prefix), confirmed via a real curl response. text, created_timestamp, and
author.name/avatar_url all match the assumed shape exactly — only
media.photos[].url remains unverified (that test tweet had no photo), still
guarded by the existing RSS-image fallback either way.
Nitter list/user RSS feeds are ingested as their own source type, enriched
via fxtwitter (author name/handle/avatar, cleaner text, attached photo) with
a graceful RSS-only fallback when that enrichment fails. Tweets always
publish directly, one per article, and never enter the LLM
clustering/synthesis pipeline — the same bypass already used for YouTube,
since merging unrelated tweets together makes no sense.
Rendering: a new distinct embed-card component (avatar, name + @handle,
full untruncated text, optional attached image, published-date-only
timestamp, no like/retweet stats) replaces the plain article row wherever a
tweet appears, on both the category-page list and the article detail page.
Verified end-to-end against the real sample Nitter RSS feed (served
locally): ingestion (all 100 items, tweet metadata correctly extracted,
retweet/quote-tweet blockquotes correctly excluded from own-content text),
publishing (bypasses clustering, tweet field threaded through to the
published article), and rendering (embed card appears on the homepage feed
and the article detail page, no duplicate title).
Known follow-up: fxtwitter's JSON field names are based on public
documentation, not a verified live response (that API is unreachable from
this sandbox) — worth a real curl check before relying on the enrichment
path in production; the RSS-only fallback path is what's actually been
exercised here.
Two hardening changes beyond just a password:
- The admin panel no longer uses stored credentials at all. The backend
generates a random API key on every startup and prints it to its own
console (never through the DB-backed logger, since that's only reachable
from inside the panel this key protects). Every /api/admin/* request must
carry it as an X-Api-Key header, checked with a timing-safe comparison on
every call — there's no session to create or steal, and restarting the
backend invalidates the previous key immediately. The old admin_users and
sessions tables, scrypt password hashing, and cookie-based session plumbing
are removed entirely (dropped via migration for existing installs, not
left behind unused). The login page keeps its existing layout but now asks
for this key and explains where to find it, storing it in the browser's
localStorage rather than relying on a server session.
- The admin panel (the masthead's cog icon and the /admin/* pages
themselves) is now disabled by default on every deployment, gated by a new
frontend-only ADMIN_PANEL_ENABLED env var. This is a separate, UI-only
visibility control — the API key above is what actually protects the
backend regardless of this flag.
Every ingested article used to show up on the homepage regardless of its
source, which meant a handful of high-volume feeds could flood "Top Stories."
Sources now default to not appearing there; a source has to explicitly opt
in via a new checkbox (also toggleable inline with a star icon) for its
articles to show up on the homepage feed. An article shows there if any of
its contributing sources opted in — merged/clustered stories aren't held to
requiring all sources to agree. Category pages, Local, tags, and events are
unaffected; this only gates the bare, no-filter homepage query.
Schema: sources.push_to_top_stories and merged_articles.top_stories, both
backfilled for existing databases via ALTER TABLE.
YouTube's public Atom feed only accepts a channel_id (or the legacy user
param) — it has no equivalent for the newer @handle format, so pasting a
handle URL straight into the source's url field wouldn't have worked. The
adapter now accepts a bare channel ID, a /channel/UC... URL, an @handle URL,
or a bare handle/username, resolving whichever was given to the actual
channel ID by reading it off the channel page when needed.
- Fix "Body cannot be empty" error on DELETE by making the JSON content-type
parser tolerate empty bodies, and by only sending Content-Type from the
frontend when a request actually has one.
- Deleting a source now cascades: raw content items and any article composed
entirely from that source are removed too, plus their media.
- Add admin endpoints/UI to clear all articles, all media, or a single
source's content without deleting the source, so things can be repopulated
fresh.
- Sources can now be assigned multiple categories via checkboxes (instead of
free text) and edited in place, not just added/deleted.
- Add a "News" default category (seeded fresh, backfilled on existing DBs) so
general news sources have a real home instead of the pseudo-category "Top
stories", which is just the homepage's all-categories chronological view.
- Widen the site's content column 15% (1080px -> 1242px).
- Add YouTube as its own source type/ingestion module: pulls a channel's
public Atom feed, and each video always publishes directly as its own
article (title, embedded video, publish date, description) rather than
going through the cross-source clustering/synthesis pipeline.
Claude
Salastil