How it works

Watcher is a weekly batch pipeline. A worker fetches and processes articles, writes the results to Postgres, and a read-only API serves this site. The feed is tuned to what I read, so the picks are mine. This page documents the system behind them.

Architecture

Three processes that never talk to each other directly. The worker writes, the API reads, and the browser only ever sees the API.

Worker. A one-shot Python job on GitHub Actions. It runs weekly, processes new articles and exits.
Database. Supabase Postgres with pgvector, holding articles, stories and a record of every run.
API. A read-only FastAPI service with per-address rate limiting and no login.
Site. Next.js on Vercel. The browser calls the API on the same domain.

Pipeline

Two gates do most of the cost control. Articles already in the database never reach the classifier, and noise never reaches the embedder.

Fetch and dedup

Hacker News plus 22 RSS feeds

Hacker News and 22 RSS feeds are fetched concurrently and normalized into one article record, identified by canonical URL. Articles already stored, repeated within the run, or published more than 14 days ago are dropped before any model call.

374 fetched, 120 already known · 29s

Classify

Claude Haiku 4.5, structured output

Each new article gets one structured call that returns topics, a relevance score from 0 to 1, a noise flag and the named entities it covers. Articles flagged as noise stop here. Haiku is used because this call runs on every article.

247 classified · 25m 26s

Embed

text-embedding-3-small, 768 dimensions

Each remaining article becomes one vector built from its headline, the first 8,000 characters of its body and its feed tags. The model's 1,536 dimensions are truncated to 768 to match the database column.

10s

Cluster

hand-written cosine, entity anchored

A single greedy pass compares each new article with the average vector of every open story and joins the closest one when the clustering rule matches. A story stays open for seven days from its first article. Groups that never reach two articles are dropped.

Synthesize

Claude Sonnet 4.6, new and growing stories

New stories, and existing stories that gained articles, get one call that writes a headline, a summary and a one-line takeaway. The model receives headlines, article text and source names, all of them for a new story and the eight newest for a growing one, and never vectors or clustering state.

3m 52s

Rank by momentum

recency weighted decay

Each article adds 0.5 to the power of its age in hours divided by 72, with age measured from publication. Scores are summed, so a story with more recent coverage ranks higher, and a card shows 100 × score ÷ (score + 5). A story with no article newer than 168 hours becomes stale and leaves the feed.

The clustering rule

The second branch is the whole idea. It buys recall on one event covered from several angles without lowering the similarity floor for everything else.

A single similarity threshold forces a bad choice. Set it high and one event splits into several stories. Set it low and unrelated stories merge. The rule has two tiers instead. An article joins an existing story at a similarity of 0.78 or more, or 0.75 when two new articles form a story. A looser match of 0.65 or more joins only when both name the same entity, such as a company, model or vulnerability, which the classifier has already extracted. That catches one event reported from different angles without lowering the bar for everything else.

Data model

Table	Holds
`articles`	One row per article: source, URL, body text, publication date, classifier output and embedding
`clusters`	One row per story: headline, summary, takeaway, average vector, entities, topics, momentum score and status
`cluster_articles`	Which articles belong to which story, and when each one joined
`pipeline_runs`	One row per run, including failed runs: counts, cost, tokens and time per stage

Cost and speed

Measure	Value
Articles fetched, classified, already stored	374, 247, 120
Cost	$1.74
Tokens in, out	1,214,709, 50,633
Wall clock	30 minutes
Fetch, classify, embed, cluster, synthesize, rank	29s, 1,526s, 10s, 2s, 232s, 8s

Measured on the 12 September 2026 run, before the news feeds and headline rewrites were added. About 58% of the wall clock is deliberate pacing between model calls. The computation itself takes under a minute.

How quality is checked

Classifier. 50 hand-labelled articles, balanced across topics. The eval fails below its bar: noise precision of at least 0.90 and recall of at least 0.75. Precision is held higher because dropping a good story is the worse mistake. The labels are written by hand because the classifier is Claude, and grading Claude with Claude measures agreement rather than correctness.
Clustering. 13 labelled article pairs check that same-story and different-story similarities separate at the configured thresholds.

Decisions worth explaining

Stories are ranked by publication date, not fetch date. A blog fetched for the first time returns years of archive, and scoring by fetch time made those posts rank as new. Each article is dated by the earlier of its publication date and the moment it arrived, which also stops a feed with a timezone error from looking fresher than it is.
A story stops accepting articles seven days after its first one. Without a limit, a large story kept absorbing anything that shared a broad name like a lab, and its average vector drifted toward general AI news. The window is measured from the first article because measuring from the newest would let a story extend its own lifetime every time something joined.
Headlines are rewritten when a story grows. A headline written once goes stale as a story gains articles. Growing stories are resent to the model with their eight newest articles, which keeps the title accurate and caps the cost of each rewrite.
Similarity is computed by hand, not with a clustering library. The merge rule is the most opinionated part of the system, and a library would hide it behind its own algorithm. The whole clustering stage takes about two seconds, so performance gives no reason to trade away that control.
Model calls run one at a time. The provider's rate limit applies to the account, so pacing calls four seconds apart cannot exceed it. A concurrent pool would need a token bucket and retry logic to save about twenty minutes on a weekly job. If speed ever matters, the batch API is the better upgrade at half the price.

Known limitations

Synthesis has no eval, so summary quality is checked by reading rather than measured.
The entity rule was tuned on news. On single-publisher blogs a shared entity can simply mean the same publisher, so digest stories still form.
A story still developing after seven days continues as a new card, so one event can appear twice until the older card goes stale.
The clustering eval's pairs come from an earlier, news-heavy corpus and do not yet reflect the current feed mix.
Google News links are redirects, so one article can arrive directly and through Google News and count as two sources.