Ten sources, one catalog
src/catalog/sources/ holds one adapter per upstream, each independently
testable:
famelack
iptv
samsung
nasa
pluto
roku
plex
xumo
launches
creators
src/catalog/merge.ts combines their output into one channel list;
src/catalog/fetch.ts (atlas.catalog.v* in KV/IndexedDB — see
Architecture) is the client’s
build-time and runtime entry point into that merged catalog.
Identity: channel_aliases
A single real-world channel can arrive from more than one source with a
different external id in each. src/catalog/aliases.ts is the register of
every id space an external identifier can live in — one per source, plus
xmltv, for the EPG grab’s own id space (iptv-org channel ids, which
arrive through a different pipeline than the iptv source itself and so get
their own namespace).
(source, external_id) is unique in the schema. First writer wins, and
the order matters: a merge states every surviving channel’s own identities
before it states any rescued from a record the merge dropped, so a live
channel’s id can never be taken over by a dead twin that happened to share
it.
Catalog slices
.github/workflows/catalog-slices.yml (scheduled every 3 hours, plus
workflow_dispatch; GitHub starts the scheduled runs 2 to 13 hours apart, a
median of 6.6) runs scripts/build-catalog-slices.ts, the merged and
filtered cut of the catalog for each platform, and publishes web.json and
native.json to the public catalog bucket in Supabase Storage with
scripts/publish-object.mjs. The app reads them straight from the bucket
(sliceUrl in src/catalog/slice.ts) and falls back to the sources when a
slice is missing, unreadable, of another format or platform, empty, or older
than three days. The app does not check a slice’s size; the build’s floors do.
The same run also publishes each slice in parts, so a cold load paints Watch
before it has the whole catalog. scripts/publish-catalog-parts.mjs uploads
every part (catalog/web/channels-<hash>.json and four more, each named for a
hash of its content and cached for a year), then the indexes
(catalog/web-index.json, catalog/native-index.json, cached for five
minutes). The app reads its index first, about 21 KB brotli, and draws Watch’s
default shelves from it while the channels and streams arrive; if the index or
a part cannot be used, it downloads the whole slice as before.
scripts/prune-catalog-parts.mjs removes parts no upload has touched for four
days and the current index does not name. web.json and native.json keep
their names and format, because the production deploy reads them.
The build step attaches Jev’s census (public.channel_paths) to every
channel, so it needs SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY in its
own env, not only the publish step’s. Two checks stop a bad run from
publishing. tooThin refuses a merged catalog under 20,000 channels or a
platform slice under 5,000. unfiled refuses a slice with no census paths, or
with fewer than 10,000 channels that have one, which would empty child desks
such as Mystery down to whatever the title regexes catch. Either one writes
nothing, the build step exits 1 with the reason, and the bucket keeps the last
good slice.
Losing famelack alone passes tooThin: on 2026-11-18 a build without it still
had 38,740 merged channels and 15,356 web channels, and only the census floor
refused it, by 124 channels. So famelack and iptv-org are required by name: if
either does not answer, the build stops before the census, exits 1 and prints
required source <name> did not answer: <first error>. A missing optional
source (Samsung, Pluto, Roku, Plex, Xumo, NASA+, launches, creators) is a
warning on the run and a line in the Monday metrics digest.
Catalog pipeline contract
Every source, state, failure, event and alert of this pipeline, with the
changes decided on 2026-11-18: famelack and iptv-org are required, a refused
build exits 1 with its reason, and the alert names the reason and goes out
only when something changed.
pnpm build:web and shipped inside every
deploy, which made each deploy about 60 MB and tied catalog freshness to
deploy frequency. A deploy now carries only the app.
Sync + EPG refresh
scripts/sync-catalog.ts upserts the public catalog (via
toPublicChannel — no stream URLs) and its aliases into Supabase, run
manually with a Supabase CLI access token:
.github/workflows/epg.yml (every 6 hours, plus workflow_dispatch)
regenerates the EPG guide. It is split into two jobs on purpose: grab checks
out the third-party iptv-org/epg grabber (pinned to a specific commit,
npm ci --ignore-scripts, read-only permissions, no deploy secret in scope at
all). Before any scraping, scripts/epg-sources.mjs reads the whole-file
sources (i.mjh.nz, nzxmltv.com, epg.iptvx.one) and the hosted epgshare01 dumps
once each, streamed; the grabber then scrapes only the channels nothing else
covers 12 hours ahead, skipping the sites listed as dead or slow in
scripts/epg-sites.json. epg-to-json.mjs keeps one supplier per channel
(scrape, then blocks, then dumps) and fits the file into its byte budget by
shortening every channel’s night equally. publish — which
never touches that third-party code — re-checks the publish threshold on the
downloaded artifact, syncs the rich per-programme detail into Supabase, and
publishes the compact guide (epg.json) to the public guide bucket in
Supabase Storage. The app reads it back through GET /api/epg. Nothing is
committed to main and nothing is deployed: a guide refresh is a data change,
and the app only deploys when its code changes. That split means a compromised
dependency in the grabber’s tree can never reach the Supabase service role.
The grab writes a
grab-status.json unconditionally (even on failure), and
a final “Report the grab’s real outcome” step runs if: always(), after the
publish, so a bad grab degrades gracefully to the last good guide
while still turning the workflow run red with a named cause — rather than
silently reporting success on an empty result.Programmes retention
The rich per-programme detail (synopsis, episode, rating, credits — the part that does not fit inepg.json) is synced into the programmes table rather
than committed as thousands of JSON files, which used to cost roughly 2 GB a
month of git history for data that expires in 24 hours. The EPG workflow
requires landing at least 90% of the rows it was handed or it reports failure
loudly, even though the guide itself has already published successfully by
that point.