This page is generated from the contract files in docs/contracts/catalog/ by scripts/contracts/catalog/build-docs.mjs. Edit the JSON, then rebuild the page; the validator fails when the two disagree.
The catalog pipeline builds the channel list every Bytestream viewer sees. A scheduled GitHub Actions job reads a set of public channel lists, files the channels under our own desks, and publishes one finished catalog per platform to Supabase Storage. The app downloads its platform’s catalog instead of reading every source itself. This contract describes that pipeline row by row: each source, state, failure, event and alert. Every row has a status. Live rows describe the code at commit e47b336 on branch fix/similar-ties and cite the file and line that does it. Planned rows are decided and not built; each one is the spec its change is built and tested against. Pending merge rows are built on a branch that has not merged.
famelack is one of the two public channel lists the catalog depends on. On 2026-11-10 17:40 UTC famelack-data moved its channel files from tv/raw/ to tv/ (famelack commit e3f9177, ‘remove legacy raw and compressed layouts’), and every tv/raw/ URL has answered 404 since. The build read tv/raw/ until PR #230 merged as 4fbfe14 on 2026-11-18; the first run on it (44363276494, 17:52 UTC) published green. The decisions below are about the next failure: the build now refuses loudly when famelack or iptv-org is missing, and the first alert carries the cause.

Terms

  • Source: a public list of channels and stream addresses that someone else publishes, such as famelack or iptv-org.
  • Census: our own table, public.channel_paths in Supabase, which files channels under desks such as Comedy or True Crime.
  • Slice: one platform’s finished catalog, catalog/web.json or catalog/native.json, in a public Supabase Storage bucket.
  • Floor: a minimum the build must reach before it publishes.
  • Watchdog: api/watchdog.ts, a Vercel cron that checks once a day how old the published objects are.
  • Index and part: the same catalog cut smaller, so the app can show Watch before it has everything. catalog/<platform>-index.json names five parts and carries Watch’s first screen; each part, such as catalog/web/channels-<hash>.json, is named for a hash of its content.
  • Alert: a Telegram message to the ops group.

What the pipeline does

1

Schedule

.github/workflows/catalog-slices.yml is scheduled at 30 */3 * * *. GitHub starts the runs 1.7 to 13 hours apart, a median of 6.6.
2

Read the sources

loadCatalog asks every source. A source that fails is left out, its flag in sources turns false, and its first error is kept in sourceErrors. If famelack or iptv-org did not answer, the build stops here and exits 1.
3

Attach the census and cut the slices

The build reads the census, files each channel under its desk, and cuts one slice for the web and one for iOS and Android.
4

Check and publish

If every floor passes, both slices are written and uploaded, and then the same catalog in parts: every part first, the two indexes last. If any floor fails, neither slice is written, the build step exits 1 with the reason, and Storage keeps the last good pair.
5

Watch

The alert job reads every finished run and sends a Telegram message when something changed: a first failure, a new reason, a daily reminder, or back to normal. Once a day at 14:00 UTC the watchdog reports any object older than its limit and logs every check.
The app reads its platform’s index first and uses it when the index is format 2, for slice format 5, for its platform, and at most 72 hours old. Otherwise it uses its slice when the slice is format 5, for its platform, not empty, and at most 72 hours old. Otherwise it reads every source itself, which is slower.

The catalog in parts

A web slice is about 23 MB, about 2 MB brotli, and a cold load used to download and parse all of it before Watch drew a shelf. Decision D9 adds a second way to the same catalog: each run also writes the slice as an index and five parts. The index carries Watch’s first screen for a visitor who has chosen nothing: up to 8 shelves of up to 20 channels, their streams and what is on now. The app paints that screen from the index, about 21 KB brotli, while the channels and streams arrive. web.json and native.json keep their names, format 5 and shape, because the production deploy reads them. The workflow publishes in this order, so a visitor never reads an index that names a part not yet in the bucket:
  1. web.json and native.json, as before.
  2. Every part of both platforms, with public, max-age=31536000, immutable. scripts/publish-catalog-parts.mjs first checks that each index names every part, that each part is on disk, and that each part’s SHA-256 starts with the 16 hex digits in its name. One failure and nothing goes up.
  3. The two indexes, with max-age=300.
  4. The prune, which removes parts that no upload has touched for 4 days and that the current index does not name. It never deletes web.json, native.json or an index, and a failed prune is only a warning.
The app loads in this order, and each failure goes on to the next way:
  1. The index. If it does not answer, the app uses the index it kept from an earlier visit, while that one is still usable.
  2. The first view, painted from the index for a visitor who has chosen nothing. A link or a last channel the first view does not hold waits for the catalog.
  3. The core: channels and streams. In a browser each part must hash to its name. If either part fails, the app loads the whole slice, then the sources.
  4. The schedule window, from an hour before the build to 8 hours after, once the page is idle; the whole schedule when the Guide or Spotlight opens; then the blurbs.

Sources

famelack and iptv-org are required (decision D1): a build without either refuses and exits 1 with required source <name> did not answer: <first error>. The other sources are optional; a missing one is a warning on the run, a row in the run summary and a line in the Monday metrics digest, and never fails the build.

The slice and its floors

Rules every build keeps:
  • Live: Both slices are written together or neither is. scripts/build-catalog-slices.ts:257
  • Live: No slice is written below any floor: merged channels under 20,000, a platform slice under 5,000, or under 10,000 channels with a census path. scripts/build-catalog-slices.ts:258
  • Live (D2): The build step exits 1 whenever it writes no slice: a floor refusal, a missing required source or a crash. It exits 0 only after writing both slices. scripts/build-catalog-slices.ts:216
  • Live: A client uses a slice only when its sliceVersion is 5, its platform matches, it has channels, and it is at most 72 hours old. src/catalog/slice.ts:94
  • Live: A live load fails only when famelack and iptv-org both fail; any other source failing leaves its flag false and the load continues. src/catalog/fetch.ts:419
  • Live: A slice keeps the source flags of the load that built it, so a published slice says which sources answered. src/catalog/sliceBuild.ts:108
  • Live (D1): No slice is written unless famelack and iptv-org both answered, on top of the three floors. scripts/build-catalog-slices.ts:241
  • Live (D2): The publish step runs only after a green build step, so it never runs on a missing file. .github/workflows/catalog-slices.yml:70
  • Live (D9): catalog/web.json and catalog/native.json keep their names, sliceVersion 5 and their shape. The parts are written beside them, after them, and a failure while writing the parts never stops them. scripts/build-catalog-slices.ts:287
  • Live (D9): An index goes up only after every part it names, and only when every part’s SHA-256 starts with the hash in its name. One mismatch or missing file and nothing goes up. scripts/publish-catalog-parts.mjs:75
  • Live (D9): The prune deletes only part names under web/ and native/ that the current index does not name and that no upload has touched for four days. It never deletes web.json, native.json or an index, and does nothing without both current indexes. scripts/prune-catalog-parts.mjs:26
  • Live (D9, D10): A client uses an index only when its indexVersion is 1 or 2, its sliceVersion 5, its platform matches and it is at most 72 hours old, and refuses one whose first.picks is there but not two id lists. Any failure in the index or the core goes on to the whole slice, then to the sources. src/catalog/parts.ts:235
  • Live (D9, D10): The first view’s shelves are the whole slice’s own default shelves, worked out by the same buildShelves over the whole slice and cut to FIRST_SHELVES rows of FIRST_PER_SHELF channels, so the core never reorders them. You might like and Trending on Bytestream, which Watch works out on the device, come from the index’s picks (INV-14); Watch keeps what every row shows when the core arrives (first-load INV-8). src/catalog/parts.ts:177
  • Live (D10): The index’s picks are worked out with the functions Watch runs (youMightLikeShelf, trendingShelf) over the whole slice as a visitor with no filters browses it: You might like for the first PICKS_TUNES channels of the first view’s first row, from channel_similar as op=similar reads it, and Trending on Bytestream from channel_stats as op=trending reads it. Each list is the row’s 12 channels and PICKS_SPARE more, and the first view carries their cards, streams and what is on them now. Given the whole catalogue and the same rankings, Watch draws the same rows. src/catalog/picksBuild.ts:44
  • Live (D10, D12): When op=similar has nothing for the tuned channel, You might like ranks the playable channels of its primary category with rankPeers: the tuned channel’s first language (spellings folded, so fre is fra), its census desk (pathChild), how many of its other categories the peer shares (general aside), a stream no territory limits, how many services carry a playable stream of the channel, the same country. Catalogue order breaks ties. Then each of STATE_MEDIA’s broadcasters (RT and CGTN in every variant) moves STATE_MEDIA_DEMOTE places down, or to the end of a shorter list, and everything else keeps its order. No family of numbered or format variants (channelFamily) holds more than FAMILY_CAP of the first FAMILY_WINDOW places, the tuned channel counting as one of its own. The build and the device run the same function, and the recommendations run ranks channel_similar with it (INV-17). src/catalog/recommend.ts:163
  • Live (D11): Inside each channel, the slice lists its streams in the order orderStreams gives a viewer with no history when it is handed the channel: a relay or a row iptv-org labels Geo-blocked or Not 24/7 after the others, and the channel’s own host first among streams that tie on every other key. A host is the channel’s own when it shares the domain of the channel’s website or the domain’s name starts with a word of the channel’s name (nhkworld.jp for NHK World-Japan). A channel’s streams keep the positions in the list they held before, so only the order inside a channel changes and web.json keeps its shape. src/catalog/sliceBuild.ts:79
  • Live (D12): channel_similar holds, for each channel of the published native slice that has a peer, its top TOP_N peers by rankPeers over that slice as a visitor with no filters browses it, with watches in the last seven days as one more tier; a channel public.channels lacks or has closed is left out on both sides, so it gets no rows and its You might like comes from the device (INV-15). The run writes channel_similar and nothing else. A slice that is missing, of another version or stale fails the run before it writes. src/recs/similar.ts:25

States

Each part of the pipeline is a small state machine. A transition emits a named event or is silent: a change of state that prints, logs and sends nothing.
One source inside one catalog load, in the slice build or in the app. Sources are asked independently; famelack and iptv-org in parallel with nasa, launches, creators, blocklist, logos and regions, then the FAST mirrors one at a time.
One run of .github/workflows/catalog-slices.yml: build both slices and their parts, publish both slices, then the parts and their indexes.
The objects catalog/web.json and catalog/native.json in Storage, by age since their Last-Modified.
One catalog load in the app (useCatalog.load, src/state/catalog.ts). Since D9 the parts come first, then the whole slice, then the sources.
The schedules a catalogue from parts leaves out of its core (useCatalog.loadProgrammes). A catalogue from the cache, the slice or the sources carries them already and never enters this machine.
Channel blurbs for a catalogue from parts (useCatalog.loadDescriptions). A catalogue from the cache, the slice or the sources carries them on each channel.
What happens after a workflow run completes or the watchdog runs. alerts.yml runs scripts/alert-on-change.mjs for every completion of a watched workflow and sends only on a change, with the run’s reason (D3, D4). The watchdog sends once a day while anything is stale and logs every check (D5).
A run of Refresh EPG (.github/workflows/epg.yml) or Recommendations (.github/workflows/recs.yml). Their own steps are outside this contract; they appear because the same alerts and the watchdog cover them.

Failures

Every way the pipeline can fail, whether anything reaches a person without someone opening a log, and which alert covers it. A4 is the Telegram message for a red run, sent when something changed; A5 is the watchdog’s daily message; A6 is the line in the Monday metrics digest.

What each failure sends

For each failure the decisions changed: what triggers it, what the run or app prints, and the Telegram text a person reads.
When: loadCatalog returns with sources.famelack or sources.iptv false. The build checks this right after loadCatalog returns, before it reads the census.What the run prints: The build prints ‘::error::required source <name> did not answer: <first error>’, writes the same line at the top of the step summary, and exits 1. The publish step does not run. The step summary lists every source with its answer.Alert text:
State: slice_run.build_failed. Alerts: A4.Code: src/catalog/fetch.ts:382, scripts/build-catalog-slices.ts:234, scripts/build-catalog-slices.ts:243
When: loadCatalog returns with samsung, pluto, roku, plex, xumo, nasa or launches false, or with a sourceErrors entry for creators (which has no flag).What the run prints: The build prints ‘::warning::optional source <name> did not answer: <first error>’, lists the source as missing in the step summary, and goes on. The Monday metrics digest lists every optional source missing from any Catalog slices run in the past 7 days. Nothing pages.Alert text:
State: source_load.dropped. Alerts: A6.Code: src/catalog/fetch.ts:404, scripts/build-catalog-slices.ts:239, src/metrics/digest.ts:92, scripts/metrics-digest.ts:54
When: loadCatalog throws because famelack and iptv-org both failed (src/catalog/fetch.ts throws famelack’s error), or anything else in the build throws.What the run prints: The build prints ‘::error::<first line of the error>’, writes it to the step summary, and exits 1. The publish step does not run.Alert text:
State: slice_run.build_failed. Alerts: A4.Code: src/catalog/fetch.ts:419, scripts/build-catalog-slices.ts:221
When: The census came back short and a floor refused the slice for it.What the run prints: The build prints the reason as ‘::error::<reason>’, writes the same line at the top of the step summary ($GITHUB_STEP_SUMMARY), and exits 1. The publish step does not run, so Storage keeps the last good slice. The census loader itself still says nothing (gap G9).Alert text:
State: slice_run.build_failed. Alerts: A4.Code: src/catalog/paths.ts:26, src/catalog/paths.ts:34, scripts/build-catalog-slices.ts:260
When: tooThin refuses: the merged catalog is under 20,000 channels.What the run prints: The build prints the reason as ‘::error::<reason>’, writes the same line at the top of the step summary ($GITHUB_STEP_SUMMARY), and exits 1. The publish step does not run, so Storage keeps the last good slice.Alert text:
State: slice_run.build_failed. Alerts: A4.Code: scripts/build-catalog-slices.ts:56, scripts/build-catalog-slices.ts:260
When: tooThin refuses: a platform slice is under 5,000 channels.What the run prints: The build prints the reason as ‘::error::<reason>’, writes the same line at the top of the step summary ($GITHUB_STEP_SUMMARY), and exits 1. The publish step does not run, so Storage keeps the last good slice.Alert text:
State: slice_run.build_failed. Alerts: A4.Code: scripts/build-catalog-slices.ts:57, scripts/build-catalog-slices.ts:260
When: unfiled refuses: no slice channel has a census path.What the run prints: The build prints the reason as ‘::error::<reason>’, writes the same line at the top of the step summary ($GITHUB_STEP_SUMMARY), and exits 1. The publish step does not run, so Storage keeps the last good slice.Alert text:
State: slice_run.build_failed. Alerts: A4.Code: scripts/build-catalog-slices.ts:83, scripts/build-catalog-slices.ts:260
When: unfiled refuses: under 10,000 slice channels have a census path.What the run prints: The build prints the reason as ‘::error::<reason>’, writes the same line at the top of the step summary ($GITHUB_STEP_SUMMARY), and exits 1. The publish step does not run, so Storage keeps the last good slice.Alert text:
State: slice_run.build_failed. Alerts: A4.Code: scripts/build-catalog-slices.ts:90, scripts/build-catalog-slices.ts:260
When: Cannot happen while the build exits 1 on every refusal and crash: the publish step runs only after a green build step (an if: without a status function implies success()), and the build step is green only after writing both files.What the run prints: If it ever did happen, publish-object.mjs prints ‘::error::ENOENT …’ and exits 1, and the alert carries that line.Alert text:
State: slice_run.publish_failed. Alerts: A4.Code: scripts/build-catalog-slices.ts:264, .github/workflows/catalog-slices.yml:70, scripts/publish-object.mjs:89
When: publish() throws when the Supabase URL or service key is missing (scripts/publish-object.mjs:68, :69).What the run prints: The publish step prints ‘::error::<message>’ and exits 1. The Telegram message carries that line as its reason.Alert text:
State: slice_run.publish_failed. Alerts: A4.Code: scripts/publish-object.mjs:72, scripts/alert-on-change.mjs:62
When: Storage refuses an upload or the bucket check; upload or ensureBucket throws with the status and body (scripts/publish-object.mjs:42, :62). The parts step uses the same upload, so the line can name a part or an index; the whole slices are already up then.What the run prints: The publish step prints ‘::error::<message>’ with the HTTP status and exits 1. The Telegram message carries that line as its reason.Alert text:
State: slice_run.publish_failed. Alerts: A4.Code: scripts/publish-object.mjs:65, scripts/alert-on-change.mjs:62
When: At the daily 14:00 UTC check, catalog/web.json or catalog/native.json is older than 16 hours (guide/epg.json: 13).What the run prints: Telegram ‘Bytestream data has stopped arriving.’ with each stale object’s age, once a day, and one JSON line per object in the Vercel log: {object, lastModified, ageHours, limitHours, stale}.Alert text:
State: published_slice.late. Alerts: A5.Code: api/watchdog.ts:37, api/watchdog.ts:62, api/watchdog.ts:125Deployment: api/watchdog.ts runs in the Vercel production deploy, which is pinned to the 2026-09-30 rollback 65fa7c1 (autoAssignCustomDomains: false). Production runs the 7-hour watchdog with no log lines until Jake promotes a deploy that contains this change.
When: A healthy pipeline whose last publish is more than 16 hours old. The longest measured gap between run starts is 13.0 hours, so this should not occur.What the run prints: The same message as a real outage; the per-object log lines show ageHours against limitHours, so a reader can tell a late schedule from a stopped one.Alert text:
State: published_slice.late. Alerts: A5.Code: .github/workflows/catalog-slices.yml:14, api/watchdog.ts:37Deployment: api/watchdog.ts runs in the Vercel production deploy, which is pinned to the 2026-09-30 rollback 65fa7c1 (autoAssignCustomDomains: false). Production runs the 7-hour watchdog with no log lines until Jake promotes a deploy that contains this change.
When: usableSlice returns null for generatedAt older than 72 hours (src/catalog/slice.ts:100); clients ignore the slice.What the run prints: The watchdog’s log line for each slice shows its age, and the daily message names it. The app reports nothing about the switch (gap G6).Alert text:
State: published_slice.expired. Alerts: A5.Code: src/catalog/slice.ts:100, api/watchdog.ts:125Deployment: api/watchdog.ts runs in the Vercel production deploy, which is pinned to the 2026-09-30 rollback 65fa7c1 (autoAssignCustomDomains: false). Production runs the 7-hour watchdog with no log lines until Jake promotes a deploy that contains this change.
When: Telegram answers anything but 200.What the run prints: The Alerts run turns red and prints Telegram’s answer. The watchdog logs {event: ‘watchdog.telegram’, sent: false, status} in the Vercel log, so a failed send is on record.Alert text:
State: alert.send_failed. Alerts: none.Code: scripts/alert-on-change.mjs:103, api/watchdog.ts:130Deployment: api/watchdog.ts runs in the Vercel production deploy, which is pinned to the 2026-09-30 rollback 65fa7c1 (autoAssignCustomDomains: false). Production runs the 7-hour watchdog with no log lines until Jake promotes a deploy that contains this change.
When: epg.yml turns red, or guide/epg.json is older than 13 hours at the daily check.What the run prints: The red run alerts with its reason, only when something changed (A4); the watchdog keeps 13 hours for the guide and logs its age (A5).Alert text:
State: other_job.failed. Alerts: A4, A5.Code: .github/workflows/alerts.yml:15, api/watchdog.ts:39Deployment: api/watchdog.ts runs in the Vercel production deploy, which is pinned to the 2026-09-30 rollback 65fa7c1 (autoAssignCustomDomains: false). Production runs the 7-hour watchdog with no log lines until Jake promotes a deploy that contains this change.
When: recs.yml turns red. Since D12 the similar step also fails, before it writes, when the native slice is missing, of another version or stale.What the run prints: The red run alerts with its reason, only when something changed (A4).Alert text:
State: other_job.failed. Alerts: A4.Code: .github/workflows/alerts.yml:15
When: A call the alert job makes to read the failed run’s annotations or the earlier runs answers non-2xx or times out.What the run prints: A failure is always sent: with ‘reason not readable (GitHub API <status>)’ in place of the reason when the run’s annotations cannot be read, or with ‘(earlier runs not readable)’ in the first line when only the run history cannot be read. The job prints the status in its log. A success whose history cannot be read sends nothing and prints ‘no alert: <workflow> success, earlier runs not readable (GitHub API <status>)’.Alert text:
State: alert.sent. Alerts: A4.Code: scripts/alert-on-change.mjs:252, scripts/alert-on-change.mjs:276
When: writeParts throws for either platform. The build removes both indexes, prints the reason and exits 0.What the run prints: The build prints ‘::error::catalog parts not written: <first error line>’ and puts the same line at the top of the step summary, then exits 0. The slices publish. The parts step prints ‘::error::catalog parts: web-index.json was not written by the build’ and exits 1, so the run is red.Alert text:
State: slice_run.parts_failed. Alerts: A4.Code: scripts/build-catalog-slices.ts:339, scripts/publish-catalog-parts.mjs:58
When: publish-catalog-parts.mjs reads both indexes and every part they name before any upload, and throws on a missing file or a SHA-256 that does not start with the hash in the name.What the run prints: The parts step prints ‘::error::catalog parts: <what and which file>’ and exits 1 before uploading anything. The whole slices are already up.Alert text:
State: slice_run.parts_failed. Alerts: A4.Code: scripts/publish-catalog-parts.mjs:70, scripts/publish-catalog-parts.mjs:91
When: loadRankings gets no Supabase URL or key, or a read of channel_stats, channel_similar or channels fails, and returns null for that ranking; or it throws.What the run prints: The build prints ‘::warning::rankings not read: trending’ (or ‘similar for <channel id>’, or the first error line), puts ‘Rankings not read: <list>. The indexes went out without those pick lists.’ at the top of the run’s summary, and publishes the index without those lists. The run stays green. The Monday metrics digest lists every ranking left out of any Catalog slices run in the past 7 days.Alert text:
State: slice_run.written. Alerts: A6.Code: scripts/build-catalog-slices.ts:311, scripts/build-catalog-slices.ts:291, src/metrics/digest.ts:80, scripts/metrics-digest.ts:54

Events

The pipeline has no event bus. Its events are the lines its scripts print, the run results GitHub records, the Telegram messages, the exceptions Sentry captures and one in-app change. Each printed event has a regular expression the validator matches against real run logs.

Observability and alerts

Alerts

How the red-run alert decides to send (A4)

Catalog slices, Refresh EPG or Recommendations completes, any conclusion. A message goes out on the first failure after a success, on a failure whose reason differs from the last alert’s, on the first success after a failure (‘back to normal’), and once a day while a workflow stays red with the same reason. Every other run sends nothing and the job log says why. The first ::error:: annotation of the failed run, read with GET /repos/sachio222/atlas/actions/runs/{id}/jobs and GET /repos/sachio222/atlas/check-runs/{job id}/annotations. If the run has none, the name of the first failed step and its conclusion. Reasons are compared with every run of digits replaced by #, so ‘web slice has 9818 channels’ and ‘web slice has 9875 channels’ count as one reason.
  • Where the last alert is kept: Nowhere new. The job derives it on every run from GitHub: the last 100 completed runs of the same workflow on the run’s branch, main for scheduled runs (GET /repos/sachio222/atlas/actions/workflows/{id}/runs?status=completed&branch=<branch>), with each run’s conclusion and completion time, and the reason of every failed run since the last success. Replaying the rule over them gives the last alert sent, its reason and its time.
  • Reruns: A rerun is the same run. Only the latest attempt of an earlier run counts, and it replaces that run’s earlier conclusion. The run being alerted on keeps its previous attempt (GET …/runs/{id}/attempts/{n-1}) as the state before it, so a rerun that turns the latest failure green is a recovery.
  • Cancelled runs: A cancelled Catalog slices run carries no news: it sends nothing, does not end a red streak, does not count as a recovery, and is logged as suppressed. A run that never finishes is the watchdog’s to catch. Refresh EPG and Recommendations alert on cancelled as today, because they are not re-run every few hours.
  • Long streaks: If the red streak is longer than 100 runs, the oldest visible failure counts as the first.
  • Failed sends: The replay assumes earlier messages were delivered. A failed send shows as a red Alerts run (alert_send_failed).
  • GitHub API errors: A failure is always sent when the GitHub API fails: with ‘reason not readable’ when the run’s annotations cannot be read, with ‘earlier runs not readable’ when only the history cannot. A success whose history cannot be read sends nothing and says so in the log.
  • Permissions: The alerts job has actions: read (the workflow’s runs and each run’s jobs), checks: read (the jobs’ annotations) and contents: read (the sparse checkout of the script). It had none before D3.
  • Measured: Replayed over the incident (the last green run, the 23 red runs from 2026-11-10 21:20 to 2026-11-18 11:57 UTC, then the green run after the fix), the rule sends 8 messages: the first failure, 6 daily reminders and ‘back to normal after 23 failed runs’. A1 sent 23 and nothing on recovery. Proven by scripts/alert-on-change.test.ts, which runs the whole job once after each of those runs against a fake GitHub.
  • Before: Replaced A1, which sent ‘Bytestream job <workflow>: <conclusion>’, the short hash and the run link for every run that did not succeed: 23 identical messages from 2026-11-10 21:21 to 2026-11-18 11:58 UTC, and nothing on recovery.

The watchdog (A5)

GitHub starts the 3-hourly schedule 1.7 to 13.0 hours apart (median 6.6). 16 hours is the longest measured gap plus one scheduled interval. The 7-hour limit would have called a healthy pipeline stopped on 6 of the 47 healthy days from 2026-09-25 to 2026-11-10; no measured gap reaches 16 hours. Every run, stale or not: one JSON line per object, {object, lastModified, ageHours, limitHours, stale}, then {event: ‘watchdog.telegram’, sent, due, status}.
api/watchdog.ts runs in the Vercel production deploy, which is pinned to the 2026-09-30 rollback 65fa7c1 (autoAssignCustomDomains: false). Production runs the 7-hour watchdog with no log lines until Jake promotes a deploy that contains this change.

Where signals go

Numbers to watch

Gaps found while writing the contract

Ranked by how much each would have shortened the 2026-11-10 incident, with the decision that closes it.
  1. A dropped source is reported nowhere Closed by D1. Closed by D1, built on feat/catalog-alerts (9d91712).
  2. The Telegram alert carries no reason Closed by D3. Closed by D3, built on feat/catalog-alerts (7aff1d5).
  3. The build’s ‘never red’ exit disagrees with the workflow going red on ENOENT Closed by D2. Closed by D2, built on feat/catalog-alerts (6c12db7).
  4. The floors do not catch losing famelack Closed by D1. Closed by D1, built on feat/catalog-alerts (9d91712).
  5. Every red run sends the same message Closed by D4. Closed by D4, built on feat/catalog-alerts (ce647a8, 616f1d2).
  6. The watchdog’s result is recorded nowhere Closed by D5, D8. D5 logs every watchdog check and send in the Vercel log; D8 wraps the watchdog in withSentry. Both reach production only when Jake promotes. The app’s own slice-unusable and catalog-error outcomes stay visible only on the device.
  7. The watchdog’s 7-hour limit does not fit the real schedule Closed by D5. Closed by D5, built on feat/catalog-alerts (d3d4a61). D5 moves the catalog limit to 16 hours.
  8. The app names only four missing sources Open.
  9. The census loader fails silently Open.
  10. Docs and comments that disagree with the code Closed by D7, D8. D7, D8 and these changes fixed the README, the build script header and the two docs pages. The slice.ts comment and the cache-control mismatch remain.
  11. A failed prune is a warning on a green run Open.
  12. An index built without its rankings reaches no one Closed by D10. D10 puts the rankings left out at the top of the run’s summary and in the Monday metrics digest (A6). The run stays green and still publishes the index without those lists.

Decisions

Taken 2026-11-18. Each decision lists every contract row it touches, and each touched row names the decision back; the validator checks both directions.

D1: famelack and iptv-org are required sources

Live. Settles Q1 and Q4. Touches 34 rows in api, data, events, failures, golden, observability, state. Decision: famelack and iptv-org are required. If either is missing from a build (sources.<name> === false), the build refuses and exits 1 with ‘required source <name> did not answer: <first error>’. Every other source is optional: a missing optional source is listed in the build summary and in the Monday metrics digest, and never fails the build. The three floors stay as a second line of defence. Reason: For eight days famelack was missing and only the census floor, short by 124 channels, kept the build from publishing a catalog without it. The census count was rising; past 10,000 the runs would have gone green with famelack still missing. A refusal that names the source puts the cause in the first red run. The optional sources come and go on their own, so they belong in a weekly list, not a page. Built: on feat/catalog-alerts in 9d91712.

D2: A refused build fails loudly

Live. Settles Q2. Touches 29 rows in data, events, failures, golden, observability, state. Decision: scripts/build-catalog-slices.ts exits 1 with the reason whenever it refuses (a floor, the census, a required source), and writes the reason to $GITHUB_STEP_SUMMARY and as a ::error:: annotation. The publish step never runs on a missing file. A crash is still exit 1, with its first error line. Reason: The build exits 0 on every path because it once ran inside every Vercel deploy, where a red step blocked the deploy. It now runs only in its own workflow, so the rule protects nothing. Today the run goes red one step late, on ENOENT, and the reason sits in a log nobody opens. Built: on feat/catalog-alerts in 6c12db7.

D3: The alert carries the reason

Live. Settles Q3. Touches 13 rows in events, failures, golden, observability, state. Decision: The Telegram text names the workflow, says what went wrong (the first error line or annotation from the failed run, read through the GitHub API), and gives the run URL. Reason: Every one of the 23 alerts in the incident said only ‘Catalog slices: failure’. The cause took a log read to find each time. Built: on feat/catalog-alerts in 7aff1d5. The contract named scripts/alert-on-change.ts. It is scripts/alert-on-change.mjs so the Alerts job runs it with plain node and no install, like scripts/publish-object.mjs.

D4: Alert on change, not on every run

Live. Settles Q6. Touches 22 rows in data, events, failures, golden, observability, state. Decision: Send on the first failure after a success, when the reason changes, on recovery (the first success after a failure, ‘back to normal’), and at most once a day while it stays the same. Everything else is suppressed and logged in the job output. The last-alert state is derived from earlier runs through the GitHub API, with no new storage. A rerun counts as its run’s latest attempt; a cancelled Catalog slices run counts as no news. Reason: 23 identical messages in eight days teach people to mute the group. Replayed over the same runs, the rule sends 8: the first failure, 6 daily reminders, and a recovery message that says when it is over. Built: on feat/catalog-alerts in ce647a8, 616f1d2.

D5: Watchdog limit 16 hours, and a log line per check

Live. Settles Q5 and Q7. Touches 24 rows in api, data, events, failures, golden, observability, state. Decision: The watchdog’s catalog limit goes from 7 hours to 16 (the guide keeps 13). It logs one structured line per object checked ({object, lastModified, ageHours, limitHours, stale}) and whether Telegram was sent. api/watchdog.ts runs in the Vercel production deploy, which is pinned to the 2026-09-30 rollback 65fa7c1 (autoAssignCustomDomains: false), so these changes reach production only when Jake promotes. Reason: GitHub starts the 3-hourly schedule a median 6.6 hours apart and up to 13 hours apart, so 7 hours called a healthy pipeline stopped on 6 of 47 days. 16 hours is the longest gap plus one interval. Today the watchdog’s result goes back to the cron and is thrown away, so nobody can tell whether it ran or sent anything. Settled since: Wrapping the watchdog in withSentry, left open as Q10, is decided as D8. Built: on feat/catalog-alerts in d3d4a61.

D6: Read famelack from tv/

Live. Settles Q8. Touches 4 rows in api, data, golden. Decision: FAMELACK_BASE moves to https://raw.githubusercontent.com/famelack/famelack-data/main/tv/. Built on fix/famelack-tv-path and merged to main as 4fbfe14 (PR #230) on 2026-11-18. Reason: famelack moved its files from tv/raw/ to tv/ on 2026-11-10. Measured with tv/, the build was back to 17,000 web channels and 15,156 with a census desk; the first run after the merge (44363276494) published 17,000 web and 18,851 native channels. Merged: 4fbfe14 (PR #230).

D7: Grafana alerts use Grafana’s own Telegram contact point

Live. Settles Q9. Touches 3 rows in docs, observability. Decision: Grafana alerts reach Telegram through Grafana’s own contact point, not api/alerts.ts. live/README.md is corrected to say so. Reason: api/alerts.ts and scripts/grafana-setup.sh already work this way; only the README said otherwise.

D8: The watchdog is wrapped in withSentry

Live. Settles Q10. Touches 5 rows in api, events, observability. Decision: api/watchdog.ts exports withSentry(handler, ‘watchdog’), like api/epg.ts, api/alerts.ts and api/catalog.ts. A throw in the watchdog reaches Sentry project bytestream-api. It reaches production only when Jake promotes, like D5. Reason: The watchdog was the one route a throw in which reached no one, and live/README.md said withSentry wraps every route. The wrapper lives in api/_lib/sentry.ts, which Vercel does not deploy as a function, so it adds no function file under the Hobby plan’s cap of 12, and it passes straight through when SENTRY_DSN is unset. Built: on feat/catalog-alerts in 692f833, e7f6847.

D9: Watch paints from a small index while the catalogue arrives in parts

Live. Settles no open question. Touches 70 rows in api, data, events, failures, observability, state. Decision: Each slice run also writes the slice as an index and five parts (channels, streams, a schedule window, the whole schedule, blurbs), each part named for the first 16 hex digits of its SHA-256. The workflow publishes web.json and native.json first, unchanged, then every part with cache-control public, max-age=31536000, immutable, then the two indexes with max-age=300, after checking every part against its name. The app reads the index first and paints Watch’s default shelves from its first view, then loads channels and streams; schedules and blurbs follow when the page is idle or the Guide opens. Any failure goes on to the whole slice, then the sources. A prune removes parts no upload has touched for four days that the current index does not name. Reason: A web slice is one document of about 23 MB, about 2 MB brotli, and every cold load downloaded and parsed all of it before the first shelf. The index is about 21 KB brotli. Production, pinned to 65fa7c1, reads web.json at sliceVersion 5, so the slices keep their names, version and shape. Built: on perf/catalog-parts in e8391c9, b05d47a, 58cab7a, 5bd8671. Cherry-picked from staging (7d3989b, 0e502b2) onto main, with the publish order, the hash checks and the prune fix added. Live. Settles no open question. Touches 19 rows in api, data, events, failures, observability. Decision: Each slice run reads op=trending’s ranking (channel_stats) and op=similar’s for the first PICKS_TUNES channels of the first view’s first row (channel_similar), as the API reads them, and works out You might like for each of those channels and Trending on Bytestream with the functions Watch runs, over the whole slice. The index (version 2) carries each list with PICKS_SPARE spares in first.picks, and the cards of their channels in the first view. Watch draws You might like from the picks until the core is in, and Trending from them until op=trending answers; a later answer adds its other channels after the drawn ones. You might like’s fallback, when op=similar has nothing, ranks by language, desk, a stream no territory limits, services and country instead of catalogue order. The client still reads index version 1. A ranking the build cannot read is a warning annotation and a line at the top of the run’s summary, and the Monday metrics digest (A6) lists the week’s; the run stays green and publishes the index without that list. Reason: A cold load tunes NHK World-Japan, for which op=similar has nothing, and the first view holds 148 of the 17,033 channels. You might like came out as six English news channels and six Catalan ones; over the whole catalogue it was English news from A to Z (A News, Abante TV, ABC 7 Bay Area), only 5 of the first 12 shared. op=trending ranks 40 channels and the first view held 3 of them, at ranks 11, 29 and 40, so those three led Trending for the whole visit once Watch kept each row’s channels (first-load INV-8). An answer from op=trending after the core also inserted the Trending row and pushed the rows under it down. Built: on feat/firstview-picks in e9fc63a, fa7a4ee, 5dc05d7, not merged yet. Takes effect for viewers once the catalog-slices workflow publishes an index of version 2.

D11: Each channel’s streams are published in the order the app should try them

Live. Settles no open question. Touches 1 rows in data. Decision: The build orders each channel’s streams before it writes the slice. The order is the one orderStreams gives a viewer with no history: HLS before an embed before DASH, then a stream on a relay host or one iptv-org labels Geo-blocked or Not 24/7 behind the rest, then famelack before other sources, then higher quality first. Among streams orderStreams ranks equally, the channel’s own host goes first (INV-16). The parts and the index’s first view take the same order, so the index stays at version 2 and web.json at version 5 with its shape unchanged. Reason: The app tries a channel’s streams in orderStreams’ order and keeps the slice’s order where orderStreams ranks two streams equally. A new visitor is tuned to NHK World-Japan, which has seven HLS streams, four of them iptv-org rows at 1080p. orderStreams ranked those four equally and the slice listed freem3u.xyz first, an address that redirects twice to a Vietnamese OTT service and reached its first video segment after 1.4 to 2.8 s in five tries. The broadcaster’s own masterpl.hls.nhkworld.jp, fourth in the slice, took 0.1 to 1.4 s. The pinned production deploy runs an older orderStreams over web.json, so the slice’s order is the only part of this change that reaches it. Over the published catalog of 2026-11-19 the build changes the first stream of 84 of 17,036 channels. Probed twice each from one machine, the new first streams worked on 63% of tries and the old ones on 53%, and where both worked the new one reached its first segment sooner on 22 channels of 37. Built: on perf/stream-order in 327255d, not merged yet. Takes effect for viewers once the catalog-slices workflow publishes a slice built from this code.

D12: op=similar ranks with the device’s function over the published catalogue

Live. Settles no open question. Touches 9 rows in api, data, failures. Decision: The nightly recommendations run (scripts/build-channel-similar.ts) reads the native slice the catalog-slices workflow publishes and ranks each channel’s playable peers with rankPeers (src/catalog/recommend.ts), the function the device runs as youMightLike when op=similar has nothing: the tuned channel’s first language, its census desk, how many of its other categories the peer shares, a stream no territory limits, watches in the last seven days (the run has them; the device does not), services, country. Catalogue order breaks what is left. No family of numbered or format variants (channelFamily: ABC News Live 1 to 10) holds more than FAMILY_CAP of the first FAMILY_WINDOW places, and the tuned channel counts as one of its own family. Each channel keeps its top TOP_N in channel_similar, and op=similar’s answer keeps its shape. Then each RT and CGTN channel in every variant (STATE_MEDIA, matched by channelFamily) moves STATE_MEDIA_DEMOTE places down from where the ranking put it, or to the end of a shorter list, and everything else keeps its order; two that land on one place keep theirs. The move comes before the family cap. An editorial choice by Jake, 2026-11-19: they still show, just not first. The run writes channel_similar and nothing else. Both its columns reference public.channels, so a channel the table lacks gets no rows and is no one’s peer, the run logs how many it skipped, and for that channel op=similar answers with nothing and the device ranks You might like with the same rankPeers (INV-15). A slice that is missing, of another version or stale fails the run before anything is written, and yesterday’s rows stand. Reason: On 2026-11-19 op=similar answered BBC News, France 24 English, Al Jazeera English, DW English and TRT World with one list: 24/7 News, 2GB, 3AW, 48 News UK, 4BC, A1TV Pakistan, A1TV UK, ABC News, ABC News GH, ABC News Live 1, ABC News Live 10, ABC News Live 2. The run scored cosine similarity over category vectors that were one-hot news for every news channel, added the same language boost to every English one, and had watches for 40 channels, so every English news channel scored 1.15 and each list fell to id order. Since D10 the index carries these lists, so viewers saw them from the first frame. NHK World-Japan (fl:bt1m1v4h1CmExI), the channel a cold load tunes, gets no list because public.channels, last synced by hand on 2026-09-12, lacks that famelack id; it keeps the device’s ranking, which this decision brings in line with the run’s. Once the ties were broken, CGTN and RT ranked 2nd and 3rd for most English world news channels, since they carry top-news and the world desk. Built: on fix/similar-ties in 85f8dae, 92f7d95, b526a50, not merged yet. Takes effect at the first recs.yml run after the merge (04:15 UTC), and in the indexes’ picks at the catalog-slices run after that.

Open questions

Questions still open after the 2026-11-18 decisions (decisions.json). Q10, the last one, became D8. Nothing is open.

Run the validator

The validator checks the contracts against each other, against the code at the contracted commit, against the golden runs and against the live pipeline. It sends nothing and writes nothing.
1

Load the environment

From the atlas repo root, load .env.local so the live checks can reach Supabase:
2

Run it

Add --offline to skip the live checks, or --no-gh to skip the GitHub run checks.
3

Rebuild the pages after editing a contract

Exit 0: the contracts pass and every live failure is explained, by a known incident or by a contracted commit that main does not run yet. Exit 1: a contract check failed. Exit 2: the contracts pass but a live check failed for a reason nothing explains.

Machine files