System & data flow
Once a day a scheduled Worker collects community activity, turns it into threads, and publishes an immutable snapshot to R2; a static web app reads that snapshot through a small read-only API. Click any step to see what it does, the numbers from the last run, and the files that implement it.
The flow
1 · Collection
2 · Processing
3 · Storage
4 · Application
Daily publication cron
Each environment has its own Worker with its own cron, so Dev and Prod never share state. Publication was hourly until 2026-10-13 and is now daily to cut R2 cost (Spec 018).
- Dev
7 22 * * *(06:07 Asia/Taipei)- Prod
37 22 * * *- Trade-off
- Data can be up to about 24 hours old; a token-protected manual trigger publishes sooner.
Sources: apps/data-publisher-worker/wrangler.development.jsonc, apps/data-publisher-worker/wrangler.production.jsonc, docs/specs/018-daily-publication/spec.md
Digest cron Dev only
Forty minutes after the Dev publication, the weekly digest reads the snapshot that was just published. If a publication is still running it waits 15 minutes, at most 4 times.
Sources: docs/specs/014-topic-digest/spec.md, docs/runbooks/cloudflare-data-publisher.md
Catch-up and retries
A run that hit a source's per-run cap runs again 5 minutes later (at most 5 times); a failed run is retried after 10, 30 and 60 minutes. One trigger causes at most 9 publications.
Sources: apps/data-publisher-worker/src/run-schedule.ts, docs/specs/018-daily-publication/spec.md
GitHub pull requests and issues
Read with the GitHub REST API from an updated-time cursor, at most 200 issues per repository per run; the rest is caught up by follow-up runs.
Sources: docs/feature-map.md, docs/specs/004-replayable-github-feed/spec.md
Jira (KAFKA project)
Anonymous reads of the Apache Jira search API over a window that grows with the gap since the last success.
Sources: packages/reference-pipeline/src/kafka-connectors.ts, docs/specs/012-kafka-mailing-list-jira/spec.md
Kafka dev@ mailing list
KIP discussions and votes live here. Subjects starting [jira] or [PR] are dropped because those records already come from Jira and GitHub.
- Shared rules
- One retry after a 5xx or 429; a response over 4 MiB fails that source as
too-large; a failing source never blocks the others.
Sources: packages/reference-pipeline/src/kafka-connectors.ts, docs/specs/012-kafka-mailing-list-jira/spec.md
GitHub pull request reviews Dev only
Who reviewed what and when, without comment text; it feeds contributor profiles.
PipelineState
Holds each source's cursor, the current phase and the last run, so a crash resumes from the last committed cursor instead of starting over.
Sources: apps/data-publisher-worker/src/index.ts, docs/architecture/decisions/0014-publish-per-source-with-independent-cursors.md
Poll the sources
Reads every source sequentially from its cursor.
Materialize threads
A deterministic TypeScript materializer groups records into threads and extracts identifiers such as KIP-1150 from titles and bodies. The same input always gives the same output.
Sources: packages/reference-pipeline/src/materializer.ts, docs/architecture/decisions/0006-use-typescript-reference-materializer-before-flink.md
Write the Feed release
An immutable snapshot: a list of entries plus one detail object per thread. A detail is stored once under the SHA-256 of its bytes and reused by later releases (content addressing).
- Last run
- Dev 791 new + 7,796 reused objects; Prod 378 new + 7,596 reused
Sources: packages/serving-contract/src/r2.ts, docs/specs/008-content-addressed-details/spec.md
Build the search index
Chunks every thread, computes corpus statistics once, and writes BM25 shards plus a terms.json that says which shard holds each term. See Search.
- Last run
- Dev 9,764 chunks in 11 shards; Prod 7,118 chunks in 8 shards
Sources: packages/serving-contract/src/search-r2.ts, docs/specs/013-search-streaming/spec.md
Switch pointers, record evidence
Objects are written first and the small current.json pointers last, so a reader sees either the old release or the new one, never half of one. The run then records what it wrote and what it cost.
- R2 per run
- Dev 794 Class A / 8,590 Class B; Prod 381 / 7,977
- Invariant
- Class A ≤ new objects + 3
Sources: apps/data-publisher-worker/src/pipeline.ts, docs/specs/006-environment-isolated-r2-publication/spec.md
Append the activity log Dev only
Per-person monthly activity files for contributor profiles (PEOPLE_ACTIVITY=log@1).
Embed new chunks Dev only
Turns each new search chunk into a vector for semantic search, capped at 2,500 estimated neurons and 400 calls a day.
Weekly topic digest Dev only
Classifies, summarises, validates and translates the last 7 days. See Weekly digest.
Pointers
The only mutable objects. Everything they point to is immutable, so rollback is a pointer change.
Feed releases and details
One release per run; details shared by content digest across releases.
Sources: docs/architecture/decisions/0013-share-detail-objects-by-content-digest.md
Search releases
Versioned independently of the feed, so the ranker can change without touching the feed.
Sources: docs/architecture/decisions/0007-publish-independent-versioned-search-releases.md
Digest objects Dev only
The English digest and its Traditional Chinese translation for one source release.
Sources: docs/specs/014-topic-digest/spec.md
Internal R2 objects
Evidence of each publication, and on Dev the review and activity month files. Not served directly.
Vectorize index Dev only
Cloudflare's vector database, behind a replaceable interface so it is never the public contract.
Search query gate Dev only
A Durable Object ledger that every semantic query must pass, so a public search box cannot overspend. Over budget means lexical results only.
Feed and detail
Reads the pointer, then the release, then one detail object. Every 200 response is cached for 30 seconds with stale-while-revalidate 120. The browser never touches R2.
Sources: apps/web/functions/api, docs/architecture/decisions/0004-use-pages-functions-and-r2-serving-projections.md
Search
Reads current.json, the manifest, terms.json, then only the shards a query term names.
Digest and people Prod 404
Return 404 on Prod because the features are off there.
Sources: docs/feature-map.md
This week content on Dev
The digest: headline, highlights, KIP proposals, topic cards, collapsed routine maintenance.
Sources: apps/web/src/routes.ts
All threads and search
The thread list with search in the same view, and a detail page per thread with its full timeline and source links.
Sources: apps/web/src/routes.ts
Freshness indicator
Every view shows how old the data is, so a stalled publisher is visible without anyone reading logs.
Sources: apps/web/src/freshness.ts, docs/specs/021-publication-overdue/spec.md
/health
Public, read-only status of the last run: per-source success, objects written, R2 requests, search index size, AI spend today. It never returns credentials.
verify:health
A committed command that compares /health with what the site serves, for Dev or Prod, using GET only.
Sources: scripts/verify/health.ts
Run numbers are from the Dev and Prod publishers' public /health read on 2026-10-20 (last runs 2026-10-19 22:07 and 22:37 UTC).
The four stages
- Collection. A cron trigger starts the publisher in each environment. It reads GitHub, Jira and the Kafka dev@ list from per-source cursors kept in a Durable Object, so one failing source never blocks the others and a crash resumes where it stopped (ADR-0014).
- Processing.
runDataPublicationin apps/data-publisher-worker/src/pipeline.ts materializes threads with deterministic rules (no language model in the publication path), writes the Feed release, and builds the search index. - Storage. R2 holds immutable releases and content-addressed detail objects; only small
current.jsonpointers change, and they are written last. A reader therefore sees one complete release, and rollback is a pointer change (Spec 006, Spec 008). - Application. Pages Functions read R2 and return JSON with a 30-second cache; the Vue app renders it. Code deploys never publish data: data comes only from the Worker (docs/operations/release.md).
Dev and Prod
The two environments are fully isolated: separate Workers, Durable Objects, R2 buckets and Pages projects, and no shared write credential (ADR-0012).
| Aspect | Dev | Prod |
|---|---|---|
| Web app | oss-knowledge-base-dev.pages.dev | oss-knowledge-base.pages.dev |
| Deployed by | every push to main, after all CI jobs pass, then checked by deployed end-to-end tests | a v* tag reachable from main (current v0.3.1, 2026-10-19) |
| Publication cron | 22:07 UTC | 22:37 UTC |
Lexical search bm25-reference@3 | on | on |
| Weekly digest | on (22:47 UTC) | off |
| Semantic hybrid search | on (SEARCH_SEMANTIC=rrf-group@2) | off |
| Contributor profiles, review ingestion | on | off |
Every Dev-only feature is a flag that is unset on Prod, listed in docs/feature-map.md. New behaviour is merged with its flag off, then enabled per environment in a separate small change, so a rollback is a configuration change rather than a revert (docs/process/workflow.md).
Why it is built this way
- Snapshots, not a live database. A daily immutable release is cheap to serve from object storage, easy to reproduce, and lets every summary cite a fixed record. The target streaming architecture (Fluss, Flink, Iceberg) is accepted in ADR-0001 but not built; only a compatibility spike exists (Spec 007).
- Rules before models. Thread materialization and identifier extraction are deterministic TypeScript (ADR-0006); language models are used only after publication, for the digest and semantic search, each with its own daily budget.
- Visible staleness. Dev once published nothing for about 49 hours without anyone noticing (Spec 011). Every view now shows the data's age: late after 24 h 30 min, stale after 27 h (apps/web/src/freshness.ts).