{"id":"adr-0001-storage","title":"ADR-0001 — Hybrid GCS + R2 Storage","subtitle":"Why hybrid object storage, and why we deferred Cloudflare-fronted GCS.","category":"operations","tags":["adr","infrastructure","storage"],"source":"articles/docs/decisions/0001-r2-over-bandwidth-alliance.md","lang":"en","words":900,"readMinutes":4,"toc":[{"depth":2,"text":"1. Context","id":"1-context"},{"depth":3,"text":"1.1 What exists today","id":"1-1-what-exists-today"},{"depth":3,"text":"1.2 What is missing vs. the blueprint","id":"1-2-what-is-missing-vs-the-blueprint"},{"depth":2,"text":"2. Decision","id":"2-decision"},{"depth":2,"text":"3. Consequences","id":"3-consequences"},{"depth":2,"text":"4. Action items","id":"4-action-items"},{"depth":3,"text":"4.1 In this PR (Path A scaffolding, no infra deploy)","id":"4-1-in-this-pr-path-a-scaffolding-no-infra-deploy"},{"depth":3,"text":"4.2 Deferred (requires production access)","id":"4-2-deferred-requires-production-access"},{"depth":3,"text":"4.3 Optional follow-up","id":"4-3-optional-follow-up"},{"depth":2,"text":"5. Inventory of hardcoded GCS URLs (replace in §4.2)","id":"5-inventory-of-hardcoded-gcs-urls-replace-in-4-2"},{"depth":2,"text":"6. References","id":"6-references"}],"html":"

ADR-0001: Hybrid GCS + R2 storage; defer Cloudflare-fronted GCS

Status: Accepted (interim) — supersedes a TBD by the migration plan in §4.\nDate: 2026-05-11\nAuthor: A. Welcing\nHandoff unit: 12 — "GCS → Cloudflare CDN for .glimbin (Bandwidth Alliance)"

\n
\n

1. Context

The (unwritten) handoff blueprint docs/handoff/03_gcp_heavy_workload_blueprint.md §2 calls\nfor "a Cloudflare CDN layer directly in front of our GCS buckets, [utilizing] the Bandwidth\nAlliance to drastically reduce or eliminate egress fees." That blueprint was never committed;\nthe actual implementation diverged. This ADR records what we actually built, why it diverged,\nand what would close the gap.

\n

1.1 What exists today

Two distinct object stores, each chosen for the workload it serves:

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
StoreBucketWorkloadConsumers
Google Cloud Storageshed-489901-atlas-artifactsLarge .lammpstrj and .glimbin molecular-dynamics trajectories (~0.9 GB across 16 open-data files, plus future user uploads)atlas-view (web)
Cloudflare R2glim-artifacts (binding ARTIFACTS)Diary entries, search caches, claim snapshots, agent stateglim-think Worker (Durable Objects + Workers AI)

GCS objects are served from https://storage.googleapis.com/shed-489901-atlas-artifacts/...\ndirectly to browsers, with Cache-Control: public, max-age=31536000, immutable and a CORS\npolicy locked to glim.lupine.dev, lupi.live, and localhost dev ports.

\n

The .glimbin streaming pipeline (atlas/atlas-view/packages/parsers/src/StreamingLoader.ts)\nuses HTTP Range Requests to fetch only the header + observed-frame slices — a 2 GB file\ntypically transfers ~30 MB per session. This dramatically blunts egress cost even without\nBandwidth Alliance.

\n

1.2 What is missing vs. the blueprint

The blueprint asked for a Cloudflare proxy in front of GCS so that egress would route through\nthe Bandwidth Alliance (zero or near-zero egress charges between Cloudflare and GCS Standard\nin the same region). We have:

\n\n

…but no Worker, no DNS record, and no consumer configuration wired to a CDN base URL.\nEvery consumer hardcodes storage.googleapis.com/... (eight files; see §5 below).

\n

2. Decision

    \n

  1. Keep the hybrid: GCS for cold-immutable artifact blobs, R2 for hot Worker-bound state.\nThe two stores are not redundant. R2 is bound directly to glim-think's Worker via\nwrangler.toml, which gives in-network reads at zero latency and zero egress — there\nis no Cloudflare/GCS substitute for an R2 binding inside a Worker. GCS hosts the\ntrajectories because the upload tooling (gsutil, glimbin_convert.py) and quota\nceiling are friendlier than R2 for multi-GB scientific datasets.

    \n
    2. \n

  2. Land the Cloudflare-fronted-GCS scaffolding in code now; defer the deploy. The\nmigration is one DNS record and one Worker route away. The work that can be done\nin the worktree — a centralized ATLAS_CDN_BASE config, a deployable proxy Worker,\na runbook — lands in this PR. The remaining work (cutover the consumers, prove\negress savings) is scheduled in §4.

    \n
    3. \n

  3. Do not migrate shed-489901-atlas-artifacts to R2. R2's free egress already\ncovers the Worker side; the trajectories are served straight to browsers and would\nneed a public R2 bucket or a Worker proxy regardless. The benefit of moving them\nis small relative to the upload-tooling churn.

    \n
    4. \n
\n

3. Consequences

Positive

\n\n

Negative

\n\n

Neutral

\n\n

4. Action items

4.1 In this PR (Path A scaffolding, no infra deploy)

\n

4.2 Deferred (requires production access)

\n

4.3 Optional follow-up

\n

5. Inventory of hardcoded GCS URLs (replace in §4.2)

atlas/atlas-view/tools/glimbin_convert.py\natlas/atlas-view/scripts/inject_open_md_entries.py\natlas/atlas-view/scripts/gcs/upload_open_data.sh\natlas/atlas-view/scripts/gcs/repoint_sourceurl_to_gcs.py\natlas/atlas-view/scripts/gcs/pull_open_data.sh\natlas/atlas-view/packages/ui/src/gallery-data.json\natlas/atlas-view/apps/web/public/gallery/open_data/README.md\n
\n

The Python and shell tooling rewrites are batched after Worker deploy because they\ngenerate the URLs that get baked into gallery-data.json.

\n

6. References

\n"}