# Session transcript — 2026-08-26 **Topic:** Supply-chain review of `lukewilliamboswell/roc-platform-template-go`, and a general hardening recommendation for Roc platform authors. --- ## User > Analyze the repo https://github.com/lukewilliamboswell/roc-platform-template-go and > https://github.com/lukewilliamboswell/roc-platform-template-go/actions/runs/32951144684/workflow. > Do we have sufficient protection against us overlooking the modification of a binary in a big PR > by an outside contributor? ## Claude ## Short answer: no The only binary check that runs on a pull request is **self-certifying** — a PR can change a vendored binary and its recorded hash in the same commit and CI goes green. I confirmed this by running the check against a tampered tree. ## What's actually in place 113 binaries are checked in under `platform/targets/**` (musl `crt1.o`/`libc.a`/`libzigc.a`/`libcompiler_rt.a` x 4 targets, MinGW `crt2.obj` + 23 `.lib` x 4 targets, plus `libSystem.tbd`). Two controls guard them: | Control | Trigger | Catches a tampered binary? | |---|---|---| | `--verify-hashes` (ci.yml:19) | every PR | **No** — compares files to `scripts/zig_runtime.sha256`, which is in the same PR | | `--check` (rebuild + byte-compare) | `push: main`, `workflow_dispatch` only | Yes — but only *after* the merge | ## Findings **1. PR-time check is self-referential.** `verify_checked_in_hashes()` (`vendor_zig_runtime.py:305`) diffs the tree against the manifest. Empirically: ``` ### tamper a binary AND its manifest line (simulating a PR that edits both) Verified all vendored runtime artifacts against scripts/zig_runtime.sha256. exit=0 ``` So detection collapses to a human noticing one changed line in `scripts/zig_runtime.sha256` among N changed files — exactly the failure mode you asked about. GitHub renders the `.a`/`.lib` itself as "Binary file not shown", so there is no diff to overlook, only a filename in a long list. **2. The reproduction check is deliberately kept off PRs, and the stated reason doesn't hold.** `SLSA_PROVENANCE.md` says "Pull requests only perform the checksum check because untrusted pull-request code must not receive attestation-writing permissions." That conflates two steps: the *attestation* needs `id-token: write`, the *reproduction* needs nothing but `contents: read`. Running `--check` on PRs — no attest step — catches this attack pre-merge. The provenance job takes **2m57s** vs CI's ~13m, so it's essentially free, and can be `paths`-gated on `platform/targets/**` + the vendoring scripts. One subtlety if you do this: the PR could also edit `vendor_zig_runtime.py` or `canonicalize_runtime_object.py` to fake the comparison. Run the **base branch's** copies of those scripts against the **PR's** files. **3. Manifest verification is allowlist-based; added files are invisible.** `generated_manifest()` walks a hardcoded `TARGET_ARTIFACTS` table. Dropping a new `platform/targets/x64musl/evil.o` into the tree passes `--verify-hashes` cleanly. `RUNTIME_PROVENANCE.md` claims CI "fails for a missing, changed, **unlisted**, or incorrectly recorded runtime artifact" — the *unlisted* claim is false. Mitigating factor: `bundle.py:41` ships an explicit allowlist, so a stray file isn't packaged unless the PR also edits `bundle.py`. Fix is a directory walk asserting the tree contains exactly the manifest's paths. **4. No branch protection, no rulesets, no CODEOWNERS.** `repos/.../branches/main` reports `protected: false` and `/rulesets` returns `[]`. Consequences: CI passing is not required to merge, and nothing forces an explicit approval on the sensitive paths. A `CODEOWNERS` covering `platform/targets/**`, `scripts/vendor_zig_runtime.py`, `scripts/zig_runtime.sha256`, and `.github/workflows/**` plus required-review is the cheapest control aimed squarely at "overlooked in a big PR" — it makes the sensitive paths announce themselves at review time instead of hiding in the file list. **5. Attestations can be minted from unmerged branches.** The run you linked (32951144684) is a `workflow_dispatch` on `fix/runtime-provenance-zig-download`, and the resulting attestation is bound to `refs/heads/fix/runtime-provenance-zig-download` @ `3329bd2` — 83 minutes before that branch merged. The consumer command in `SLSA_PROVENANCE.md` (`gh attestation verify --repo ...`) constrains the repository, not the ref, so an artifact attested from an unreviewed branch verifies as genuine. Not an outside-contributor path (dispatch needs write access), but it weakens "provenance means it came from reviewed `main`". Guard the attest step with `if: github.ref == 'refs/heads/main'`, or document verification with a `--cert-identity` that pins the ref. **6. Minor:** `ci.yml` uses mutable tags (`actions/checkout@v4`, `setup-python@v6`, `setup-go@v6`, `upload-artifact@v4`, `download-artifact@v4`) while `runtime-provenance.yml` SHA-pins everything. Same class of problem — unreviewed code you're trusting. Pin `ci.yml` too. ## What's genuinely good `pull_request` rather than `pull_request_target`, workflow-level `permissions: contents: read`, no secrets in CI — a fork PR can't exfiltrate or write anything. `libhost.a`/`host.lib` are gitignored, so the host is built from source and never a reviewable-binary problem. `host/go.mod` has no `require` block at all — zero third-party Go dependencies. Zig tarballs are SHA-256 pinned in both workflows, and the third-party setup actions are SHA-pinned. The reproducibility engineering itself (canonicalized paths, stable archive member ordering, stripped CodeView) is solid work — it's just not being pointed at the PR. ## Ranked fixes 1. Run `--check` on PRs touching `platform/targets/**` or the vendoring scripts, using the base branch's scripts. Closes the actual hole for ~3 min of CI. 2. Add CODEOWNERS + required review + required status checks on `main`. 3. Make `--verify-hashes` reject unlisted files, and correct the claim in `RUNTIME_PROVENANCE.md`. 4. Restrict attestation to `refs/heads/main`. Want me to write these up as a shareable page for the repo maintainer, or draft the workflow changes? --- ## User > What general solution would you recommend so that when other people with limited security > experience make their own Roc platform they have a low chance of leaving security holes. ## Claude ## The core insight Every control in that repo — checksum manifest, reproduction script, attestation — exists to compensate for one root fault: **binaries are committed to git and a human is nominally expected to review them.** Humans cannot review binaries. So the general solution isn't "teach platform authors to build better verification"; it's to remove the thing that needs verifying, and to ship the remainder as infrastructure they inherit rather than documentation they read. This is a family problem, not a one-repo problem. I checked the other platform repos: | Repo | Binaries in git | CI | `main` protected | |---|---|---|---| | template-go | 113 | ci + provenance | no | | template-zig | 4 | ci + release | no | | template-rust | 6 | ci + release | no | | template-c / -swift | 0 | **none** | — | | odin-platform-template (3rd party) | 4 | **none** | — | | roc-lang/basic-cli | 6 | ci + release | yes (1 ruleset) | | roc-lang/basic-webserver | 7 | ci + release | yes (1 ruleset) | And `roc-lang/release-package` already exists — composite actions for `validate-release`, `prepare-bundles`, `test-bundle`, `publish-release`. That's the leverage point. The answer is to extend it, not to build something new. ## What I'd recommend, in priority order ### 1. Fetch vendored runtimes by pinned hash; don't commit them The 113 files in template-go are 100% derivable from a Zig tarball whose SHA-256 the repo *already pins and verifies in CI*. Committing the outputs, then building 450 lines of reproduce-and-canonicalize machinery to prove they equal what Zig emits, is solving a problem created by the commit. Replace it with `scripts/fetch_runtime.py`: pinned version + pinned tarball hash -> extract -> done. A runtime update becomes a **two-line text diff** — a version string and a hash — which a non-expert can genuinely review. No manifest, no canonicalization, no reproduction job, no attestation apparatus, and no self-certifying-checksum hole because there's nothing to self-certify. Honest tradeoff: dev builds and the bundle step gain a network dependency, and a fresh clone no longer works offline. Cache it in CI. The bundle still *ships* the binaries — this changes what humans review, not what users get. ### 2. Make the release provably come from CI, from a tag For an inexperienced maintainer this is the most likely real hole, far more likely than a crafted malicious PR: they run `scripts/bundle.py` locally and upload the result, which makes their laptop the trust root for everyone's binaries. `basic-cli`'s `release.yml` already gets this right and is the model to standardize — `permissions: {}` at the top with job-level grants, and an explicit check that the release was dispatched from the default branch. Add attestation of the published bundle, generated in that same job. Then "did this come from reviewed source?" is a mechanical question with a mechanical answer. Roc's design helps here: bundle URLs are content-addressed, so an app that pinned a platform can't have it swapped underneath. The threat is a *new* bad release, not retroactive tampering — which is exactly what a CI-only release pipeline closes. ### 3. Default-deny binary changes in PRs For whatever binaries remain, make the invisible thing loud. `git diff --numstat` prints `-` `-` for binary files, so the whole check is dependency-free: ```yaml - name: No unreviewable binary changes run: | git fetch --depth=1 origin "$GITHUB_BASE_REF" changed=$(git diff --numstat "origin/$GITHUB_BASE_REF...HEAD" | awk '$1=="-" && $2=="-" {print $3}') [ -z "$changed" ] || { echo "PR modifies binary files:"; echo "$changed"; echo "Requires the 'vendored-binary-update' label."; exit 1; } ``` Gate the bypass on a maintainer-applied label. This is the control that maps directly to "overlooked in a big PR," and its value is that it requires *zero* security expertise to understand: binary changed -> CI red -> someone must say yes on purpose. Compare with a checksum file, which fails quietly by passing. ### 4. Lint the workflows mechanically The catastrophic mistakes non-experts actually make are `pull_request_target` combined with checking out the PR head (full repo compromise, copied from blog posts to make PR comments work), secrets exposed to PR-triggered workflows, and unpinned third-party actions. **zizmor** catches all three statically; **actionlint** catches the rest. Ten lines in CI, no expertise required, and it would have flagged template-go's mutable `actions/*@v4` tags. ### 5. Ship 1-4 as a reusable workflow, not as a guide This is the part that determines whether any of it reaches people. Add to `roc-lang/release-package`: ```yaml jobs: guard: uses: roc-lang/release-package/.github/workflows/platform-guard.yml@v1 ``` Five lines gets a platform author binary-diff denial, workflow linting, runtime-pin verification, and least-privilege defaults. When the Roc org improves a check, every platform picks it up on the next tag bump. Documentation decays and goes unread; a referenced workflow does not. ### 6. Repo settings need a separate answer, because templates can't carry them CODEOWNERS travels in a fork; branch protection and rulesets do not. `basic-cli` and `basic-webserver` have rulesets; none of the templates do, which means every fork starts unprotected. - **Inside `roc-lang/`**: one org-level ruleset applied to all platform repos. Done once, covers everything, cannot be forgotten per-repo. - **Outside** (like the odin template): a `platform-guard` check that reads the repo's own public API — `branches/main.protected`, `rulesets` — and warns in CI when protection is missing. OpenSSF Scorecard's action does roughly this if you'd rather not build it. ## The meta-point Prefer **few, loud, simple invariants** over elaborate verification. The Go template's provenance system is genuinely impressive engineering, and its sophistication is precisely what hid a trivial hole: the PR-time check validates a manifest that the PR itself can edit. A maintainer with limited security experience will not build that system correctly, will not maintain it, and — worse — may copy it and inherit a false sense of coverage from machinery they can't audit. "No binaries in git, releases only from CI, binary diffs fail the build" is weaker in theory and much stronger in practice, because non-experts can actually keep it true. Want me to draft the `platform-guard` reusable workflow and the `fetch_runtime.py` replacement against the Go template, as a concrete proposal to take to Luke?