Pinned to a release that already had the fix

SHA-pinning a shared reusable GitHub Actions workflow buys you exactly one thing: the workflow file that runs today is the workflow file you reviewed, not whatever is on the maintainer’s default branch this morning. It does not buy you a working workflow. A pin freezes the bad version just as reliably as the good one, and nothing in the mechanism tells a consumer that a newer release exists, so a fleet of repositories can sit pinned to a known-broken release indefinitely, each one looking, from the inside, like an unrelated local failure.

The pin also has a narrower boundary than it looks like it has. It locks the workflow file’s own text, not everything that file reaches out to at run time. I have watched a reusable codeql workflow reference a sibling config file in its own repository with config-file: rknightion/.github/codeql/codeql-config.yml@main, which resolves live on every run regardless of which release the caller is pinned to. Deleting that file broke sixteen repositories pinned to the same release SHA, all at once, each with nothing to show for it but a bare Not Found inside a Load language configuration step. The pin looked intact and the diff was empty, because the failure came from something the pinned file fetches at run time, which a SHA never touches.

This piece is about the opposite failure mode: not a pin that lied about being frozen, but a pin that was frozen correctly, onto a release that turned out to need one more fix. The centrepiece is a real fleet-wide outage in a shared container-publish workflow, the one-line repair that ended it in every affected repository, and the diagnostic habit that finds this class of problem before you waste an evening blaming the wrong commit.

What actually broke

rknightion/.github carries a container-publish reusable workflow that every container-shipping repository in the fleet calls to build, scan and publish its image. Release v1.18.0 built the image with docker/build-push-action using outputs: type=oci,dest=${{ runner.temp }}/image-${{ env.PLATFORM_PAIR }}.tar and then handed that same path to Trivy as the scan input. That output produces an OCI tar: an archive whose index.json and oci-layout live inside the tar entries rather than in a real directory on disk. Trivy can read a Docker archive tar, which carries a top-level manifest.json, and it can read an OCI layout when it is an actual directory. It cannot read an OCI layout packed into a tar. The scan step failed to open the artefact at all, in every repository that called the workflow, on every publish.

The fix, released as v1.18.1 barely an hour after v1.18.0, was a one-line change to the build step’s outputs value: type=oci,dest=${{ runner.temp }}/image-${{ env.PLATFORM_PAIR }},tar=false. Dropping the .tar suffix and adding tar=false tells docker/build-push-action to write a real OCI directory instead of packing one into a tar, which is exactly the shape Trivy’s OCI reader expects. The same commit renamed the build step from “Build OCI archive” to “Build OCI layout” and threaded the changed path through to the oras cp --from-oci-layout call further down the job, since that also needed a directory rather than a tar. I pulled this diff directly against the two tags in the reusable’s own repository; it is exactly as small as it sounds, and it is the entire fix.

None of that helped the repositories that were still pinned to v1.18.0. Renovate, the dependency bot that opens the pin-bump pull requests across this fleet, had not yet caught up in eleven of them by the time the bug was found, so all eleven kept building an artefact their scan step could not open, on every push. The repair, once identified, was a one-line pin bump in each caller’s own workflow file: change the trailing # v1.18.0 comment and the commit SHA before it to the v1.18.1 tag’s SHA. No cross-repository write and no change to the reusable itself. The entire cost of the outage was the time it took to work out that this was the fix, not the time it took to apply it.

I own both ends of this one. I shipped v1.18.0, I found the bug, and I shipped v1.18.1 an hour later, so I never needed the mechanism to tell me a fix existed. The eleven stuck callers did, and so does anyone consuming a workflow they do not own. From inside a repository that calls a shared workflow, a platform team’s reusable behaves exactly like a third party’s: you find out it is broken when your own build breaks, and nothing in the mechanism says a later release already fixed it. Owning the reusable changed how fast I found the fix, not how the callers would have learned there was one.

The method

Finding that fix quickly came down to six habits, none of them specific to this bug.

Check whether a later release already fixes it, before diagnosing anything. A shared workflow that starts failing with no local diff is not evidence of a local cause. gh release list -R rknightion/.github shows every tag and its date in one call. Diffing the workflow file between the pinned tag and the latest one, git diff v1.18.0 v1.18.1 -- .github/workflows/container-publish.yml in a checkout of the reusable’s own repository, either shows you the fix directly or rules it out in under a minute. Both are cheap. Skipping this step and going straight to instrumenting the caller’s own job is the expensive path, because there is nothing to find there.

When a shared workflow breaks with no local diff, look at the reusable’s own repository for commits in the window, not the consuming repository’s history. The consuming repository’s git log is the only diff a stuck caller can see, so it is exactly where you should not look first. git log <last-known-good-tag>..<current-tag> -- .github/workflows/container-publish.yml against the reusable’s own history, compared to the timestamp of the last green run in the caller, tells you whether anything changed in the thing that actually runs.

Find the caller that is already ahead of the rest, and read it as the control. In this fleet, at least one repository had already been bumped to v1.18.1 or later by the time the others were still failing. Reading its most recent publish run and confirming that Trivy initialised and scanned normally, rather than failing to open the artefact, is enough to prove the fix works without touching anything in the broken repositories. A control that already exists is stronger evidence than a fix you have not yet applied anywhere.

A caller still red after the pin bump is not necessarily the same bug. This is the trap that costs the most time if you miss it, because the instinct is to reopen the diagnosis. One repository in this fleet, grafana-cloud-org-insights, was already sitting on the newest release of the reusable, v1.21.0, when its container publish started failing again, weeks after the tar bug was long fixed. Trivy was not failing to open the artefact this time. It initialised cleanly, detected the base image correctly, scanned the packages and wrote its SARIF, then failed the job because it found eighteen real HIGH and CRITICAL findings, every one of them reporting no fixed version upstream, spread across perl-base, util-linux, libsqlite3-0 and a handful of other base packages, on a python:3.14-slim base whose pinned digest already matched the current upstream image. That is the gate doing its job on packages nobody upstream has patched yet, not the scanner regressing. The fix there was a reviewed, per-CVE ignore file with an expiry date on every entry, passed through the reusable’s own trivy-ignore-file input, so a package that does get a fix upstream starts failing the gate again rather than staying silently suppressed. Reading which step failed, and why, is the only way to tell these two situations apart. “Still red” is a symptom, not a diagnosis.

Prove a fix by re-running the failed run at its unchanged SHA, not by pushing something new. gh run rerun <run-id> --failed re-executes the same commit against whatever the caller now resolves, which for a pinned reusable means the newly bumped pin and nothing else. Same commit, same caller, only the external dependency changed. A fresh push changes two things at once and proves neither.

Sweep for every caller before assuming it is one repository’s problem. gh api -X GET search/code -f q='container-publish.yml user:rknightion' (and the equivalent scoped to org:m7kni) lists every workflow file that references the reusable, fleet-wide, in one call. A shared-workflow bug is a fleet incident with a per-repository repair, and treating the first failure you see as an isolated one means finding the rest of them the slow way, one broken publish at a time.

The rest of the family: a green run that means nothing

The tar bug at least failed loudly. Several related traps in the same fleet fail by staying green, or by producing no error at all, which is worse.

A called workflow’s job-level permissions are capped by the calling job’s permissions, and that check runs at startup, before any if: condition on the job is evaluated. A permission needed only by a job that will be skipped on this run is still mandatory: the workflow-level permission block has to grant it regardless, because GitHub validates the whole call before it decides what to skip. Removing a permission because “that job doesn’t run any more” produces a startup_failure with zero jobs listed and no log at all; gh run view --log-failed on a run like that returns only log not found, because nothing ever started. There is no error message pointing at the missing permission, because nothing ran far enough to produce one.

workflow_run triggers match on the triggering workflow’s name: field, not its filename. Rename a workflow’s name: without updating every workflow_run listener’s workflows: array and the listener does not error. It stops firing, silently, and a green history for the listener is not evidence it is still connected to anything; there is simply nothing left to trigger it. The same trigger needs an explicit github.event.workflow_run.conclusion == 'success' gate, because it fires for failed and cancelled runs too, and the default github.sha inside a workflow_run handler is the default branch’s current HEAD, not the commit that was actually tested, which races ahead under rapid pushes; the correct reference is github.event.workflow_run.head_sha.

And a Helm chart validation step running kubeconform -ignore-missing-schemas passes while checking nothing at all for any custom resource kind the public schema catalogue does not carry. It silently skips every kind with no matching schema and reports success for the ones it never looked at, so a chart’s CRDs go through unchecked while the job stays green throughout. The fix is not the flag; it is pointing kubeconform at an actual schema source, such as a datreeio/CRDs-catalog URL, and leaving the ignore flag off, so a kind with no schema fails loudly instead of passing by omission.

What all three share with the tar bug’s aftermath is the shape, not the cause: a green run, a clean exit code, or total silence is never proof that the thing you care about happened. The exit code proves the process ran to completion. It says nothing about whether it did the job you meant it to do.

What dependency-bot lag does and does not fix

Renovate did eventually bring every caller in this fleet back onto the newest release; as of writing, every repository that calls container-publish is pinned to v1.21.0. That is not the same as saying the lag is solved. Watching one repository’s own merged pull request history shows Renovate opening a pin-bump pull request for the reusable and then updating that same open pull request in place as further releases land, rather than opening a new one for each release. That is efficient when it works, because whoever merges it lands on the newest available version in one step rather than working through several incremental bumps. It also means the exposure window is not “days until the fix ships”; it is “days until someone merges whatever pull request happens to be open,” and during that whole window the repository sits on whichever version was last actually merged, which can be considerably older than the one the bot has already queued up.

Waiting does not fix that. A bot opening the right pull request is not the same as the pull request being merged, and nothing about the mechanism forces urgency onto a fix that happens to matter more than an ordinary dependency bump. The sweep for every caller, run by hand rather than waited for, is what actually closes the exposure window on a fix that is known to matter: list every repository calling the reusable, check what each one is pinned to, and bump the ones still behind, rather than trusting that the automation will get there before the next thing breaks. The one-line fix in each caller is cheap enough that there is rarely a reason to wait for the bot to notice on its own schedule once you already know which release you need.

None of this argues against SHA-pinning a reusable workflow. It argues against treating the pin as the end of the diligence rather than the start of a different kind: it will not drift under you, but it will not tell you when it should.