Bisect in CI: Reproducing the Commit Is the Hard Part

"Add git bisect run to the pipeline" reads like a one-line change. The flag is real and it works, but the line assumes something your CI has probably never had to guarantee: that any historical commit can be checked out, provisioned, built and tested from the commit alone — with no help from today's working environment.

At your desk that assumption is invisible. One toolchain version, dependencies already installed, a database that happens to be in the right shape, a warm cache. In CI, the moment the loop checks out a commit from eight months ago, three things become true at once: the toolchain in the image has moved past what that commit was written for, the lockfile may no longer resolve, and the schema in that commit predates the migrations your job runs before the tests. None of that is a Git problem, and all of it decides whether the bisect can answer.

This is the second of two posts about git bisect. The first one is about the predicate — whether the question you ask is deterministic and specific enough to be worth asking. This one is about everything that must be true before the question can be asked at all. They meet at the same place: a bisect run is a loop that spends builds, and a loop that spends builds is only as good as its cheapest reproducible step.

What has to be true before the first step

Stripped of the Git, an automated bisect is a `for` loop over commits in which every iteration does the same five things. Each one is a place where a pipeline that was written for "run the tests on HEAD" quietly stops being sufficient.

Step, once per candidate commit	What it has to be	What you get when it is not
Materialise the tree	A checkout of exactly that commit, or none at all	A predicate that only reads the object database does not need a working tree at all — --no-checkout keeps a BISECT_HEAD ref instead (bare repositories imply it).
Provision	Toolchain and dependencies that resolve for that commit	An old commit cannot install its dependencies, so the step fails. Whether that reads as "bad" or as "untestable" is your decision, and it is the decision that matters most.
Build	Compiles at that commit	A build failure is not a regression signal. Treating it as one attributes the bug to whatever the build broke on, which is usually the commit that renamed a file.
Assert	Deterministic, and specific to the failure you are chasing	An intermittent or broad assertion gives a confident answer to a question you did not ask.
Clean up	Nothing leaks into the next iteration	A cache keyed on anything coarser than the commit will hand the next step a build from a different tree, and the loop will test code it is not standing on.

Stateful services are the step everyone forgets. Walking backwards through commits means walking backwards through schema: the checkout at an old commit expects a database shaped for that commit, while the migrations your job runs bring today's schema, not that one. Down-migrations are the textbook answer and almost nobody has them working fifty commits back, so the rule that holds is the blunt one — stand the database up from scratch, or from a snapshot pinned alongside the commit, on every iteration, and let a revision whose schema cannot be built return 125 rather than a verdict.

The manual already tells you where to put the scripts: outside the repository, so the bisect, the build and the tests do not interfere with each other's view of the tree. That advice is easy to skip locally and expensive to skip in CI, where the same directory is a mount point, a cache target and a working tree.

The one rule that makes the whole thing trustworthy

Everything above collapses into a two-line predicate shape:

The distinction in the first line is what separates a bisect that tells you something from one that tells you nothing. A commit whose dependencies cannot be installed is not evidence about your bug, and reporting it as bad invites the loop to converge on infrastructure and call it a finding. Returning 125 makes it a skip: the search routes around the commit instead of blaming it. (The exit-code contract itself — including the two codes git refuses to interpret — is the subject of the other post.)

In a local session you can get away with a naive predicate, because you know why a build failed. In CI nobody is watching, so the discipline stops being optional: the pipeline is the only thing that can tell "this commit is broken" from "this commit is old".

What it costs, and why CI feels it more

Locally, a step is a test run. In CI, a step is a provisioning plus a build, and those are the expensive parts. The loop is logarithmic, so the number of steps grows slowly — but you pay for every one of them in full, and skipped commits do not buy the range back.

The step counts below are measured, from the same predicate over ranges of different sizes. The two cost columns are the same counts multiplied by a per-step budget — a modest two minutes, and a realistic ten.

Candidate commits	Steps	At 2 min per step	At 10 min per step
8	3	6 min	30 min
32	5	10 min	50 min
64	6	12 min	1 h
256	8	16 min	1 h 20

That is the good case, where every commit can be tested. Take away ten commits — a dependency that no longer resolves — and the same 32-commit range executed the predicate thirteen times instead of five, because a skipped commit consumes a step without shrinking the interval, and the run ended without a single answer. That is 2.6× the budget for less information than the smaller run produced.

Two consequences follow, and they are pipeline decisions rather than Git ones. Bound the range. Anchor the bad end at the failing build and the good end at a known-good release tag rather than at the beginning of time, and narrow it further with a pathspec if you know where the problem can live — you are paying per step, so every commit you can exclude up front is money. Give the job its own timeout. A bisect job that inherits the test job's timeout will be killed halfway through an expensive run and report a timeout, which tells you nothing about the regression.

The exit status your job must read

There are two exit statuses in play and they are easy to confuse: the one your predicate returns, which git interprets, and the one git bisect run returns to your job. The second is the contract your pipeline branches on, and it is not the tidy enum most jobs assume. It comes from is_bisect_success(res) ? 0 : -res, so any path that already carries an error code passes that code through negated.

git bisect run exits	What happened	What the job must do
0	The loop finished and named a culprit (or a merge base).	Publish the commit, and reset the session. Read the log to distinguish a named culprit from the ambiguous endgame.
2	Inconclusive: There are only 'skip'ped commits left to test — the interval was entirely untestable.	Report no answer, and reset the session.
1	The run failed inside git: the bogus-exit-code path, or another internal error.	Report a job error, not a regression, and reset the session.
128, 56, …	The abort path. The status echoes your predicate: a predicate exiting 128 makes bisect run exit 128, 200 becomes 56, and 255 becomes 1.	Report a job error. Do not enumerate these values — they are derived from the predicate's own exit code.

The last row is the one worth reading twice, because it breaks the natural assumption that a failed job reports 1. Measured on 2.56.0: a predicate exiting 128 makes the run exit 128, one exiting 200 makes it exit 56, and only 255 — which is what exit(-1) becomes — collapses to 1. A job that branches strictly on the set {0, 1, 2} will misclassify almost every real abort.

So branch on zero for success and two for inconclusive, and treat everything else as a job error without trying to enumerate the failure values. If the job needs to tell the cases apart, read the message text — is the first 'bad' commit, There are only 'skip'ped commits left to test, bisect run failed, bogus exit code — which is stable and human-readable. And the misreading to avoid above all: treating "the run finished" as "we have a culprit". A job that only checks for a non-zero status passes the inconclusive case through as a success with no commit attached, and someone spends an afternoon reading a log that contains a list of eleven candidates and no answer.

Where CI is worse than your laptop

Flakiness compounds. A predicate that is right ninety percent of the time already produced two different answers in two local runs. CI adds contention, cold caches, timeouts and a shared network to the same assertion, so the probability that a full run is internally consistent is lower than it was locally — and the answer will look exactly as confident.

The loop can run your command out of order. When the first run returns 126 or 127, git does not blame the commit; it re-runs the same command against a known-good revision to find out whether the failure belongs to the script. That is a defence, and it is also an extra step on a revision you did not ask about. If your predicate has side effects — writing a report, poking a shared database — they now happen once, out of order, on the good anchor. Keep the predicate read-only.

Caching is a trap. A cache keyed on the branch, the workflow, or "the latest successful build" will happily serve a build produced from a different tree, and the loop will test code that is not in the commit it is standing on. Key on the commit SHA, or disable the cache for the bisect job and accept the provisioning cost — the table above already tells you what that costs.

Nobody is watching. Locally you notice that the build failed for the wrong reason. In CI the job reports a number. Budget for the run to produce a report a human can read: the culprit, the step count, which commits were skipped, and the range you searched.

Blame the merge, not the internals

In a pipeline that gates on a merge queue, the interesting unit is not a commit inside a branch — it is the merge that brought the branch in. That is the thing you can revert, review, and talk about with the author. git bisect start --first-parent keeps the search on the mainline and reports the merge commit as the introduction point, ignoring everything inside the merged branch. It also removes an entire class of false positives, because the intermediate commits of a feature branch are exactly where the non-buildable and half-finished states live.

Make the job clean up after itself

Every path above leaves a bisect session open — including the successful one, which finishes with HEAD detached at the culprit and the session state still on disk. In a throwaway container it eventually disappears, but while the job runs the state is there and the next step inherits it. Git 2.56 added --reset-when-found, which closes the session as soon as the culprit is found and is accepted by both start and run; --reset-when-found=found leaves the working tree on the culprit, which is what you want if the next step collects a diff or an artifact from it.

That flag only covers the success path, so the job still needs the equivalent of a finally block: reset the session on every exit, and treat "session still open" as a job error rather than as state to be discovered by the next step.

When not to automate this

• Your build is long. Read the table again at ten minutes a step: half a day is a plausible cost for one answer, and the team will stop waiting long before it arrives. If the answer is that valuable, run it once on demand, not on every failing build.
• The environment is not hermetic. This is the real precondition, and the reason the technique works in some codebases and not at all in others. An arbitrary commit needs its dependencies resolvable as they were then: a hermetic build system such as Nix or Bazel, or a base image pinned per revision inside the repository, or a lockfile plus a mirror that still serves the old versions. Without one of those, CI fails on something mundane and fatal — a package removed from the registry, a dead apt URL, an old version no longer published. The loop then spends its budget skipping and ends inconclusive: the measured 13-step run, for every range you try.
• The suite is flaky. Fix determinism before automating the loop; a flaky predicate scales its wrong answers with the number of steps.
• The regression is not in the repository. Configuration, data, a dependency that moved, a change on the platform — the loop will attribute it to the nearest commit and be wrong with full confidence.
• Nobody will read the output. A bisect job that nobody triages is an expensive way to leave a comment on a build.

And a note on scope, because it is the honest version of "just add the flag": if the only thing standing between you and an automated bisect is that no one can build a commit from six months ago, then you have discovered something more valuable than the regression — you have discovered that your builds are not reproducible, and the bisect was merely the first thing that needed them to be.

Summary

Running bisect in a pipeline is not a harder version of running it locally; it is a different problem. Locally you supply a question and the environment is whatever you have. In CI you must supply the environment per commit — provision, build, and assert reproducibly — and then read the loop's own exit status honestly: zero for a culprit, two for inconclusive, one for a broken run, and anything above those echoing your predicate's own code, with a session left open behind all of them. Bound the range, because you pay per step and skipped commits do not refund anything; key your cache by commit; keep the predicate read-only and specific; and let the job clean up after itself. Do that and the automation does what the flag promises. Skip it and you have built a machine that produces confident answers about infrastructure.