I can’t count the times I dropped .terraform-version in a new Terraform repo. tfenv read it, fetched the right binary, and switched. When a new Terraform release broke a plan at the worst possible moment, tfenv use put the old one back and I went on with my day. That command has never once asked me to think hard thoughts.
gridctl v1.0.0-rc.3 shipped recently with three pull requests that borrow some of that magic for gridctl’s own upgrade command. Why did an idea from a bash version manager belong in a Go binary that orchestrates MCP servers and agent environments?
Patterns Outlive the Tools#
tfenv is a fork of an idea. It credits rbenv in its README, and rbenv’s idea has been copied into pyenv, nodenv, goenv, and a dozen others. Keep installed versions side by side. Make switching a pointer flip INSTEAD of a download. Let the project declare what it needs. Verify what you fetch. Many an ecosystem has rediscovered this pattern, usually right after someone loses a lunch break, afternoon, and dinner.
This has always been interesting to me. The pattern isn’t tied to Ruby or Terraform. It’s tied to a situation: a tool you upgrade often, a workflow that depends on it, and what the cost is when you get something wrong (let’s be honest, it happens to all of us). Agents and MCP are new, but a CLI that replaces itself in place is not, and there is no reason to learn the same lesson twice just because the technology changed underneath it.
Before this release, gridctl upgrade did one thing well. It fetched the latest release, verified the SHA256 against the published checksum file, and swapped the binary atomically. Clean, and completely one-directional. If the new release broke the stack you had running, recovery meant finding the old archive on GitHub and reinstalling it by hand, with working network, while whatever you were in the middle of sat there marinating like a turkey. Picture doing that during a demo.
Start at the Ground Floor#
It usually tends to be the things that are not so exciting that matter most. Terraform already had a little knob I love: required_version in a .tf file, and the binary simply won’t run when it doesn’t satisfy it. tfenv doesn’t do that check for you. It reads .terraform-version, and its min-required mode scans your .tf files for the first version it finds without evaluating the constraint. The enforcement belongs to the tool that owns the config, so that’s where it went in gridctl, on a stack and on a pack manifest:
| |
gridctl validate exits 1 when the running binary doesn’t satisfy it, and the error names the fix:
| |
Apply, serve, and hot reload check the ground floor on the merged file, so a parent reached through extends can set a floor the child inherits. A rejected reload leaves the loaded configuration running. pack add refuses a pack whose floor you don’t meet before it imports anything, and pack apply checks the recorded value again.
I want to be precise about older binaries. I expected a release that predates the key to reject it as an unknown field. rc.2 doesn’t. It unmarshals the YAML, drops the field it doesn’t recognize, and reports the stack as valid. I ran it while writing this post. Put requires: in a stack and run it on rc.2, and the floor is ignored. That’s fail open my friends. Strict unknown-key checking shipped in this same release, so upgrade to rc.3 before you add the key.
A dev build still syntax-checks the string and then skips enforcement, with one warning. Otherwise nobody working on gridctl itself could run a stack that pins a release.
Keep the Bytes You Replaced#
Keeping the bytes is the tfenv part. A standalone gridctl upgrade now keeps the binary it replaced and the binary it installed under ~/.gridctl/versions. Two entries by default: the active release and the previous one. --keep raises that for one run, and --keep 0 leaves the store untouched. --list shows what’s there, and the marks will look familiar to anyone who has run tfenv list:
Examples in this post run against a local release server and two builds of main, since rc.3 is the first release that retains anything. It’s the same seam the integration test uses. Upgrading from rc.2 to rc.3 leaves the store empty, because rc.2 didn’t know how to fill it; the first rollback target you get is rc.3 itself, on the upgrade after it.
--rollback restores the previous entry. --rollback <tag> restores a named one. Both are offline. Nothing is downloaded, and the stored SHA256 is checked before the installed binary is touched. A mismatch prints both digests, exits 1, and leaves your install alone. Nothing retained, or a tag that isn’t in the store, exits 2, which is a different answer than “something broke” and is meant to be scripted against.
The store also pays off going forward. A later gridctl upgrade --version <tag> uses a retained copy when the checksum matches and says so:
| |
A few honest limits. I truly believe in truth pills. A Homebrew install still prints the brew upgrade command and doesn’t roll back unless you pass --force, because two things managing one path is how you end up with both being broke. Windows refuses --rollback and skips retention. And a dev build is never retained, since a store full of binaries named dev tells you nothing.
Who Owns the State File#
tfenv swaps the binary and leaves the state file to Terraform. Terraform refuses a state format it doesn’t understand, which is how a newer snapshot can strand an older binary. That’s a clean division of labor, and tfenv gets to stay a pile of shell scripts because of it.
Here is where it got tricky with gridctl. It owns its state: daemon state files, tool and skill pin files, the project lock, and the import lock. A newer release may have stamped any of those in a way the older binary can’t read. Roll back blindly and the “fixed” binary refuses to start the stack you rolled back to save.
Every binary now advertises the five schema versions it knows through gridctl info --json. Upgrade records that advertisement on each retained entry: the replaced binary from its own constants, and a freshly downloaded one by running info --json on the new binary before it’s installed, with a 10 second timeout and an environment trimmed down to PATH, HOME, and GRIDCTL_HOME. When you ask for a rollback, the check runs before the confirmation prompt and compares the target’s advertisement against the files on disk:
A header, one indented line per problem, and a reminder that --force exists. Exit 2. Pass --force and the same lines print as warnings while the rollback continues. A retained entry from a release that didn’t advertise schemas yet gets a compatibility warning and proceeds, because there is nothing to compare against and refusing would strand anyone upgrading from an older release.
The other half of state is the daemons already running. Swapping the binary on disk doesn’t change the process that was started by the old one, and before this release you’d find out about the mismatch whenever something subtle blew up. Daemon state files now record binary_version, gridctl status gained a VERSION column plus a stderr warning when a daemon disagrees with the CLI, and gridctl doctor reports the same thing. Pass --restart on the upgrade or rollback that swaps the binary and it restarts those daemons from recorded state, through the supervisor for a service-owned daemon, by destroy plus apply for a plain one, and by destroy plus serve for a stackless one. Sheesh, that is a brainful (at least for me).
Here’s the whole loop. An upgrade that retains the old release, a status check that points at the daemon still on it, and a rollback that puts everything back:
Prove Before Trust Is a Must!#
Rollback is a feature you only exercise when something is already wrong, which is the worst time to discover it has a bug of its own. This is why you test things, right?
The store has unit tests for the round trip, pruning that keeps active and previous, a checksum mismatch that reports both digests, a cross-process lock, symlink refusal, and manifest tags that aren’t a single directory name. The command tests run the upgrade flow against an httptest release server, and that’s where a mismatch is shown to leave the installed binary alone. The integration test builds two race-instrumented binaries with different -ldflags versions, serves one through a local release server, and walks the whole thing: upgrade, a second install that hits the store and makes zero download requests according to the server log, an offline rollback with the server shut down, a daemon started on the old binary showing up as a mismatch in status, a --rollback --restart that replaces that daemon’s PID, and refusals for a hand-bumped project lock and a hand-bumped pin file.
If you build a self-updating tool, test the rollback against a server, even a fake one. “Offline” is a claim, so the test shuts the release server down before it rolls back. Output text saying nothing was downloaded is not the same as a request log with nothing in it.
Landing With a Balance#
None of this is novel. I’m not smart enough for that level of novel anyhow. The pattern came from rbenv by way of tfenv, and it has worked for a long time. My biggest task was determining which parts transfer and which don’t. Retained versions, a declared floor, and verified bytes all transfer oh so elegantly. The symlink flip doesn’t; gridctl still replaces the binary in place. “Leave the state to someone else” doesn’t either, so that part had to be thought through.
The upgrade command still verifies checksums rather than authenticated provenance. If you want origin verification before installing, the release verification guide covers installing the same verified bytes by hand, and I wrote about why that distinction matters in Checksums Don’t Prove Who Built It.
If you’re building anything that upgrades itself, look at what the version managers in a mature ecosystem settled on before reinventing any wheels. The only constant is change, and the brilliant folks who solved some of these problems a decade ago were kind enough to leave notes.
