Shipping VINlist in 11 seconds
A VINlist release used to take about 100 seconds from my command to the live site. Now it takes 23, or 11 without a code change. Nothing got less careful on the way: every check still runs, and the build is exactly as optimized as before. The release just stopped waiting in line. And the people using the site now see far fewer "new version" prompts.
Why 100 seconds was too long
I ship small changes often: 52 times in the day before this article went out, 224 times in the week and 1,011 times in the 30 days before it. Every release is one push to GitHub, so these counts come straight from the repository's push record. At 100 seconds each, that month alone was 28 hours of staring at a terminal, and most of it wasn't work. It was one step waiting for another that had nothing to do with it.
| Before this article | Releases | At 100 s each | At 23 s each |
|---|---|---|---|
| The last day | 52 | 1 h 27 min | 20 min |
| The last week | 224 | 6.2 h | 1.4 h |
| The last 30 days | 1,011 | 28 h | 6.5 h |
Where one release's 100 seconds went:
| Step, one after another | Time |
|---|---|
| Lint, format, clippy and the dependency audit | ~6 s |
| Tests | 22 s |
| Release build of the server image | 28 s |
| Release build of the maintenance tools | 7 s |
| Upload and switch over | 10–15 s |
| Page check on the live site | ~35 s |
What changed
- Build while checking. The release build now starts together with the checks instead of after them. Nothing leaves my PC until every check passes, and a failed check stops the build.
- Tests side by side: 22 s to 3 s. Every release runs 1,283 tests: 1,029 for the Rust server and tools, and 254 for the JavaScript in the pages. They live in 15 separate test programs, and each one already used every core, but they ran one after another. Now they all start at once. The libraries we depend on are also compiled with optimizations in test builds; our own code isn't, so it still compiles fast. The slowest test went from 5.2 s to 1.8 s, an image-processing test from 4.2 s to 0.6 s. Release builds are untouched.
- The checks themselves side by side: 10 s to 5 s. Lint, the format check, clippy and the tests all start immediately. Clippy and the tests each get their own build folder, so they don't queue for the same lock.
- The page check after the release, not during it. After every release, a headless browser loads 360 pages as 9 kinds of visitor and fails on any script error. That takes about 35 seconds, and the site is already live by then, so it now runs after my command returns. If it fails, my phone gets an alert with the one command that undoes the release.
- Don't send what didn't change. The commit id used to be built into the image, so every release produced a new 17 MB image even when the code was the same. Now the commit is passed in when the app starts, and unchanged code gives an identical image, which stays home. The same goes for the 58 MB of tools in every upload, which rarely change: they're sent only when they do. A release with no code change used to upload about 37 MB; now it's about 2.
- Stop throwing away the build cache. A cleanup after every release deleted old image layers, and those were the build cache, so the next build started cold. Keeping a day of them saves up to 15 seconds.
- Small waits. One connection to production is reused for the whole release (four calls went from 8.1 s to 3.6 s). The push to GitHub happens while the image is still building. The download of the vulnerability database starts first thing.
How much
Each step got its own timer, and every release after a change showed where the time went next.
| Release | Time |
|---|---|
| Before | ~100 s |
| Build while checking, tests side by side, page check after | 41 s |
| One connection, only changed tools, push while building | 31 s |
| Build cache kept | 29 s |
| Checks side by side, commit no longer built into the image | 16 s |
| The same, with an unchanged image that stays home | 11 s |
11 seconds is a release where the server's code didn't change. When it does, it's about 23 seconds: the server is compiled from scratch with full optimization, which alone takes 10, and the new image has to be built and uploaded. The checks finish well before that, so the compile is what sets the pace.
What I didn't cut
Incremental release builds would save most of those 10 seconds, at the cost of slightly less optimized code. Skipping the dependency audit would save a few more. I kept both: the speed came from removing waiting, not from checking less.
Undo in seconds
Production now keeps the previous version next to the live one: its images, catalog files and tools. The database stays as it is; its changes only ever add, which an older version ignores. One command switches back in a few seconds, with nothing rebuilt or uploaded, and running it again returns to the newer one. If a new version doesn't come up at all, the release switches back by itself.
Fewer "new version" prompts
A page that stays open while a new version goes out offers a reload, so nobody keeps running stale code. Until now every release asked every open tab. At 30 to 50 releases a day, someone halfway through an article or comparing two cars was interrupted again and again, for changes to pages they would never see.
Now each page carries a fingerprint of what it runs: its templates and its own scripts. When a new version is out, the open page asks for the new fingerprint of that one page, and offers a reload only when it changed. A change to the statistics page now prompts only people on the statistics page. Style tweaks and shared translation text are left out on purpose: the page keeps working with the old ones, and the next page opened has the new ones.
The funny part
When I asked whether a release could get under a minute, and then under 10 seconds, the coding agent I use estimated about 40 seconds for the first. Ten, it said, was out of the question as long as every release still compiles and runs every check; 20 might be possible. We ended at 11 for a release without a code change and 23 with one, and picked up the per-page prompts and an undo in seconds along the way.
From my first question to the 11-second release took about an hour. Every release printed where its time went, and the next one fixed the slowest part. That's the power of quick iteration: the estimate was made before anyone had measured, and an hour of measuring beat it.
Fast is nice. Right matters more.
