Moving VINlist to Fedora CoreOS
The containers had no operating system left in them. The server under them still had a conventional one: Debian, set up package by package. On October 7 VINlist moved to a new server running Fedora CoreOS, where the operating system is one read-only image that updates itself as a whole.
Why CoreOS
- SELinux, enforcing. Every container gets its own security label. Even a container that broke out of its own limits couldn't read another container's files, the database or the host's. The old server had no such layer.
- A system that can't be quietly changed. The operating system is a read-only, integrity-checked image (composefs). There is no package manager to install tools with, which makes it much harder for an intruder to stay.
- Updates that undo themselves. Each update installs a complete new system image, in a weekly window (Zincati). If it doesn't boot, the server returns to the previous one on its own.
- Very little to maintain. The old server had 417 Debian packages. The new one is the base image plus three small additions.
Building a server from nothing, for real
The whole setup is code in the repository: a CoreOS first-boot configuration (Ignition), one provisioning script and one switch-over script. I rehearsed it first in a virtual machine on my PC. Then the new server was built from zero and its data restored only from our encrypted off-site backups, the same way it would be after a disaster. Every row matched. It was the first time the disaster runbook ran end to end for real.
The switch took 61 seconds. Visitors saw the maintenance page for about 35 of them. The script checks every step and rolls back on any failure: same build on both servers, the new one ready but holding, the old one stops writing, the data moves, all 23 tables match row for row, the new one starts, the old one leaves.
What the rehearsals caught
None of these reached visitors, and each would have cost an evening to find on a live server:
- Every container start waited 90 seconds. CoreOS doesn't report "network online" unless something asks for it, and rootless Podman waits for it. One line in the provisioning fixed a deploy that took 11 minutes.
- SELinux blocked the app from its own database. The database lives in a data folder, and the app's container reaches it through a shortcut (a symbolic link). Podman's
:zoption should label a folder so containers may use it, but it labeled the shortcut instead of the folder behind it, and SELinux kept the app out. Provisioning now labels the real folder. - Restored files kept the wrong label. Moving a file keeps its SELinux label, so the restore now relabels everything it moves.
- The service user's ID was taken. CoreOS ships a built-in user with ID 1000, the one our backups expect. Provisioning removes it.
- Identical databases looked different. A newer sqlite3 rejects double-quoted strings, and the default sort order differs between distributions. The data check now uses the C locale and plain SQL.
- A healthy server looked broken.
grep -qunderpipefailturned a match into a failure, and busybox's DNS lookup gets no answer from the newer container DNS. Both checks now test what actually matters.
What it gave us
Both servers ran the same build, minutes apart, with the same CPU model and size.
| Debian | CoreOS | |
|---|---|---|
| Server time per page | 13–40% faster on every page | |
| Listing views per second, sustained | 175 | 217 |
| CPU time per request | 5.3 ms | 4.3 ms |
| App memory at peak load | 579 MB | 419 MB |
| VIN photo scan, hard photo | 1,373 ms | 1,126 ms |
| Database commit | 0.92 ms | 0.85 ms |
I can't fully separate the newer kernel from a possibly quieter physical host; both get some credit. SELinux itself added no measurable cost. Through Cloudflare, visitors see the same page times as before, because most of a visit is the trip across the network.
What it doesn't do
The containers still share one kernel. SELinux narrows what an escape could reach; only separate virtual machines would remove the shared kernel.
Doing less
Three changes in two weeks, each one about doing less: fewer dependencies, nothing inside the containers, and an operating system I don't have to tend. That was the point of the keynote for me, and it's how VINlist will keep being built.
All of this comes from a die-hard JavaScript developer. For years I used JavaScript for the full stack, everywhere, and gladly, because one language meant shared modules, shared tools and shared conventions. That reason is fading. Agents can now write the code in tools that give more simplicity, at the cost of more sophisticated code, so I see little point in sticking with higher-abstraction languages. Rust is where VINlist is today. We'll see what other bare-metal options come along to replace it.
Shipping this article took about 100 seconds. The next one took 23… or 11.
