Containers without an operating system
The same question from DHH's keynote, one layer down: why does a web server's container hold an entire Linux distribution? Ours held Debian because that's where the runtime came from. Once the server was a single Rust program, it didn't need one.
What was inside before
Counting only the base images the old containers were built on, before Bun, Tesseract and the app were added:
| Before | Today: web · proxy · photo reader | |
|---|---|---|
| Files in the image | 2,857 or more | 99 · 1 · 60 |
| Programs and libraries | 658 or more | 1 · 1 · 58 |
| Shells | 5 | 0 |
| Package managers | 2 | 0 |
| Setuid programs | 8 | 0 |
In megabytes, image by image. The photo VIN reader used to live inside the web app image and now has one of its own, so the fair comparison is the whole set:
| Container image | Bun, before | Rust, today |
|---|---|---|
| Web app | 303 MB: Bun on Debian 166 MB, Tesseract with its language data about 97 MB, the app and its packages about 40 MB | 16.7 MB: the server binary 14.8 MB, the site's own files 1.9 MB |
| Proxy | 166 MB: the same Bun on Debian, running a small script | 4.2 MB: one binary |
| Photo VIN reader | inside the web app image | 85 MB: Tesseract, the libraries it needs and its English language data |
| All images | 469 MB in two images (303 MB on disk, since they share the Bun base) | 106 MB in three images |
Someone who managed to run code in the old container landed in a full Debian system with a shell and a package manager. Today the web image is one program plus the site's static files. There is nothing to run and nothing to install.
How a Rust program runs with nothing around it
- Static linking. The server is built for musl, so it carries everything it needs, including SQLite compiled in from C.
- No certificate bundle. TLS uses rustls with the trusted root certificates compiled into the binary.
- No time zone files. I checked that nothing reads them rather than assuming it.
- Read-only, no privileges. The container runs as a numeric user with a read-only filesystem, no Linux capabilities and no writable scratch space beyond its data folder.
- The photo VIN reader moved out. Reading a VIN from a photo needs Tesseract, a large C library. It now runs in its own container on a network with no route to the internet, so the web image carries none of it.
System calls: allow what's used, refuse the rest
Containers normally get a generous default list of kernel system calls: 374 in Podman's default profile. I recorded what each of our programs actually calls and turned those recordings into allowlists. Everything else is refused before it reaches the kernel.
| Container | System calls allowed |
|---|---|
| Podman's default | 374 |
| Web app | 90 |
| Photo VIN reader | 78 |
| Proxy | 73 |
| Cloudflare tunnel | 71 |
The building blocks of most container escapes, such as ptrace, mount, unshare, keyctl, bpf, io_uring and userfaultfd, are refused in every one of them.
The price of going static
The first measurement on the live site showed something odd: a listing page answered 299 requests per second one at a time, but only 113 per second twenty at a time. Three locks were serializing the work. musl's memory allocator takes a global lock, so the server now uses mimalloc. SQLite's memory statistics take a global mutex on every allocation, so they're switched off at build time, with a test that fails if a build ever turns them back on. And the database schema check ran on every new connection, so it now runs once. After the fix the same page served 4,883 requests per second, twenty at a time. The cost is memory: about 300 MB in use instead of 166 MB.
How far can you go? No operating system at all
A container still shares the host's kernel. The logical next step is a unikernel: the program compiled together with a minimal kernel into one bootable image, with no other processes at all. I tried it. The same server binary and data ran under Nanos in a QEMU virtual machine on my own PC, compared with the same binary as a normal Linux process on the same CPU core.
| Page, one request at a time | Unikernel | Linux process |
|---|---|---|
| Home page | 0.56 ms | 0.36 ms |
| Specs catalog (850 KB) | 0.73 ms | 0.14 ms |
| Static file | 0.07 ms | 0.03 ms |
| Decode a VIN | 2.76 ms | 3.97 ms |
It booted in about a tenth of a second and served every page correctly. VIN decoding, which reads a large database heavily, was even 1.4 times faster. Everything that sends a lot of data was slower, because each response crosses a virtual network card. It used about 50% more CPU per request and twice the memory. And with two CPUs it hung as soon as two large responses went out at once. Most rented cloud servers can't run virtual machines inside them either, so each part would need a server of its own.
So VINlist stays on containers with nothing inside them. If stronger isolation becomes worth paying for, it will come from dedicated hardware with a small virtual machine per component. Not from a unikernel, at least not until I've figured it out and it's worth it.
Which left one operating system still worth worrying about: the server's own. It ran Debian with 417 packages I had installed and set up one by one. After emptying every container, that felt like cleaning the whole house and leaving the garage full of junk. Part 3 clears out the garage: a system that can't be quietly changed, undoes its own bad updates, and moved to a new server in 61 seconds.
