When a customer rents an H100, they want to be SSHed in before the coffee cools. Ours took 40 minutes and I had stopped promising anything specific. This is the story of where those 40 minutes went and how we got it down to 9.
We timed it properly before changing anything. The breakdown was:
| Step | Time |
|---|---|
| Disk image allocation | 11 min |
| OS boot and network bring-up | 6 min |
| GPU driver load and firmware check | 8 min |
| Pre-pull of customer containers | 9 min |
| Instance agent registration | 6 min |
Nothing was catastrophically broken. It was just a pile of sequential work, each piece small enough that nobody had bothered to question it.
The disk image allocation was doing a full copy of a 200 GB image on first boot. We moved to a copy-on-write base image shared across all instances of the same class, which turned 11 minutes into about 40 seconds. It means two instances share the base layer until either of them writes to it, and for our workloads that rarely happens before the customer takes over.
The driver load was slow because we loaded every driver for every card on every boot. Now the image knows which card it is on and loads one driver. The firmware check moved to the provisioning step, not the boot step, so hardware health is verified before the instance exists, not after.
Container pre-pull is the one we almost did not touch. Pulling a 40 GB container on a busy machine takes minutes and the network is the network. But the instance agent now reports "ready to work" as soon as SSH is up and the container pull runs in the background, instead of the instance being blocked on it. Customers who use their own image see the real time either way, and the "ready" signal is truthful now.
For a customer renting one H100 for a weekend, 31 minutes is nothing. For a team running 20 instances, it is a morning. And for the support queue it is the difference between "it is booting, wait" and "you are in." Most people do not remember what they did while waiting for a machine, they just remember that they waited. Removing the wait removed a whole class of support tickets we used to answer with the same sentence.
We did not make SSH faster, because it was already fast. We did not touch the network stack beyond ordering. We did not add a progress bar anywhere, though we were tempted, because a progress bar that lies is worse than none. The number we measure now is time to a working shell, and nothing else.