Model hosting is a pool of SGLang instances, one per model, with a small router in front. That is the whole architecture. This post is about why we picked SGLang, how we run it, and how we swap a model's weights without ever restarting the instance.
We evaluated the usual suspects: vLLM, TensorRT-LLM, and the simpler options that wrap llama.cpp. SGLang won on three points that matter for a multi-tenant service:
One instance per model, pinned to one GPU or one NVLink pair, with the version pinned per deployment. We benchmark a release on a spare box before any production upgrade. A serving engine is the last thing you want to upgrade on a whim, and SGLang releases move fast, so this discipline is most of our uptime.
The router in front maps model ids in a request to the right
instance. It also enforces the spending caps, which is why a deployment can
only ever cost what you set. The instance itself never sees an API key.
The annoying part of serving is model updates. A customer fine-tunes their model and wants the new weights live without the endpoint dropping. We use SGLang's weight update endpoint for exactly this: the deployment tooling points the running instance at a new checkpoint, the instance loads it and swaps the pointer when it is ready, and requests that arrive mid-swap wait a moment. It has been quiet for a year now, which is exactly what we want from an update path.
Our deploy tooling loads checkpoints directly from a path on the shared storage, and for older customer checkpoints we keep the legacy format support on, because breaking uploads to save ourselves ten minutes is not a trade we want to make.
We run a pinned, lightly patched build in production, and the patches that are generic enough go upstream. Most of what we send is bug reports with a minimal reproduction case; the maintainers have been quick to answer, which is worth more to us than any amount of marketing. If you run SGLang too and hit something weird, file it and ping us, we usually know whether it is new.