We build on open source, and give some back

SGLang

Model hosting runs on SGLang. Each hosted model is one SGLang instance pinned to a GPU, with a small router in front of the pool. We picked it for the OpenAI-compatible surface and for the prefix cache, which saves us a lot of compute on repeated prompts.

We run a pinned version per deployment and benchmark releases before we upgrade, because a serving engine is the kind of thing you do not upgrade on a whim. When we hit a bug that looks real, we reproduce it and file it upstream with a minimal case. The patches we send back are usually small and boring, which is how we like them.

What we contribute

• Bug reports with reproduction cases, mostly around long-context serving and the weight loading path.
• Small patches for our own deployment, sent upstream when they are generic enough. The rest we keep in a private fork.
• Benchmark numbers when the maintainers ask, which happens more often than you would think.

Everything else we run

• PyTorch and CUDA for the training and fine-tuning images.
• containerd, Kubernetes, and our own boot image tooling for instances.
• ClickHouse for usage and billing numbers, Grafana for graphs.
• Postgres for the control plane, because boring is a feature.

If you run SGLang too and hit something odd, file it. We read the issue tracker and we usually answer.