The open source AI stack has matured fast. Open-weight models, high-throughput inference engines and Kubernetes-native ML tooling now cover most of what organisations used to buy as a managed service — if the platform underneath is built properly.
That platform is infrastructure work: GPUs, scheduling, storage, networking and CI. It’s the same work we do for clouds, applied to AI.
The stack we work with
- vLLM — high-throughput LLM inference with an OpenAI-compatible API, for serving open-weight models efficiently on your GPUs.
- Triton Inference Server — one serving layer for many kinds of models, with vLLM, PyTorch and ONNX Runtime as backends, and NVIDIA’s TensorRT where you want it.
- LiteLLM — one gateway in front of every model, local or external, with keys, quotas and usage tracking per team.
- Kubeflow — pipelines, notebooks and training on Kubernetes, so experiments become repeatable workflows.
- PyTorch — the framework most open models are trained and fine-tuned in, running on the platform we build.
How we approach it
We build AI platforms the same way we build clouds: declarative, tested in CI and close to upstream. GPU nodes are provisioned by the platform — on OpenStack, on Kubernetes, or on bare metal with Metal3 — and model serving is deployed and upgraded like any other workload.
When something in the stack doesn’t work for you, we fix it in the project rather than in a private fork.
Where we help
- First platform — from a few GPU servers to a working, shared inference service your teams can use.
- Growing platforms — GPU scheduling and sharing, multi-model serving, observability and cost per team.
- Hard problems — throughput that doesn’t match the benchmarks, memory pressure on large models, drivers and kernels that disagree.