Services / AI stack

Open source AI

AI platforms built from open source, end to end. Model serving, training pipelines and API gateways on your own GPUs, without a proprietary platform in the middle.

The open source AI stack has matured fast. Open-weight models, high-throughput inference engines and Kubernetes-native ML tooling now cover most of what organisations used to buy as a managed service — if the platform underneath is built properly.

That platform is infrastructure work: GPUs, scheduling, storage, networking and CI. It’s the same work we do for clouds, applied to AI.

The stack we work with

  • vLLM — high-throughput LLM inference with an OpenAI-compatible API, for serving open-weight models efficiently on your GPUs.
  • Triton Inference Server — one serving layer for many kinds of models, with vLLM, PyTorch and ONNX Runtime as backends, and NVIDIA’s TensorRT where you want it.
  • LiteLLM — one gateway in front of every model, local or external, with keys, quotas and usage tracking per team.
  • Kubeflow — pipelines, notebooks and training on Kubernetes, so experiments become repeatable workflows.
  • PyTorch — the framework most open models are trained and fine-tuned in, running on the platform we build.

How we approach it

We build AI platforms the same way we build clouds: declarative, tested in CI and close to upstream. GPU nodes are provisioned by the platform — on OpenStack, on Kubernetes, or on bare metal with Metal3 — and model serving is deployed and upgraded like any other workload.

When something in the stack doesn’t work for you, we fix it in the project rather than in a private fork.

Where we help

  • First platform — from a few GPU servers to a working, shared inference service your teams can use.
  • Growing platforms — GPU scheduling and sharing, multi-model serving, observability and cost per team.
  • Hard problems — throughput that doesn’t match the benchmarks, memory pressure on large models, drivers and kernels that disagree.

Have an infrastructure problem worth solving properly?

Tell us what you're running and where it hurts. We'll tell you honestly whether we can help.

hello@infraloop.cloud