Production ML Platform on Kubernetes

Built and ran Montai’s ML compute platform on Kubernetes, Ray, and Flyte, which cut annual cloud spend by over $500,000 and dropped turnaround time for researchers from days to minutes.

Annual cloud-cost reduction
$500K+
Time to insight
Days → minutes
How the production ML platform on Kubernetes serves model users Researchers trigger on-demand runs from internal apps; Flyte orchestrates autoscaling Ray workers on Kubernetes with built-in MLflow tracking and Prometheus/Grafana alerting. Model users (Cross-functional) flows to Internal apps (On-demand runs) [requests]; Internal apps (On-demand runs) flows to Flyte workflows (Orchestration) [workflows]; Flyte workflows (Orchestration) flows to Ray on Kubernetes (Autoscaling) [tasks]; Ray on Kubernetes (Autoscaling) flows to MLflow (Runs · versions) [runs]; Ray on Kubernetes (Autoscaling) drives Prometheus (Grafana · alerts) [metrics]. Owned by Sudhanva: Kubernetes ML platform. Outcomes: Annual cloud spend: Idle capacity to $500K+ saved; Time to insight: Days to Minutes. KUBERNETES ML PLATFORM CROSS-FUNCTIONAL Model users ON-DEMAND RUNS Internal apps ORCHESTRATION Flyte workflows AUTOSCALING Ray on Kubernetes RUNS · VERSIONS MLflow GRAFANA · ALERTS Prometheus requests workflows tasks runs metrics ANNUAL CLOUD SPEND Idle capacity $500K+ saved TIME TO INSIGHT Days Minutes Owned by Sudhanva Data Control
How the production ML platform on Kubernetes serves model users KUBERNETES ML PLATFORM CROSS-FUNCTIONAL Model users ON-DEMAND RUNS Internal apps ORCHESTRATION Flyte workflows AUTOSCALING Ray on Kubernetes RUNS · VERSIONS MLflow GRAFANA · ALERTS Prometheus requests workflows tasks runs metrics ANNUAL CLOUD SPEND Idle capacity $500K+ saved TIME TO INSIGHT Days Minutes Owned by Sudhanva Data Control
Researchers trigger on-demand runs from internal apps; Flyte orchestrates autoscaling Ray workers on Kubernetes with built-in MLflow tracking and Prometheus/Grafana alerting.

Confidentiality note. Internal service names, proprietary scientific details, and employer-specific implementations are kept general here. Every scale figure, responsibility, technology, and outcome below comes from already-published facts in Sudhanva’s résumé and public profile.

Problem

Researchers and ML engineers needed a way to run multi-GPU jobs, track experiments, and deploy versioned models without filing infrastructure tickets or spinning up separate clusters for each project.

Scale and constraints

Scale

  • Shared production and research compute across science and ML teams.
  • Ran both 10B+-row batch inference jobs and multi-GPU training workflows.
  • GPU-aware autoscaling and right-sizing cut annual cloud spend by more than $500,000.

Constraints

  • Support both interactive experiments and large batch jobs on the same cluster without custom infrastructure work per model.
  • Track cluster utilization, job health, and cloud spend per workload.
  • Scale GPU nodes up on demand and scale idle capacity down to zero.

Sudhanva’s ownership

Sudhanva owned the ML infrastructure and built the internal web tools researchers used to launch jobs.

  • Autoscaling Kubernetes cluster setup and GPU workload scheduling.
  • Ray and Flyte orchestration layer.
  • Prometheus, Grafana, alerting, MLflow experiment tracking, and model versioning.
  • Internal documentation, onboarding, and self-serve job submission tools.

Architecture at a safe level

Internal service names and proprietary scientific details are omitted below, while the data flow, orchestration, and operational setup reflect the production system.

  • Combined workflow orchestration, distributed GPU compute, monitoring, and the model registry on a single Kubernetes platform.
  • Built internal web apps on top of Flyte and Ray so scientists could launch versioned training and inference runs on demand.
  • Tuned node-pool autoscaling, bin-packing, and resource requests against actual GPU utilization to stop paying for idle nodes.

Engineering decisions and tradeoffs

  • Standardized container images and Flyte task templates so researchers could iterate quickly without breaking production dependencies.
  • Scaling idle GPU pools to zero saved cloud budget, though we had to tune warm pools and image caching so jobs still started quickly.

Results

  • Reduced cloud infrastructure costs by more than $500,000 annually.
  • Reduced time to insight for internal model users from days to minutes.
  • Gave researchers on-demand runs with automatic MLflow tracking and model versioning.

Reliability and operations

  • Prometheus metrics, Grafana dashboards, and alerts caught node pressure, OOMs, and stuck workflows early.
  • Versioned Flyte workflows replaced ad-hoc shell scripts and unversioned notebooks.

Transferable lessons

  • Scientists adopt an internal platform when launching a job on it is faster than starting a standalone EC2 instance.
  • Per-job cost tracking and idle node scale-down need to be wired in when the cluster goes up, before GPU usage grows.

Technologies

  • Kubernetes
  • Ray
  • Flyte
  • Prometheus
  • Grafana
  • MLflow
  • Terraform

Explore more

From the blog

Related posts

How I Consolidated Milvus Across AZs

Step-by-step runbook for migrating a multi-AZ Milvus deployment onto a single dedicated Kubernetes node without losing a single vector.

#aws #devops #kubernetes

Install JupyterHub on AWS Elastic Kubernetes Service (EKS)

Field notes and step-by-step runbook for deploying multi-user, GPU-enabled JupyterHub on AWS EKS using custom Docker images and Zero to JupyterHub.

#aws #kubernetes #machine-learning
Next case study · 2× release velocity Multi-GPU Delivery Platform