Production ML Platform on Kubernetes

Problem

Research and engineering teams needed a dependable shared path for running models, tracking experiments, and moving versioned workloads into production without rebuilding infrastructure for each project.

Scale and constraints

Scale

• Shared production infrastructure supported cross-functional model users.
• The platform hosted large batch prediction and multi-GPU workflows.
• Resource optimization reduced annual cloud spend by more than $50,000.

Constraints

• Support different workload shapes without turning every request into platform-engineering work.
• Keep production operations observable and recoverable.
• Control cloud cost while allowing workloads to scale on demand.

Sudhanva's ownership

Sudhanva owned the end-to-end ML infrastructure and built internal applications on top of it.

• Autoscaling Kubernetes infrastructure and workload orchestration.
• Ray and Flyte platform integration.
• Prometheus, Grafana, alerting, experiment tracking, and model versioning.
• Documentation and training for cross-functional users.

Architecture at a safe level

The public architecture is intentionally described through system responsibilities rather than internal service names or proprietary topology. The relevant boundaries were workload execution, orchestration, data movement, observability, and delivery.

• Built a shared platform layer around orchestration, execution, observability, and lifecycle concerns.
• Exposed common workflows through internal applications so users could run models on demand.
• Used measured resource behavior to tune capacity and eliminate avoidable cloud spend.

Important engineering decisions and tradeoffs

• Shared abstractions had to remain flexible enough for research while preserving production guardrails.
• Autoscaling reduced idle cost but required careful observability and workload-aware limits.

Results

• Reduced cloud infrastructure costs by more than $50,000 annually.
• Reduced time to insight for internal model users from days to minutes.
• Enabled on-demand model runs with experiment tracking and model versioning.

Reliability and operations

• Prometheus metrics, Grafana dashboards, and alerting made failures and capacity constraints visible.
• Documented, versioned workflows replaced ad hoc execution paths.

Transferable lessons

• A production ML platform succeeds when it removes repeated operational work for model users.
• Cost management and observability are platform features, not cleanup tasks.