Billion-Row Batch Inference for Scientific Models

Built and ran a distributed TensorFlow prediction pipeline at Montai that scores 10B+ candidate rows end to end in under three hours.

Rows
10B+
End to end
Under 3 hours
How the billion-row batch inference pipeline runs Over ten billion input rows stream through distributed preprocessing, a fleet of TensorFlow models, and materialized output tables on Kubernetes, Ray, and Flyte, finishing end to end in under three hours. Input rows (10B+ per run) flows to Preprocessing (Input pipeline) [10B+ rows]; Preprocessing (Input pipeline) flows to TensorFlow models (Batch prediction) [batches]; Shared ML platform (K8s · Ray · Flyte) drives Preprocessing (Input pipeline) [scales]; Shared ML platform (K8s · Ray · Flyte) drives TensorFlow models (Batch prediction) [runs]; TensorFlow models (Batch prediction) flows to Predictions (Materialized); TensorFlow models (Batch prediction) drives Prometheus (Grafana · alerting); Predictions (Materialized) flows to Scientific teams (Candidate review) [results]. Owned by Sudhanva: Preprocessing, Shared ML platform, TensorFlow models and Predictions. Outcomes: End-to-end run: Under 3 hours. 10B+ PER RUN Input rows INPUT PIPELINE Preprocessing K8S · RAY · FLYTE Shared ML platform BATCH PREDICTION TensorFlow models MATERIALIZED Predictions GRAFANA · ALERTING Prometheus CANDIDATE REVIEW Scientific teams 10B+ rows batches scales runs results END-TO-END RUN Under 3 hours Owned by Sudhanva Data Control
How the billion-row batch inference pipeline runs 10B+ PER RUN Input rows INPUT PIPELINE Preprocessing K8S · RAY · FLYTE Shared ML platform BATCH PREDICTION TensorFlow models MATERIALIZED Predictions GRAFANA · ALERTING Prometheus CANDIDATE REVIEW Scientific teams 10B+ rows batches runs results END-TO-END RUN Under 3 hours Owned by Sudhanva Data Control
Over ten billion input rows stream through distributed preprocessing, a fleet of TensorFlow models, and materialized output tables on Kubernetes, Ray, and Flyte, finishing end to end in under three hours.

Confidentiality note. Internal service names, proprietary scientific details, and employer-specific implementations are kept general here. Every scale figure, responsibility, technology, and outcome below comes from already-published facts in Sudhanva’s résumé and public profile.

Problem

Scoring scientific models across more than ten billion candidate rows was too large for single-node scripts or interactive notebooks. Scientists needed a daily batch job with reproducible outputs and materialized tables they could query directly.

Scale and constraints

Scale

  • 10B+ input rows scored per production run.
  • End-to-end execution completed in under three hours.
  • Multi-model TensorFlow scoring across shared GPU capacity.

Constraints

  • Keep data ingestion, preprocessing, and GPU execution balanced so GPUs do not sit idle waiting on I/O.
  • Produce identical outputs across runs for the same model and input snapshot.
  • Run within the cost and capacity limits of a shared Kubernetes cluster.

Sudhanva’s ownership

Sudhanva designed the batch prediction pipeline and owned the production ML infrastructure it ran on.

  • Architecture and implementation of the distributed batch prediction workflow.
  • Throughput tuning across TensorFlow model execution and the upstream input pipeline.
  • Production scheduling and monitoring on Kubernetes, Ray, and Flyte.

Architecture at a safe level

Internal service names and proprietary scientific details are omitted below, while the data flow, orchestration, and operational setup reflect the production system.

  • Profiled data ingestion, preprocessing, GPU inference, and output writes together so storage I/O kept pace with GPU throughput.
  • Ran distributed preprocessing and multi-GPU inference across Ray and Flyte on Kubernetes with pinned container and model versions.
  • Added Prometheus metrics and Grafana dashboards for stage throughput, queue depth, and GPU utilization.

Engineering decisions and tradeoffs

  • Kept deterministic execution settings enabled so higher batch throughput never changed numerical outputs.
  • Capped per-run GPU concurrency so the job finished in under three hours without starving other workloads on the cluster.

Results

  • Processed more than ten billion rows in under three hours.
  • Cut drug-candidate screening turnaround from weeks to hours.

Reliability and operations

  • Prometheus alerts and Grafana dashboards tracked run progress, stage latency, and task failures.
  • Versioned Flyte workflows replaced manual notebook runs.

Transferable lessons

  • At ten billion rows, storage I/O and serialization bottleneck the run long before GPU compute does.
  • For downstream scientists, the metric that matters is wall-clock time from job start to a queryable output table.

Technologies

  • TensorFlow
  • Kubernetes
  • Ray
  • Flyte
  • Python
  • Prometheus
  • Grafana

Explore more

From the blog

Related posts

Machine Learning Best Practices For Your Company

An onboarding checklist for ML and data engineering teams on Git hygiene, secret handling, DVC, notebook discipline, and experiment tracking.

#machine-learning #devops
Next case study · $500K+ annual cloud-cost reduction Kubernetes ML Platform