Billion-Row Batch Inference for Scientific Models

Problem

Scientific model evaluation required running TensorFlow predictions over data volumes too large for a notebook-oriented workflow. The system needed predictable throughput, repeatable execution, and outputs that downstream teams could use without manual intervention.

Scale and constraints

Scale

• More than one billion input rows per production run.
• End-to-end processing completed in under three hours.
• Multiple TensorFlow models participated in the prediction workflow.

Constraints

• Keep model execution and the surrounding data pipeline efficient at large scale.
• Produce repeatable results for scientific evaluation rather than a one-off benchmark.
• Operate inside the reliability and cost boundaries of a shared production ML platform.

Sudhanva's ownership

Sudhanva built the batch prediction pipelines and owned the production ML infrastructure they ran on.

• End-to-end batch prediction workflow design and implementation.
• TensorFlow execution and input-pipeline performance work.
• Operational integration with the shared Kubernetes, Ray, and Flyte platform.

Architecture at a safe level

The public architecture is intentionally described through system responsibilities rather than internal service names or proprietary topology. The relevant boundaries were workload execution, orchestration, data movement, observability, and delivery.

• Treated preprocessing, model execution, and output materialization as one measured pipeline rather than optimizing model inference in isolation.
• Used distributed platform capabilities to scale data movement and model execution while keeping runs reproducible.
• Instrumented the production path so runtime and resource behavior could be observed and improved.

Important engineering decisions and tradeoffs

• Throughput improvements had to preserve scientific correctness and repeatability.
• Resource allocation balanced faster completion against shared-cluster cost and capacity.

Results

• Processed more than one billion rows in under three hours.
• Accelerated drug-candidate evaluation and experimental iteration.

Reliability and operations

• Production runs were observable through the platform monitoring and alerting stack.
• Repeatable orchestration reduced reliance on manual notebook execution.

Transferable lessons

• For large batch inference, data loading and preprocessing deserve the same attention as model compute.
• End-to-end runtime is the useful product metric; isolated accelerator utilization is only one input.