Real-Time Transformer Prediction at Autodesk

In the summer of 2022, halfway through my master’s in AI at Northeastern, I spent three months in San Francisco as a Machine Learning Engineer Intern at Autodesk. I worked on next-click prediction, training a sequence transformer and deploying it into a live product event stream that processed more than a million user interactions a day.

The problem

In desktop and cloud design tools, every command, menu click, and tool switch emits an event. A sequence transformer works well for predicting a user’s next command, provided the prediction returns while the user is still in that step of the workflow.

Because the event pipeline was already running in production, we had two hard constraints:

1. Do not break upstream producers. The inference service had to read the existing event schema and message bus without requiring changes from client teams.
2. Keep latency low during traffic spikes. At 1M+ interactions per day with bursty daytime usage, the service had to keep p99 latency down and scale horizontally when traffic climbed.

What I did

I built the model and the serving service together rather than waiting until the end of the internship to figure out deployment.

Most of the engineering work went into online/offline feature parity (making sure the live stream windowed and tokenized command sequences the exact same way as the offline training scripts) and profiling inference so the transformer stayed within our latency budget. We also set up horizontal autoscaling, health checks, and dashboards for throughput, consumer lag, and prediction distributions.

Streaming inference fails differently from batch jobs. If an offline batch job runs 15% slower after a commit, you wait an extra ten minutes for the output table. On a live event bus, a 15% slowdown during peak hours causes consumer lag to grow, memory usage to climb, and predictions to arrive too late for the UI to use them.

What I took away

Our latency budget and feature-windowing rules ended up dictating how deep the transformer could be and how long a command history we could pass at inference time. Sizing the model around the serving constraints in week one saved us from training a checkpoint that was too slow to ship in week ten.

For the other side of inference, scoring 10B+ rows in scheduled batch jobs, see the billion-row batch inference case study.