Shopify Details PyTorch and vLLM Continual Learning Loop at PyTorchCon 2026
The pipeline shows a practical way to turn production failures into labelled training data without manual annotation, with a small tuned model beating a frontier model on a narrow task. Teams running LLM features in production can copy the judge-critic-replay pattern to improve quality while cutting inference cost via prompt compression. The keynote livestream is available to watch online.
At PyTorchCon 2026, Shopify engineer Cody Mazza-Anthony presented the company's continual learning loop built on PyTorch and vLLM. Using the Buyer Profile feature as an example, a calibrated judge flags low-scoring outputs in production, critics diagnose each failure, and a replay with a critic hint is rescored by the judge. Accepted repairs expand the training data, letting a small specialist model outperform a frontier model on this task. The talk also covered optimizing inference through prompt compression.
Now: @Shopify’s continual learning loop with PyTorch and @vLLM_project takes the stage at #PyTorchCon 2026 with Cody Mazza-Anthony @cmazzaanthony.
Using Buyer Profile as an example, Cody walks through how a calibrated judge flags low-scoring outputs in production. Critics diagnose each failure, and a replay with a critic hint is scored again by the judge. Accepted repairs expand and improve the training data.
Cody shows how a small specialist model can outperform a frontier model on this task, then turns to optimizing inference through prompt compression.
Watch the keynote livestream: https://t.co/6TtaEkw0x7
