Shopify details continual learning loop with PyTorch at PyTorchCon 2026
The pipeline turns production failures into labelled training data automatically, which explains how a small specialist model can beat a frontier model on a narrow task. Teams running LLM features in production can adapt this judge-critic-replay pattern to their own evaluation loops. It pairs with the PyTorch and vLLM stacks many ASEAN engineering teams already deploy for serving.
Shopify engineer Cody Mazza-Anthony presented the company's continual learning loop built on PyTorch and vLLM at PyTorchCon 2026. Using the Buyer Profile feature as an example, a calibrated judge flags low-scoring outputs in production, critics diagnose each failure, and a replay with a critic hint is rescored by the judge. Accepted repairs then expand the training data. Mazza-Anthony showed a small specialist model outperforming a frontier model on this task, and also covered inference optimization through prompt compression.
Now: '@Shopify’s continual learning loop with PyTorch and @vLLM_project takes the stage at #PyTorchCon 2026 with Cody Mazza-Anthony @cmazzaanthony.
Using Buyer Profile as an example, Cody walks through how a calibrated judge flags low-scoring outputs in production. Critics diagnose each failure, and a replay with a critic hint is scored again by the judge. Accepted repairs expand and improve the training data.
Cody shows how a small specialist model can outperform a frontier model on this task, then turns to optimizing inference through prompt compression.
Watch the keynote livestream: https://t.co/6TtaEkw0x7
