Netflix分享了优化LLM评估工具的详细过程，值得学习。该工具通过多阶段评估和持续监控，显著提高了推荐效果，值得一读。
Netflix通过LLM评估工具在数百万个节目推荐解释中运行，每周服务于数百万会员。该工具分为四个阶段：出生、训练、部署和监控，通过持续的人机协作确保推荐质量。一项五周的A/B测试显示，该工具将观看转向了之前未观看的内容，并提高了浏览到播放的成功率，没有出现与质量相关的问题。
This is one of the most useful writeups I have seen on keeping an LLM judge effective in production....
This is one of the most useful writeups I have seen on keeping an LLM judge effective in production. (bookmark it) Netflix runs judges over hundreds of thousands of show-level recommendation explanations per week, served to millions of members on mobile. They describe the judge as a lifecycle with four phases rather than an artifact that you only validate once. > Birth defines multiple evaluation criteria and builds curated benchmarks with human labels and rationales. > Training refines the judge's rubric through Reasoning-Aligned Rubric Tuning, using a meta-judge over reasoning output as the learning signal. > Deployment puts one judge in two roles, quality gating and reflective generation. > Monitoring runs continuous human-in-the-loop alignment that detects drift and triggers re-tuning behind a review gate. A five-week A/B test over tens of millions of members shifted viewing toward previously unwatched content and increased successful browse-to-play sessions against a no-explanation control, with no quality-related takedowns. Paper: arxiv.org/abs/2608.18300 Track more trending AI papers in our academy: academy.dair.ai 💬 0 🔄 0 ❤️ 5 👀 775 📊 2 ⚡