Nebius 出了个 Data Lab，能帮你从日志里挖出模型翻车的真实案例，直接做成数据集来微调和评估，比啥基准都管用。
作者指出现有基准往往在部署后失效，因为真实用户会使用你未测试过的表述。解决方案是分析推理日志和追踪，提取真实提示、响应、拒绝和格式错误。Nebius在Token Factory内推出Data Lab工作区，可将失败案例转化为评估和微调数据集。推荐循环：读日志→找失败→建数据集→评估→微调→部署→重复。
You could make a living by helping companies fix t…
You could make a living by helping companies fix their evals.
You wouldn't need anything else:
1. Show me how you are evaluating your product 2. This is how you can make it better
I'm always hearing the same story:
• Someone picks a benchmark early on • Everyone becomes obsessed with optimizing against it
The minute you deploy your application, that benchmark becomes useless.
People using your application don't send the prompts you tested against.
They phrase things you didn't think of. They paste in formats you never saw. They ask in languages you didn't evaluate.
The solution is to start using your inference logs and traces. Look at the following:
• Prompts • Responses • Where the model refused to answer • Where the model got the format wrong
Logs and traces aren't for compliance only; they are the highest-signal dataset you have access to.
Nebius shipped a workspace for this called Data Lab. This lives inside Token Factory.
You can use it to find any failure cases and turn them into an evaluation and fine-tuning dataset.
This is a better training set than anything you could buy or collect: it will contain real use cases where your model failed.
Here is the loop you should aim toward:
• Read your logs • Find the failures • Build a dataset from them • Use this for evaluation • Fine-tune your model • Deploy • Repeat
Here is a link to a blog post with more information: https://t.co/qVCikY7B9m
Thanks to the Nebius team for partnering with me on this post.