CRAFT能帮你找到模型具体弱在哪里，不是笼统说不行，而是通过评分标准拆解能力，还能自动生成训练数据修bug。金融和法律测试里比随机采样效果好很多。
CRAFT是一种新方法，将基于评分标准的评估数据集转化为模型特定弱能力诊断。它从每个提示-评分标准对中提取能力描述，聚类成层次能力树，在4个开源模型、金融和法律两个领域及13个保留基准上评估。在金融领域，CRAFT对所有4个模型的平均分超过基线；在法律领域，对4个模型中的3个表现最强。结果表明，基于评分标准诊断比基于提示或类别能更精确地定位模型弱能力，并生成更有效的微调数据。
CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data
Evaluations should do more than measure a models current performance. They should tell us what to fix for the next model iteration and provide a way to generate targeted post training data. Most evaluation pipelines identify weak examples, topics, or categories, but they leave the underlying capability failure implicit: they say where a model fails, not why. We introduce CRAFT, a method that converts any rubric based evaluation dataset into a model specific diagnosis of weak capabilities. CRAFT treats each grading criterion as a capability probe: it extracts a capability description from every prompt rubric pair, clusters these descriptions into a hierarchical capability tree, scores the target model at every node, and selects low performing nodes dynamically across tree levels, at the granularity where each failure is clearest. The selected weak capabilities then direct the generation of targeted supervised finetuning data. Holding the data generation, finetuning, and evaluation setup fixed, we compare CRAFT against prompt level EvalTree clustering and untargeted random generation on four open source models, two professional domains (finance and legal), and 13 held out benchmarks disjoint from the diagnostic data. CRAFT achieves the strongest finance domain average for all four models under repeated temperature decoding; on legal domain, it is strongest for three of four models and remains within the decoding variance bands of the best baseline on the fourth. Diagnosing weaknesses at the level of rubric criteria, rather than prompts or categories, thus yields both a sharper picture of what a model cannot do and measurably better models after finetuning on that diagnosis.