Amazon paper finds tax and legal benchmarks misgrade correct AI answers
Benchmark scores on rule-following tasks may reflect grading bugs rather than model weakness, so leaderboard gaps deserve scrutiny before drawing conclusions. Teams evaluating models on tax, legal or compliance tasks should read scorer code and manually inspect a sample of failed answers before trusting low numbers. The paper's audit method, tracing references to source rules, offers a reusable checklist for anyone relying on public benchmarks.
