Sakana AI releases JFinBench, a Japanese financial benchmark of 321 questions
The finding that knowledge of Japan-specific laws and accounting standards, not math ability, drives the gap is the key takeaway for anyone localizing AI into Japanese regulated industries. Teams deploying LLMs for tax, accounting, or compliance in Japan can use the 321-question dataset to test candidates before purchase. The human-in-the-loop pipeline also offers a reusable recipe for refreshing domain benchmarks as exam content changes annually.