Perplexity Finds CTF Tasks Poorly Measure Agent Security Skills
The non-obvious point is that even security-specialised models can underperform on real red-teaming work, so vendor claims of security fine-tuning deserve scrutiny. Teams evaluating agents for security-sensitive deployments in finance, infrastructure or SaaS should design internal task suites instead of relying on CTF leaderboard scores. The finding applies regardless of region, since evaluation methodology, not model origin, is the bottleneck.
