云基础设施开发者终于有了一个贴近真实工作流的评估基准——SWE-InfraBench 测试的是增量修改而非从头写代码，做 IaC 或 DevOps 的团队值得关注，看看当前模型在 AWS CDK 上的真实表现。
SWE-InfraBench 是一个新基准，用于评估大语言模型在云基础设施即代码（IaC）任务上的表现。与现有基准不同，它聚焦于 AWS CDK 的增量代码修改，而非从头生成整个代码库。数据集来自数十个真实 IaC 代码库，要求模型根据自然语言指令修改现有代码，并通过测试用例验证。评估结果显示，当前最强模型 Sonnet 3.7 的成功率仅为 34%，而推理模型 DeepSeek R1 只有 24%，表明 LLM 在云基础设施代码领域仍有显著局限。该数据集已在 Kaggle 上公开。
SWE-InfraBench: Evaluating Language Models on Cloud Infrastructure Code
Building infrastructure-as-code (IaC) in cloud computing is a critical task, underpinning the reliability, scalability, and security of modern software systems. Despite the remarkable progress of large language models (LLMs) in software engineering -- demonstrated across many dedicated benchmarks -- their capabilities in developing IaC remain underexplored. Unlike existing IaC benchmarks that predominantly center on declarative paradigms such as Terraform and involve generating entire codebases from scratch, our benchmark reflects the incremental code edits common in enterprise development with imperative tools like the AWS CDK. We present SWE-InfraBench, a diverse evaluation dataset sourced from dozens of real-world IaC codebases that challenge LLMs to perform realistic code modifications in AWS CDK repositories. Each example requires models to implement changes to existing codebases based on natural language instructions, with success determined by passing provided test cases. These tasks demand sophisticated reasoning about cloud resource dependencies and implementation patterns beyond conventional code generation challenges. Our evaluation results reveal significant limitations in current LLMs showing that even state-of-the-art systems struggle with many tasks -- the best model, Sonnet 3.7, succeeds in only 34\% of cases, while specialized reasoning models like DeepSeek R1 achieve just 24% success. The SWE-InfraBench dataset is available at: https://www.kaggle.com/datasets/64e59070fd51c0278560b01eb5dc4f3c447d5268cdabe5a350d2969e4413fea5