双智能体 LLM 流水线实现可扩展的对话式考试系统
Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline
口试能逼学生讲清推理过程，但一个老师考不了全班。这篇论文用两个分工的智能体做对话式考试，一个提问一个暗中按 rubric 打分，还专门防提示注入骗答案。做教育产品或评估工具的人可以看看架构设计。
arXiv 论文 2611.17220 提出 scalable conversational assessment，用 LLM 模拟口试，对学生进行基于文本的交互式论证能力评估。系统采用双智能体架构：interviewer 直接与学生对话但只知道下一个问题，hidden evaluator 掌握整体评估目标、按 rubric 追踪进度并指示何时追问或进入下一题，以此降低答案泄露风险。
评估先用不同能力等级和对抗策略的 LLM 人格学生模拟测试，包括答案提取者、提示注入者、自信编造者，并用虚构的考核主题防止预训练知识泄漏。系统随后在一门小型 AI literacy 选修课中用于形成性和终结性评估。
通过人工与事后 AI 审查转录文本，团队发现追问生成和“只给学生本人贡献的观点计分”仍需改进。论文解决的是口试难以规模化的问题：既保留追问和论证检验，又无需人工考官一对一参与。
Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline
Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral exams scale poorly. In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument. We use a two-agent architecture to separate concerns and reduce the likelihood that answers are leaked to the student: the interviewer interacts directly with the student, but only receives guidance about the next question to ask the student; the hidden evaluator knows the goals of the whole activity, tracks the student's progress using a rubric, and guides the interviewer on when to follow-up on the current question or proceed to the next question. The system was first tested using a collection of simulated students with LLM personas of a particular ability level or adversarial tactic (e.g., answer-extractors, prompt-injectors, confident fabricators). A novel element of this evaluation was that it required the use of a fictional assessment topic to prevent personas that shouldn't have information from using knowledge present in the LLM's pre-training. The system was also deployed in a small, elective AI literacy course in both formative and summative contexts. Through manual and post-facto AI review of the transcripts we identified opportunities to improve the follow-up question generation and ensure that grading only credited ideas that the student (and not the interviewer) contributed.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM, 双智能体, 教育评估, AI literacy.
Source attribution: curated from arXiv cs.AI (@Geoffrey Challen, Craig Zilles).