739 封真实邮件实测：逻辑回归在邮件分类上跑赢生成式 LLM
Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox
有人拿自己 739 封真实邮件做实验，发现简单逻辑回归分类比 LLM 准，但放到实际动作里反而更常漏掉要回的邮件，结论很反直觉。
论文介绍自托管个人邮箱智能体 Foyer 及一套测量协议，含分层抽样、冻结快照与线程分组切分。作者对 739 封真实意大利语/英语邮件做人工标注，在九个系统上做对比。结果显示生产环境的关键词规则在邮件类型字段只有 28% 准确率，甚至低于 58% 的多数类基线；仅用约 200 条标注，TF-IDF + 逻辑回归在每个字段都拿到最高准确率（type 0.79、urgency 0.74、needs-reply 0.83、is-noise 0.90）。但准确率不等于产品指标：组合成动作后，最准确的系统有 18% 的比例忽略需要处理的邮件，而最好的 LLM 只有 4%；对噪声概率设置成本阈值后可把该比例减半并成为最便宜的方案。
Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox
Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself. We present Foyer, a self-hosted personal inbox agent in daily use, and a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human labels in the agent's review loop) that yields ground truth before any training. On 739 real Italian/English emails labelled by the owner (231 unbiased, 500 adversarially sampled), we compare nine systems under thread-grouped cross-validation: the production keyword rules, a TF-IDF + logistic-regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B/31B). Four results. The production rules score 28% on message type against a 58% majority class: the field shown most prominently carried no information. With ~200 labels, logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicts the two rarest classes and is statistically tied with Gemma 4 on type; few-shot prompting does not close the gap. The sender's relationship (client, supplier, service) is recovered from other emails of the same contact with 98.6% accuracy where the contact has been seen, and is not inferable from a single message by any model: it belongs in the contact graph, not the prompt. Finally, accuracy is not the product metric: composed into actions, the most accurate system most often ignores emails needing action (18% vs 4% for the best LLM); a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system.