训练、检索还是混合?安大略住宅租赁法引用的四臂对比

Train, Retrieve, or Both? A Four-Arm Head-to-Head for Correct Statutory Citation on the Ontario Residential Tenancies Act

精选理由

这篇论文用Qwen2.5-7B做了个四路对比,发现微调加检索混合方案在法条引用上精确匹配0.481还零幻觉,比纯微调或纯检索都强,而且用轻量bge-small就够用。

AI 摘要

论文使用Qwen2.5-7B-Instruct对比了零样本、仅SFT、仅RAG和SFT+RAG混合四种方法在安大略住宅租赁法条文引用上的效果。混合方法以0.481精确匹配(节+条)取得最高分,且将幻觉降至零。SFT提升了高召回候选集中条款选择的鲁棒性,而仅用bge-small嵌入就超越了更大专用检索模型。扩大训练集未带来提升,0.70目标尚未达到。

原文 · arXiv cs.LG

Train, Retrieve, or Both? A Four-Arm Head-to-Head for Correct Statutory Citation on the Ontario Residential Tenancies Act

Self-represented tenants, landlords, and help-desk staff need to be pointed at the provision of law that actually governs a question, with a correct statutory citation. We study this task on the Ontario Residential Tenancies Act, 2006 (RTA) and its core regulation, asking the operator's question empirically: is fine-tuning enough, or is hybrid retrieval needed? We run a four-arm head-to-head on Qwen2.5-7B-Instruct (base zero-shot, LoRA SFT-only, RAG-only, and an SFT+RAG hybrid), scored on citation exact-match (section+subsection) over a small, human-verification-pending real eval set. The base model cannot cite the RTA and SFT-only mis-recalls sections; retrieval is essential and drives hallucination to zero by construction; and the SFT+RAG hybrid scores highest at 0.481 exact-match with zero hallucinated citations. Its edge comes from SFT making provision selection more robust to the higher-recall candidate sets that hurt zero-shot RAG. Notably, this cheap bge-small hybrid matches or beats a pipeline built on bigger, specialized retrieval models (a larger embedder and a cross-encoder reranker), and a larger/improved training set does not help either: strong statutory-citation performance here does not require specialized retrieval models or more data. The artifact zeroes hallucination and clears the lift-over-base bar but does not reach the aspirational 0.70 exact-match target. All results are on a small, human-verification-pending real eval set and are reported as preliminary.