研究显示小型语言模型通过嵌入几何分析，在发票分类上超越大模型，仅需少量样本就能达到高准确率。
研究分析了SBERT和DeBERTa预训练模型的嵌入几何结构，发现财务语料库的句子嵌入空间由与供应商身份强相关的局部各向同性簇组成。在单个GPU上微调的SBERT在发票分类任务上达到0.96的准确率，优于零样本LLM和供应商身份基线。对于这一重要泛化问题，SBERT仅需约100个客户特定发票即可达到0.9的F1分数，表明内部SLM实施方案具有良好前景。
Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry
Categorising invoices into the correct General Ledger (GL) code underpins financial reporting and tax compliance. This is a skilled accounting judgement rather than a routine task: the correct category depends subtly on the nature of the purchasing business, the vendor and the invoice text. Whilst AI is increasingly being adopted across industries to automate tasks, including invoice categorisation, implementations built on in-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability. We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa). The sentence-embedding space of this financial corpus is globally anisotropic but composed of locally isotropic clusters, extending prior token-level findings to sentence embeddings in a financial setting, and these clusters are strongly correlated with the vendor identity. SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline, increasing performance for smaller, challenging categories and new clients. For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising. Combining these results with geometric analysis shows that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.