论文精选

用机器学习将零售产品名映射到消费价格类别:规则+词袋管道与可靠性加权人工标注

Machine Learning for Coding Retail Product Names to Consumer-Price Categories: A Rule-plus-Bag-of-Words Pipeline with Reliability-Weighted Human-in-the-Loop Labeling

精选理由

做价格统计或零售数据处理的团队,会发现这个简单但有效的管道能快速解决产品分类痛点,建议直接参考其规则+词袋的轻量方案。

AI 摘要

本文研究如何将短、噪声、无标准编码的零售产品描述自动映射到消费分类(如UN COICOP),以支持消费者价格测量。作者提出一个管道:文本标准化、基于前缀树的规则预分类器、以及每个类别的二元确认模型。在人工标注环节,采用可靠性加权投票协议,并对比了Dawid-Skene方法。实验发现,在无泄漏控制下,词袋模型即可达到约0.99的F1分数,线性分类器与多层感知机表现相当,约67个标注样本就足够。研究还讨论了价格水平质量控制,为统计机构使用交易数据提供设计经验。

原文 · arXiv cs.LG

Machine Learning for Coding Retail Product Names to Consumer-Price Categories: A Rule-plus-Bag-of-Words Pipeline with Reliability-Weighted Human-in-the-Loop Labeling

Consumer-price measurement increasingly draws on alternative data sources -- scanner, web-scraped, and transaction/receipt data. A recurring obstacle is that product descriptions in such sources are short, noisy, and abbreviated, with no standard product code, so each item must first be mapped to a consumption classification (e.g., the UN COICOP scheme) before prices can be compared. This paper studies that mapping as a general, reproducible method. The pipeline is: (i) text normalization and tokenization of noisy item names; (ii) a prefix-tree (trie) rule-based pre-classifier driven by per-category key-phrases and stop-phrases; and (iii) a per-category binary confirmation model deciding whether an item belongs to a tentatively assigned category. For labels at scale we use a human-in-the-loop protocol in which annotators give a binary valid/reject judgment, aggregated by a dynamically updated reliability weight; the model joins the same rule, enabling continual fine-tuning. Our empirical finding is deflationary: in a controlled, leakage-free study (one category, real positives vs. hard negatives, five seeds), bag-of-words models essentially saturate the task (F1 about 0.99) -- a linear classifier matches a multilayer perceptron, explicit word-order (n-gram) features add nothing, and about 67 labeled examples already suffice. A Monte-Carlo study of the labeling protocol shows the reliability-weighted vote barely beats plain majority (its additive weights saturate) while Dawid-Skene recovers labels markedly better. We also discuss price-level quality control and design lessons for statistical offices considering transaction data. All figures are illustrative; no confidential data, code, or documentation is reproduced.