这篇论文用Claude Opus 3从NICE文档里自动提取结构化信息，F1超90%，速度比人工快30倍，做医疗评估的朋友可以看看。
该论文提出一个基于schema的框架，用生成式AI从非结构化文本中提取层级化、嵌套式信息，并自动与金标准做语义评估。在卫生技术评估机构NICE的文档上，该框架用Claude Opus 3成功提取了14个属性中的12个，F1分数超过90%。提取单个文档的时间比人类领域专家快约30倍。研究还验证了该框架在不同生成式AI模型间的通用性，以及在不同HTA组织和语言间的可迁移性。
Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI
We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of $>$90\% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.