五款 7B-8B 开源模型土耳其语文档问答评测：本地 6GB 显卡跑出 49%-75% 准确率
Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints
有人在 6GB 显存的笔记本上，用土耳其语 109 页报告实测了五款 7B-8B 开源模型，还顺手证明花哨检索方案打不过 TF-IDF，做 RAG 的可以看看。
一篇 arXiv 论文评测了五款 7B-8B 开源模型在土耳其语领域文档问答上的表现。主基准基于 109 页工业研发报告构建 100 道验证题，再用 112 页公共部门报告和独立 100 题集复现，全部在 6GB 显存的 NVIDIA RTX 3050 笔记本 GPU 上以 4-bit 量化本地运行。端到端准确率在 49% 到 75% 之间，评估协议通过证据标注把检索失败与模型推理失败分开。七种词法、稠密和混合检索配置经 McNemar 检验后，均未显著超过字符 TF-IDF 基线。
Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints
Most Turkish-capable large language models (LLMs) are evaluated using general-purpose benchmarks rather than long, structurally complex domain documents. This paper evaluates five open-weight 7B-8B models for Turkish document question answering under a resource-constrained local deployment setting. The primary benchmark contains 100 systematically validated questions derived from a 109-page industrial R&D report, and the evaluation protocol is replicated using a second 112-page public-sector report and an independently constructed 100-question set. All models are evaluated locally on an NVIDIA RTX 3050 laptop GPU with 6 GB VRAM using controlled prompting, decoding, and 4-bit quantisation. The principal methodological contribution is an evidence-annotated evaluation protocol that separates retrieval failure from downstream model reasoning failure without requiring additional model calls. On the primary benchmark, end-to-end accuracy ranges from 49% to 75%. Seven lexical, dense, and hybrid retrieval configurations are additionally compared using 95% Wilson intervals and exact paired McNemar tests; none significantly outperforms the character TF-IDF baseline on either document. Evidence recall saturates differently across the two reports, showing that retrieval and effective context capacity can be binding constraints for some documents but not others. These results demonstrate that model selection, retrieval behaviour, and hardware limits must be evaluated separately when deploying open-weight LLMs for Turkish domain documents.
