Papers

MarineSparkLM-4B:基于 LoRA 适配的海洋工程轻量模型

MarineSparkLM-4B: A Lightweight Language Model for Marine and Offshore Engineering Built by Parameter-Efficient Adaptation

Why we picked it

这篇论文把训练过程写得很透明:一台 DGX Spark、两块 LoRA、十来个小时就做出领域模型,还公布了两条失败路线。做垂直领域微调的工程师能直接抄它的数据清洗和评测方法,也能看到检索增强带来的实际提升幅度。

研究团队以 40 亿参数的 OceanGPT-basic-4B-Instruct 为基座,在单台 NVIDIA DGX Spark 上通过 LoRA 适配训练出海洋工程领域模型 MarineSparkLM-4B。每个适配器仅更新 4.06 亿参数中的 3300 万(0.81%),第一个适配器用 4 小时 40 分钟、第二个仅用 10 分钟完成训练。

Body

第一个适配器在 6160 万 token 的语料上做 next-token prediction,语料来自 13879 篇源文档,经去重和两级关键词过滤后保留 7069 篇,验证损失从 2.082 降到 1.976。由于测试发现该适配器在检索增强的压载水问答提示下仍会偏题,团队又用 2295 个问答对训练了第二个指令适配器,且损失只计算答案 token。

在 20 个保留提示、满分 120 分的人工评测中,基座模型、续训适配器、指令适配器、指令适配器加检索分别得 42、56、84、88 分。绝对准确率仍然偏低:无检索时 40 题对 14 题,有检索时对 20 题。论文还公开了完整流程和两个负结果,包括一次 100 步内过拟合的指令训练。

Original · arXiv cs.AI

MarineSparkLM-4B: A Lightweight Language Model for Marine and Offshore Engineering Built by Parameter-Efficient Adaptation

Marine and offshore engineering text is spread across preprints, open-access journals, encyclopedias and institutional documents, and the amount of it is small by pretraining standards. We ask how far a lightweight language model for marine and offshore engineering can be taken with parameter-efficient adaptation on a single workstation-class device. Starting from the 4-billion-parameter OceanGPT-basic-4B-Instruct, we train two independent LoRA adapters, each updating 33.0 million of 4.06 billion parameters (0.81%). Training took 4 hours 40 minutes for the first adapter and 10 minutes for the second on one NVIDIA DGX Spark. The first adapter is trained by next-token prediction on a 61.6-million-token corpus covering ten engineering themes. The corpus was built from 13,879 source documents, of which 7,069 were retained after deduplication and a two-tier keyword filter that removes cross-domain lexical collisions. Its validation loss falls from 2.082 to 1.976 over 1,500 steps. Direct testing showed that this adapter could still drift off topic on a retrieval-grounded ballast-water prompt, so we trained a second adapter on 2,295 question-answer pairs (114 hand-written and 2,181 generated by the base model from corpus passages) with the loss restricted to answer tokens. On 20 held-out prompts scored by hand for accuracy, topicality and specificity (maximum 120), the plain base model, the continuation adapter, the instruction adapter and the instruction adapter with retrieval scored 42, 56, 84 and 88. Absolute accuracy stays low (14 of 40 without retrieval and 20 of 40 with it). We report the full pipeline, two negative results (a reverted keyword fix and an instruction run that overfit within 100 steps), and the limits of the evaluation, which uses one small prompt set, no inter-rater agreement measure and model-generated reference answers.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include OceanGPT, LoRA, fine-tuning, open-source models.

Source attribution: curated from arXiv cs.AI (@Cheng Siong Chin).