这篇论文用LLM编排多个专业智能体,自动搞定数据工程到部署监控的全流程,比单智能体和纯AutoML更可靠,适合做生产级自动化参考。
该论文提出一个基于LLM编排的多智能体框架,将大数据即服务生命周期分解为数据摄取、数据清洗、特征工程、AutoML训练、模型评估、MLOps部署、监控和漂移检测等专业智能体。中央LLM编排层协调代理执行、验证中间输出、管理流程上下文并支持动态工作流组合。框架包含共享工件治理、可重复性支持、人在回路检查点和漂移感知反馈循环。在包含缺失值、分类变量、异常值、类别不平衡和模拟协变量漂移的受控表格基准数据集上,与手动ML、仅AutoML和单智能体LLM基线对比,该多智能体BDaaS管道实现了有竞争力的预测性能,并提高了工作流完成率、工件可追溯性、部署就绪度、可重复性和漂移恢复能力。
Trustworthy Self-Composable Big-Data-as-a-Service: An LLM-Orchestrated Multi-Agent Framework for Automated Data Engineering, AutoML, MLOps Deployment, and Drift-Aware Lifecycle Optimization
Big-Data-as-a-Service (BDaaS) platforms require re liable automation across data ingestion, cleaning, feature engi neering, model development, deployment, and post-deployment monitoring. However, existing LLM-based data science agents and AutoML systems mainly focus on isolated workflow stages, leaving limited support for lifecycle-level orchestration, artifact governance, human oversight, and drift-aware adaptation. This paper proposes a trustworthy self-composable BDaaS frame work based on LLM-orchestrated multi-agent collaboration. The proposed architecture decomposes the BDaaS lifecycle into specialized agents for data ingestion, data cleaning, feature engineering, AutoML training, model evaluation, MLOps de ployment, monitoring, and drift detection. A central LLM or chestration layer coordinates agent execution, validates interme diate outputs, manages workflow context, and enables dynamic workflow composition. The framework also incorporates shared artifact governance, reproducibility support, human-in-the-loop checkpoints, and drift-aware feedback loops. A prototype-based evaluation is conducted using controlled tabular benchmark datasets with missing values, categorical variables, outliers, class imbalance, and simulated covariate drift. Compared with manual ML, AutoML-only, and single-agent LLM baselines, the pro posed multi-agent BDaaS pipeline achieves competitive predictive performance while improving lifecycle-level reliability, including workflow completion, artifact traceability, deployment readiness, reproducibility, and drift recovery. The results suggest that LLM-orchestrated multi-agent systems can extend conventional AutoML toward trustworthy, adaptive, and production-oriented BDaaS lifecycle automation.