视频生成模型新训练方式：多专家分布式训练，推理时组合

精选理由

这种分布式训练思路颠覆了传统大模型训练范式，做模型训练或视频生成的开发者可以看看论文，或许能启发新的低成本训练方案。

AI 摘要

一位技术博主分享了一种新颖的视频生成模型训练方法，团队没有使用大型互联GPU集群，而是用多个小型、独立的GPU集群分别训练不同的“专家”模型。这些专家模型在训练时无需通信，训练完成后通过一个智能路由器在推理时动态组合，协同工作。这种方法降低了硬件门槛，且效果出色。论文链接已附，值得技术爱好者深入阅读。

原文 · Ate-a-Pi

For the nerds out there: The way this model was t…

For the nerds out there:

The way this model was trained was pretty cool, and probably, the first time this has been done to train a video generation model.

If you want to train a model, you rent a huge GPU cluster and do it there.

But this model was trained differently:

The team that built this model trained different "experts" using separate GPU clusters and data. These GPUs were all different, and didn't need any communication between them.

So instead of allocating 100 interconnected GPUs to train the model, they used 3 GPUs here, 5 there, 4 more over there, etc.

After training, they added a smart router on top of all of the trained experts. This router's job is to take an inference request, and route it to the appropriate experts.

In other words, instead of training a single model, they trained multiple smaller models that work together at inference time.

And the results were really good!

Here is the link to their paper: https://t.co/0hDxR6lRlC. It's a very good read.

查看原推