Combine several fine-tuned models into one by averaging their weights — composition at the weight level, with no extra training.
What it is
A model soup takes several models fine-tuned from a shared base and averages their parameters into a single set of weights. Done well, the merged model inherits strengths from each ingredient and often generalizes better than any single one — for the cost of an addition, not a training run.
“Soup of Experts” extends the idea toward specialization: a bank of expert weights that can be recombined on demand into a model tuned for the task in front of you.
Why it's worth watching
Rooted in Model Soups (ICML 2022) and TIES-merging, now accelerating: Soup-of-Experts (arXiv:2502.01804), MergeBench (arXiv:2505.10833), and a 2026 ACM Computing Surveys review.
A live tooling ecosystem (mergekit and the Open LLM Leaderboard's shared “merge recipes”) means practitioners, not just researchers, are composing weights daily.
The composition angle
If context folding composes in time, model soups compose in weight space. Both say the same thing: capability comes from combining specialized parts, not scaling one monolith.