By averaging the parameters of multiple fine-tuned models, practitioners can create a single model that inherits the strengths of its constituents.
The concept of model soups involves taking several models that have been fine-tuned from a shared base model and averaging their parameters into a single set of weights. Unlike traditional training, which requires significant compute to optimize a single model for a specific task, weight averaging allows for the creation of a new model at the cost of a simple mathematical addition.
This lineage traces back to research on model soups (ICML 2022) and TIES-merging, evolving into more recent frameworks. The goal is to produce a merged model that generalizes better than any individual ingredient by capturing the specialized knowledge distributed across the fine-tuned checkpoints.
While standard model merging focuses on creating a single optimized model, the soup of experts approach extends this toward specialization. Instead of a static merge, it envisions a bank of expert weights that can be recombined on demand to suit the specific requirements of a task.
It is important to distinguish this from a sparse mixture of experts (MoE) architecture. While a MoE uses routing mechanisms to activate specific sub-networks during inference, a soup of experts operates through the composition of weights, creating a unified parameter space through merging rather than architectural sparsity.
The field is moving from theoretical research toward practical application. Tools like mergekit allow practitioners to implement these techniques outside of a research lab. This is evidenced by the 'merge recipes' shared on the Open LLM Leaderboard, where users actively compose weights to achieve higher performance.
As the field matures, evaluation frameworks like MergeBench and comprehensive reviews, such as the 2026 ACM Computing Surveys review of model merging, are essential for understanding when these compositions succeed and when they fail to generalize.
How does a soup of experts differ from a mixture of experts?
A mixture of experts (MoE) is an architectural design that uses routing to select active sub-networks during inference. A soup of experts is a method of model merging that uses weight averaging to combine multiple fine-tuned models into one.
Do I need to retrain a model to create a soup?
No. The core mechanism of weight averaging is to take existing fine-tuned models and average their parameters, avoiding the computational cost of a new training run.
Tracking merge recipes, benchmarks, and what actually survives averaging. Drop an email for the notebook.