Researchers train AI model that hits near-full performance with just 12.5 percent of its experts

Researchers at the Allen Institute for AI and UC Berkeley have developed EMO, a mixture-of-experts model that specializes in content domains rather than word types. This allows for the removal of 75% of the experts while only losing about one percentage point of performance, making MoE models practical for memory-constrained environments. This advancement is significant as it can drastically reduce memory requirements while maintaining high performance. The new model achieves near-full performance using just 12.5% of its experts, opening new avenues for AI applications.