Researchers pinpoint why larger language models pick up skills that small ones miss

A recent study reveals that small language models struggle with rare tasks due to frequent tasks overshadowing their learning. Analyzing models from 4 million to 4 billion parameters, researchers found that increasing the frequency of target tasks in training data could effectively address this issue. The findings indicate that larger models acquire skills that smaller ones miss, emphasizing the importance of training data preparation. Instead of merely scaling up models, enhancing task frequency may lead to significant performance improvements.