German AI consortium releases Soofi S, an open 30B model that tops benchmarks in both English and German

The German consortium behind the AI model Soofi S acknowledged in version 3.0 of its tech report that test questions from the science benchmark GPQA accidentally ended up in the training data. The community identified the error by examining the publicly available data. The team subsequently removed the benchmark from its evaluation and recalculated all results. This mistake raises questions about how the data used in training AI models affects the accuracy of results, highlighting the need for greater transparency in training processes.