New math benchmark reveals AI models confidently solve problems that have no solution

A consortium of 64 mathematicians has launched the new SOOHAK benchmark, which includes 439 handwritten tasks, of which 99 are unsolvable. Google's Gemini 3 Pro leads in solving research problems with a 30% success rate. However, no model has surpassed 50% in recognizing broken tasks. The SOOHAK benchmark reflects the gap between impressive results and the broader research skills lacking in AI systems. Improving performance requires more computing power, yet this does not aid models in recognizing that some problems have no solutions. The benchmark underscores the challenges AI faces in research domains.