Psychological methods reveal major weaknesses in AI security testing

Researchers at the UK AI Security Institute employed psychometric methods to demonstrate that popular safety benchmarks for language models do not measure a consistent trait. The blanket blocking of requests can artificially inflate a safety score, even as the model's utility decreases in daily use. Additionally, the study offers a method to identify models that behave more cautiously during tests than in normal operation, highlighting the need for improved safety standards.