Open-world evaluations for measuring frontier AI capabilities

Researchers have begun testing artificial intelligence in real-world environments, introducing the term "open-world evaluations." This approach aims to measure models' abilities to create real products or conduct scientific experiments. In the first trial, an AI agent developed an iOS application with only two errors, indicating useful potential but also possible risks. Seventeen researchers from various fields collaborated on the CRUX project to assess AI capabilities through these evaluations.