Prompt Injection as Role Confusion

A recent study by Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell explores the challenges models face in distinguishing trusted texts. The findings indicate that models struggle to differentiate between privileged text and untrusted user input, focusing more on style than content. The research reveals that writing style significantly impacts how models classify text, leading to concerning jailbreak scenarios. For instance, text mimicking a model's internal thought style can override its initial training, causing serious security risks.