Analysis & Opinions - Communications of the ACM

LLMs’ Data-Control Path Insecurity

| May 09, 2024

Someday, some AI researcher will figure out how to separate the data and control paths. Until then, we’re going to have to think carefully about using LLMs in potentially adversarial situations—like on the Internet.

Back in the 1960s, if you played a 2,600Hz tone into an AT&T pay phone, you could make calls without paying. A phone hacker named John Draper noticed that the plastic whistle that came free in a box of Captain Crunch cereal worked to make the right sound. That became his hacker name, and everyone who knew the trick made free pay-phone calls.

There were all sorts of related hacks, such as faking the tones that signaled coins dropping into a pay phone and faking tones used by repair equipment. AT&T could sometimes change the signaling tones, make them more complicated, or try to keep them secret. But the general class of exploit was impossible to fix because the problem was general: Data and control used the same channel. That is, the commands that told the phone switch what to do were sent along the same path as voices.

Fixing the problem had to wait until AT&T redesigned the telephone switch to handle data packets as well as voice. Signaling System 7—SS7 for short—split up the two and became a phone system standard in the 1980s. Control commands between the phone and the switch were sent on a different channel than the voices. It didn’t matter how much you whistled into your phone; nothing on the other end was paying attention.

This general problem of mixing data with commands is at the root of many of our computer security vulnerabilities. In a buffer overflow attack, an attacker sends a data string so long that it turns into computer commands. In an SQL injection attack, malicious code is mixed in with database entries. And so on and so on. As long as an attacker can force a computer to mistake data for instructions, it’s vulnerable.

For more information on this publication: Belfer Communications Office
For Academic Citation: Schneier, Bruce.“LLMs’ Data-Control Path Insecurity.” Communications of the ACM, May 9, 2024.

The Author