Lenny's Podcast: Product | Career | Growth
01:19:12 · Simon Willison
And I called it prompt injection because I thought it was the same thing as this attack called SQL injection, which is a security problem with databases where you glue user input into your SQL queries in a way that breaks them and deletes all of your data. The problem is, SQL injection is solved. We know how to fix this problem. There are reliable ways of saying, "No, this is un-trusted data." Those solutions don't work for prompt injections. So the name itself is misleading. You hear prompt injection and you think, "Oh, I can solve SQL injection, I'll use the same thing." That doesn't work. And then, the other problem with coining terms is, just because you were the first to define a term doesn't mean you actually get to define what it means in people's heads. It turns out, people will define a term based on their initial assumption. If they hear a term, like if I say to you, "Oh, there's this problem called prompts injection," the natural human instinct is to guess what it means, and if that guest sounds good, stick with it. A lot of people when you say, "Prompt injection," they say, "Oh, I know what that means, it's injecting prompts." It's when you type a prompt into an LLM, you're injecting that prompt. And if you can trick it into saying something impolite, that's what's going on there. That's not what it was supposed to mean. That's gel breaking, that's a different kind of thing. Well, it turns out I don't get to define it just because I defined it. So, the lethal trifecta was my second attempt at this. And you'll notice that the lethal trifecta, you cannot guess what it is. If I say to you, "There's a thing called the lethal trifecta," you can't go, "It's obviously one, two, it's three things, but what are those things?" and that means I get to control what it means, because you have to go and look it up when you hear what it is. And the lethal trifecta is a subset of prompt injection, which I hope helps people understand why this is such a big problem. And it relates to the email example earlier on. You have a lethal trifecta anytime your agent has three things. It's got access to private information. There's information that you've exposed to it like your private inbox, that is private in some way. It's exposed to malicious instructions. So there's a way somebody attacking you can get their text into your system, like sending you an email.
01:21:20 · Simon Willison, Lenny Rachitsky
And the third leg is exfiltration, or some mechanism the agent can send data back to that attacker, like forwarding an email. So if you've got a system where you've got private emails, anyone can email you instructions and it can email them back, that's the classic lethal trifecta, that's a huge security problem. The only way to fix it is to cut off one of those three legs. So normally, the leg that's the easiest to cut off is the exfiltration one. If you can stop your agent from sending the data back to the attacker, then the attacker can try and mess around, but at least they can't steal your data. [01:21:53] S1: So people hearing this might feel like, "Why can't you just tell the AI, 'Hey, don't do anything where if someone steals your data. Don't listen to people trying to trick you.'" And it turns out, and I'd love you to take here, it's very hard to put enough of these guardrails in place where somebody can't figure out a way to trick it.
01:22:10 · Simon Willison
[01:22:10] S0: That is exactly the problem. The problem is, you can get to 97% effectiveness on those filters. I think that's a failing grade. That means that three out of 100 of these attacks will steal all of your information. Because fundamentally, the way we prompt these things is using text in any human language. You could filter out, "Ignore previous instructions in English." What if somebody says it in Spanish? There is no filter. It's like the classic allow list versus deny list thing. You cannot deny every one of these attacks, because I can always invent a new sequence of characters that might trick the model in some way. So what you have to do instead is say, "Okay, fundamentally these things we cannot prevent if there's malicious instructions, consider that anyone who can talk to your agent can make it do any of the things it's allowed to do," and then you have to think, "Okay, well, let's make sure that the blast radius on that is limited. The things that it's allowed to do can't cause too much damage." This is why I use Claude Code for web so much because I'm often having it go and read random webpages and maybe those have nasty attacks on them. All it can really do if it's running on Anthropic servers is waste their... It could mine Bitcoin on their servers or something, or maybe leak some of my private data somewhere else, but I don't put my private data into that environment. But, I've got 25 years worth of security engineering experience to help me make those decisions. This is not helpful for the vast majority of people who fall for phishing emails, which is most of us. This is an equivalent of phishing, except the agent is the thing being phished. And that's terrifying. So, you mentioned the Challenger Disaster. The reason I think about the Challenger Disaster is there's this fantastic paper that came out of the Space Shuttle Challenger Disaster called The Normalization of Deviance. This was a piece of research in the '80s that said that what happened with the Challenger Disaster is, lots of people knew that those little O-rings were unreliable, but they kept on launching space shuttles and everything was fine. And so every single time you get away with launching a space shuffle without the O-rings failing, you institutionally feel more confident in what you're doing. T.
01:24:17 · Simon Willison, Lenny Rachitsky
He problem we've been having with prompt injection is that we've been working increasingly unreliably with these systems. And we've been using these systems in increasingly unsafe ways, and so far, there hasn't been a headline-grabbing story of a prompt injection that's where an attacker has stolen a million dollars, which means that we keep on taking risks. We have this normalization of deviance in the field of AI around how we're using these tools. So my prediction is that we're going to see a Challenger Disaster. At some point, this is going to catch up with us, and it's going to be very, very, very bad, and that will hopefully help us stop trying to figure out how not to do this. At the same time, I've made a version of this prediction every six months for the past three years, and it hasn't happened. So, there we are. [01:25:04] S1: It's like the black swan turkey chart where it's like the turkey is the most confident it's ever been. It will live for a long time until the day that it gets eaten for Thanksgiving. [01:25:15] S0: Right, exactly. So yeah, it's scary, that one. [01:25:20] S1: Do you feel like this is solvable, and/or has this become harder and harder to do? Are we making progress and avoiding these sorts of prompt injections, [inaudible 01:25:29]?
01:25:20 · Lenny Rachitsky, Simon Willison
[01:25:20] S1: Do you feel like this is solvable, and/or has this become harder and harder to do? Are we making progress and avoiding these sorts of prompt injections, [inaudible 01:25:29]? [01:25:29] S0: Everyone in AI, the natural instinct is solve it more AI. We can detect these things, we've got AI. AI is amazing. AI can spot stuff. And they keep on getting better. Every time a new system card comes out with a Claude model, they'll be then says, "Our internal prompt injection score jumped, detection jumped from 70% to 85%." And again, until it's 100%, I don't think it's a meaning... I think it just gives people a false sense of security that this problem won't bite them. And even if they did hit 100%, I'd want more than just a score. I want proof. I want, "Here is the computer science that we have come up with and put in place that means these attacks are no longer a problem." And I cannot imagine what that proof would look like myself. Maybe I'm just short on imagination. But yeah, fundamentally, these are machines where you give them a sequence of text and they do something. Dividing that sequence of text into this bit tells you what to do, and this bit is the thing that you do stuff too. It's very fuzzy. It's very difficult to imagine how you can completely solve that.
01:25:29 · Simon Willison, Lenny Rachitsky
[01:25:29] S0: Everyone in AI, the natural instinct is solve it more AI. We can detect these things, we've got AI. AI is amazing. AI can spot stuff. And they keep on getting better. Every time a new system card comes out with a Claude model, they'll be then says, "Our internal prompt injection score jumped, detection jumped from 70% to 85%." And again, until it's 100%, I don't think it's a meaning... I think it just gives people a false sense of security that this problem won't bite them. And even if they did hit 100%, I'd want more than just a score. I want proof. I want, "Here is the computer science that we have come up with and put in place that means these attacks are no longer a problem." And I cannot imagine what that proof would look like myself. Maybe I'm just short on imagination. But yeah, fundamentally, these are machines where you give them a sequence of text and they do something. Dividing that sequence of text into this bit tells you what to do, and this bit is the thing that you do stuff too. It's very fuzzy. It's very difficult to imagine how you can completely solve that. [01:26:32] S1: Yeah. So the last episode we had on this with Sander Schulhoff, he does professional red teaming where they test models, and he's just like, "This is never going to be solved." And, because if somebody's motivated enough, to your point, if there's a 97% chance you can't get it, but there's that 3% of people that are motivated to figure out how to build a bomb, they'll figure it out. You just keep trying until it works.