Lenny's Podcast: Product | Career | Growth
00:08:50 · Simon Willison
[00:08:50] S0: Let's talk about the two, there's the vibe coding side of things and then there's the ... and I like Andrej Karpathy's original definition of vibe coding, which is when you don't even look at code and you basically just go on the vibes. You say, build me something that does X, and it builds it and you play with it. If it looks good, then great, and if it doesn't quite do it, you keep on going back and forth with it. But it's very hands-off, you're not looking at code, it's ... so he originally said, "This is great for having fun and prototyping," and it then exploded way out of that. I think today vibe coding is effectively it's ... the definition I use is it's when you're not looking at the code, you don't care about code and maybe you don't understand the code. Non-programmers can now tell Claude what to build and it can build them a little app, and I love that. I absolutely love that we're democratizing the art of getting a computer to do stuff for you, of automating tedious things in your life by knocking out these little tools. Of course, the problem is that there is a limit on how much you can do with that responsibly. I like to tell people, if you're vibe coding something for yourself where the only person who gets hurt if it has bugs is you, go wild. That's completely fine. The moment you're vibe coding code for other people to use where your bugs might actually harm somebody else, that's when you need to take a step back and say, hang on a second, this is not a responsible way of using these tools. The challenge is that understanding what's responsible and what isn't is in itself a expert level skill, so knowing that once you start dealing with scraping other people's websites, maybe you'll damage their websites by hitting them too hard. There are so many ways that you can cause damage if you don't know what you're doing. But I love that liberation and I love that people can come to meetings with a prototype that they knocked up of that idea that illustrates the idea. I think those things are wonderful. The ongoing debate has been what do we call it when a professional software engineer uses these tools to write real code that's production-ready that they've reviewed and they've checked all of the details of? A lot of people call that vibe coding as well.
00:10:55 · Simon Willison
I think that devalues vibe coding as a term, 'cause it's useful to say, "I vibe coded this," as in I haven't even looked at how it works, it's not production-ready, but it's kind of a cool prototype. The moment vibe coding mean everything that touches AI, it effectively ends up needing programming, because we're all moving in a direction where our code is mediated through AI at some point. So, what do we call it for professionals? I've gone with agentic engineering, because I think the thing to emphasize is these coding agents, right? If you're asking ChatGPT to knock out some code, that's a different thing from if you're running Codex and having it write the code, debug the code, test the code, all of that. I think that agentic engineering is such a deep and fascinating discipline, because the art of getting really good results out of this, the art of having them help you build software you could deploy to a million people, that's never going to be easy, that's never going to be trivial. That's always going to require a great deal of depth of experience in how software works and how these agents work, and I love that. I'm kind of writing a book about it now that I'm publishing a chapter at a time on my blog. The best form of writing, because I don't have an editor or any pressure from a publisher, is just when I feel like writing another chapter I can do that, but there's so much to discuss. But yeah, so I think right now the frontier is how do we build professional software using coding agents? How do we build software that is ... I don't just want to build software that's good, I want us to build software that is better than we were building before. If the agents let us move a bit faster, but we're still churning out the same quality of software, that's less interesting to me than if the software we're producing has less bugs, more features, it's higher quality, it's better software, because we're harnessing these tools. The really interesting future is something which some people have been calling the dark factory pattern or software factories. This is the idea where right now if you're a professional using these tools, the way you do it is you tell them what to build and then you look at the code and you review that code really carefully and make sure it's doing the right thing.
00:13:01 · Simon Willison, Lenny Rachitsky
What does it look like if you're not reviewing the code, if you're not looking at that code, but you're also not vibe coding, you're not throwing everything to the wind and seeing what happened? You're applying professional practices and quality expectations to code that you're not directly reviewing. The reason it's called the dark factory is there's this idea in factory automation that if your factory is so automated that you don't need any people there, you can turn the lights off. The machines can operate in complete darkness if you don't need people on the factory floor. What does that look like for software? This company called StrongDM has been pushing this and doing some really interesting experiments around this. That I think is the ... that's futuristic. We're trying to figure out what that looks like and how we can responsibly build software in that way right now, and making some quite interesting discoveries about things that work and things that don't work, but that to me is the next barrier. [00:13:57] S1: Let's follow that thread, so what is this factory doing? So there's an element of no one's looking at the code really, but how does that change how software's built? Are people still coming up with the ideas and telling in this factory, build this thing for me? [00:13:57] S0: Oh, exactly. [00:13:57] S1: Okay. [00:14:10] S0: So, this is the fascinating thing is ... so there's a policy of nobody writes any code and quite a few companies are beginning to introduce that now, because-
00:14:10 · Simon Willison, Lenny Rachitsky
[00:14:10] S0: So, this is the fascinating thing is ... so there's a policy of nobody writes any code and quite a few companies are beginning to introduce that now, because- [00:14:18] S1: Just to be clear, the policy is you cannot write code, it has to be written by AI. [00:14:21] S0: You can type code into a computer, exactly. [00:14:23] S1: Yeah.
00:14:24 · Simon Willison
[00:14:24] S0: Honestly, six months ago I thought that was crazy, and today probably 95% of the code that I produce I didn't type it myself. So that world is practical already, because the latest models are good enough that you can tell them, "Oh no, rename that variable and refactor that and add this line there," and they'll just do it and it's faster than you typing on the keyboard yourself. The next rule though is nobody reads the code, and this is the thing which StrongDM started doing back in, I think it was August last year. They said, "Okay, we're not going to read the code." So, what does that mean? How do you produce software that works and is good if you're not reading the code? They've come up with a whole bunch of answers. One of the most interesting was the way they did testing where in traditional software some companies will have a QA department. The engineers write about your software and then you throw it over the wall to the QA department and they test it furiously to figure out if it's working or not. That I think went out of fashion a bit over the past five to 10 years from what I've seen in Silicon Valley, because you kind of want your engineers to take responsibility for the code they're writing being good. But what if you can simulate that QA department? So, what StrongDM were doing is they had a swarm of agent testers who were actually simulating end users. So, the software that they were building ... this is crazy. The software is security software for access management, so when you start as a company and somebody needs to assign you access to Jira and then give you access to Slack and all of that kind of thing, they were building software for that. That's very security-adjacent. That's not the kind of thing that you should be vibe coding at all based on most people's understanding of how the world works. They're a legitimate security company who've been doing this stuff without AI for years, so it's not like they didn't understand the risks. So, the way they did their testing is they had this swarm of simulated employees all in a simulated Slack channel saying things like, "Hey, could somebody give me access to Jira?" The Slack channel itself is simulated, we'll talk about that in a moment, and 24 hours a day they're making requests and saying, " Hey, I need access to Jira," and all of those kinds of things at an enormous cost.
00:16:33 · Simon Willison, Lenny Rachitsky
They were spending $10,000 a day on tokens, I think, simulating with these end users. [00:16:38] S1: A day?