Lenny's Podcast: Product | Career | Growth
00:13:01 · Simon Willison, Lenny Rachitsky
What does it look like if you're not reviewing the code, if you're not looking at that code, but you're also not vibe coding, you're not throwing everything to the wind and seeing what happened? You're applying professional practices and quality expectations to code that you're not directly reviewing. The reason it's called the dark factory is there's this idea in factory automation that if your factory is so automated that you don't need any people there, you can turn the lights off. The machines can operate in complete darkness if you don't need people on the factory floor. What does that look like for software? This company called StrongDM has been pushing this and doing some really interesting experiments around this. That I think is the ... that's futuristic. We're trying to figure out what that looks like and how we can responsibly build software in that way right now, and making some quite interesting discoveries about things that work and things that don't work, but that to me is the next barrier. [00:13:57] S1: Let's follow that thread, so what is this factory doing? So there's an element of no one's looking at the code really, but how does that change how software's built? Are people still coming up with the ideas and telling in this factory, build this thing for me? [00:13:57] S0: Oh, exactly. [00:13:57] S1: Okay. [00:14:10] S0: So, this is the fascinating thing is ... so there's a policy of nobody writes any code and quite a few companies are beginning to introduce that now, because-
00:14:10 · Simon Willison, Lenny Rachitsky
[00:14:10] S0: So, this is the fascinating thing is ... so there's a policy of nobody writes any code and quite a few companies are beginning to introduce that now, because- [00:14:18] S1: Just to be clear, the policy is you cannot write code, it has to be written by AI. [00:14:21] S0: You can type code into a computer, exactly. [00:14:23] S1: Yeah.
00:14:24 · Simon Willison
[00:14:24] S0: Honestly, six months ago I thought that was crazy, and today probably 95% of the code that I produce I didn't type it myself. So that world is practical already, because the latest models are good enough that you can tell them, "Oh no, rename that variable and refactor that and add this line there," and they'll just do it and it's faster than you typing on the keyboard yourself. The next rule though is nobody reads the code, and this is the thing which StrongDM started doing back in, I think it was August last year. They said, "Okay, we're not going to read the code." So, what does that mean? How do you produce software that works and is good if you're not reading the code? They've come up with a whole bunch of answers. One of the most interesting was the way they did testing where in traditional software some companies will have a QA department. The engineers write about your software and then you throw it over the wall to the QA department and they test it furiously to figure out if it's working or not. That I think went out of fashion a bit over the past five to 10 years from what I've seen in Silicon Valley, because you kind of want your engineers to take responsibility for the code they're writing being good. But what if you can simulate that QA department? So, what StrongDM were doing is they had a swarm of agent testers who were actually simulating end users. So, the software that they were building ... this is crazy. The software is security software for access management, so when you start as a company and somebody needs to assign you access to Jira and then give you access to Slack and all of that kind of thing, they were building software for that. That's very security-adjacent. That's not the kind of thing that you should be vibe coding at all based on most people's understanding of how the world works. They're a legitimate security company who've been doing this stuff without AI for years, so it's not like they didn't understand the risks. So, the way they did their testing is they had this swarm of simulated employees all in a simulated Slack channel saying things like, "Hey, could somebody give me access to Jira?" The Slack channel itself is simulated, we'll talk about that in a moment, and 24 hours a day they're making requests and saying, " Hey, I need access to Jira," and all of those kinds of things at an enormous cost.
00:16:33 · Simon Willison, Lenny Rachitsky
They were spending $10,000 a day on tokens, I think, simulating with these end users. [00:16:38] S1: A day?
00:16:38 · Simon Willison
[00:16:38] S0: I believe so, but it meant that the software was being very robustly tested in all of these different ways. Yeah, it's kind of similar to having a manual QA team, except one that never sleeps. I thought that was fascinating as a example of thinking outside of the box. Taking this question, how do we tell our software's good if we're not reviewing the code and trying to find creative answers to it. The other thing that was interesting is that the Slack channel itself wasn't actually Slack, because it turns out if you test against real software like Slack and so forth, they all have rate limits and they won't let you just run 10,000 simulated people at a time. So, what they did is they built their own simulation of Slack and Jira and Okta and all of this software they were integrating with. The way they did that is they basically took the API documentation for the public APIs for Slack and the open source client libraries, and they told their coding agents, "Build this. Build me a simulation of this API," and they did. So, this company is ... and this is one of the things ... I went to a demo that they gave back in October. One of the things that really sat with me is that they had their own simulated version of Slack and Jira and all of these different systems that they could then build their software against, which cost them nothing, because once they spun it up it was a little Go binary that sat there, and they even had interfaces. They had a fake version of the Slack interface that they'd vibe coded up that let them see what was going on. Absolutely fascinating.
00:18:08 · Lenny Rachitsky
[00:18:08] S1: That is such a cool story, and I love these stories of just companies at the bleeding edge trying to see what's possible and have an advantage essentially. So, what I'm hearing here is the QA piece is like the new piece in this factory. So we already have Codex, Claude Code, they can go off and build stuff. Is the innovation here, okay, now you've built all this stuff, is it actually any good? Is there a reason why Codex and Claude Code couldn't do this themselves? Why do you need this factory concept?