Lenny's Podcast: Product | Career | Growth
00:37:42 · Simon Willison
[00:37:42] S0: I feel like the word artisanal does... Artisanal handcrafted software I think is going to be valued more. Something I've noticed in my own work is sometimes I'll have an idea for a piece of software, Python library or whatever, and I can knock it out in an hour and get to a point where it's got documentation and tests and all of those things and it looks like the kind of software that previously I've spent several weeks on, and I can stick it up on GitHub and everything, and yet I don't believe in it. And the reason I don't believe in it is that I've got to rush through all of those things. I think the quality is probably good, but I haven't spent enough time with it to feel confident in that quality. Most importantly, I haven't used it yet. It turns out when I'm using somebody else's software, the thing I care most about is I want them to have used it for months. I want other people to have put that software into practice. So I've got some very cool software that I've built that I've never used. It was quicker to build it than to actually try and use it. And so the way I've been dealing with that is I always put alpha on it. If you see my software and it says it's an alpha, that probably means I haven't actually used it yet for most of my project, which is a bit of a cheat code, alpha [inaudible 00:38:53]. But isn't that interesting? It used to be if you looked at software and it had high quality tests and documentation and everything, it meant it was good, and now that signal is gone.
00:39:02 · Lenny Rachitsky, Simon Willison
[00:39:02] S1: It's almost like we need a proof of work for this versus the blockchain. [00:39:05] S0: A proof of usage. [00:39:07] S1: Proof of use. [00:39:07] S0: Yes, exactly. [00:39:08] S1: Oh, man. On this note of handcrafted code, I don't know if you know this, this is so interesting, data labeling companies are buying old GitHub repos of handwritten code to train their models on, and they're paying a lot of money for artisanal human written code. [00:39:24] S0: Oh, that's fascinating. That's the pre-World War II, the metal that you can dig up from old shipwrecks, which is before the first nuclear explosions, and so it's not got the radiation baked into the metal. It's that whole thing. [00:39:42] S1: Wow. That's a great metaphor. [00:39:44] S0: That's fascinating. [00:39:44] S1: Yeah. So they're looking for code pre-2022, I think, whenever ChatGPT emerged. [00:39:49] S0: Wow. [00:39:51] S1: So if you've got some, you can make a fortune. [00:39:54] S0: Problem is, I open source all my stuff, so it's already out there. It's been used to train the models already. [00:39:54] S1: It's been slurped up already. [00:40:00] S0: Yep. [00:40:01] S1: Oh, man. Okay. Let me ask you this question. I'm just curious about this prediction. I know you're not a prediction person, although you do make predictions and you seem to be right often. When do you think 50% of engineers in the world will be... AI will be writing 100% of their code? How close to that do you think we are?
00:40:17 · Simon Willison
[00:40:17] S0: So I'm going to refactor that to 95% of their code. I don't think we'll get to it. But yeah, it's very difficult to say worldwide, partly because there are cultural differences. I spend way too much time on hack and use, and something I've noticed about hack and use is a conversation that starts at midnight Pacific time and goes until 8:00 AM, very different tone because it's the Europeans. And the Europeans are a lot more AI skeptic than the Americans are generally. So I think different countries are going to have different cultures around this. At the same time, I think it's become undeniable this year that this stuff produces good code. It used to be that you could say, "I don't use this stuff because the code is bad." And that was a justifiable position. That's not justifiable anymore. The code is now good. It's good code, for my definition of good code at least. So we're saying 50% of engineers, let's say 15 engineers, majority of their code, it could happen by the end of this year. It could, because the technology is good enough now. And I feel like the challenge now is getting people to learn how to use this stuff, which is difficult because using this stuff, everyone's like, "Oh, it must be easy. It's just a chatbot." It's not easy. That's one of the great misconceptions in AI is that using these tools effectively is easy. It takes a lot of practice and it takes a lot of trying things that didn't work and trying things that did work. But yeah, I expect by the end of this year, it will not be uncommon to have an engineer say that almost all of that code is written by AI.
00:41:56 · Lenny Rachitsky, Simon Willison
[00:41:56] S1: That was the same rough idea I had. And how crazy is that, how quickly- [00:42:00] S0: It's wild. [00:42:00] S1: ... this job has changed and what is possible. And I think people, this is a good example of people underestimate how quickly things can change. I think Dario was predicting this a year or two ago, just, "Oh, 100% of code's going to be written by AI," and we were just like- [00:42:14] S0: We all laughed at him. Yeah. [00:42:15] S1: Right. Exactly. Like, "What are you talking about? It's so bad. It's so bad at writing code." And this might come for other jobs that people don't see coming, which is scary and interesting and exciting. [00:42:27] S0: Honestly, I'm not an AI doomer in the slightest. The economics of it do make me nervous. Are we really going to wipe out a tenth of white collar knowledge work jobs in the next few years? I really hope not, because I don't know how the economy adapts to that. So yeah, that's complicated. [00:42:46] S1: Yeah. Actually, I'm doing a report that's coming out, it'll come out ahead of this episode, looking at the job market in tech. And surprisingly, just at tech companies, we're at the highest number of open engineering roles, open PM roles. [00:43:00] S0: Interesting. [00:43:02] S1: Except for during the crazy peak during COVID. So it's kind of coming back to that. Basically, it's the highest number of open roles in three and a half-ish years for engineers and PMs at tech companies globally. [00:43:16] S0: That's very interesting. It's funny, isn't it? Because you get all of these headline grabbing-
00:43:16 · Simon Willison, Lenny Rachitsky
[00:43:16] S0: That's very interesting. It's funny, isn't it? Because you get all of these headline grabbing- [00:43:21] S1: Layoffs. [00:43:22] S0: Yeah. Was it Block that laid off 4,000 people recently? [00:43:26] S1: Yeah. Yeah. [00:43:26] S0: But the question there is always how much of that is AI and how much of it is overhiring during COVID and re-corrections and all of that kind of thing? It's always very difficult to tell. So the number of open jobs, on the one hand, maybe that's a better signal, but on the other hand, the recruitment market has been driven completely crazy by all of this stuff. All of the job ads are written by AI, the resumes are AI. People in recruitment are saying that it's never been this hard to filter through and hire people, and people who are hiring jobs say they applied to 200 things and got nobody hearing back. So it's hard, right? The macroeconomic indicators for this stuff are lagging. And at some point we should start getting more confident numbers about what the impact actually is. [00:44:13] S1: Yeah. Interestingly, the number of recruiter open roles is also approaching record numbers- [00:44:19] S0: Hilarious. That's so interesting. [00:44:20] S1: ... which is an interesting leading indicator of demand for hiring. So there's interesting trends in spite of the layoffs. So yeah, what a wild world. So you've mentioned this book you're working on. This is the agentic engineering pattern stuff, right? [00:44:33] S0: Yes.
00:44:33 · Simon Willison, Lenny Rachitsky
[00:44:33] S0: Yes. [00:44:34] S1: Okay, cool. So I want to talk about this. So you point it out. People think it's easy to build with AI. It's like, "Oh, it's going to do all these things for us. What are we going to do all day?" To your point, it's actually not. There's a lot of very specific skills you need to do this well, and you're putting them together on your blog. We'll point to it. I want to talk through a few of them to help people do this better. So one is this idea of just writing code is cheap now. You touched on this a bit. Maybe just share why this is such an important thing to know and keep in mind.