The Pragmatic Engineer
00:23:28 · Gergely Orosz, Titus Winters
[00:23:28] S0: two fifty million lines of code in perspective because today, with AI, AI can generate, you know, like, we have crazy numbers. It it can generate tens of thousands of lines of code in an hour, etcetera, and and you can get to millions if if you want to, honestly, just an agent can do it. However, pre AI, Uber's mobile application was about a million lines of code on iOS and on Android and that was each written by 200 engineers for for years. BUN runtime at Anthropic just rewrote something like, I think it was like 500 or 900,000 lines of code and that was written by a small team. I find it rare to find examples of a project that has more than pre AI, more than let's say 3,000,000 lines of code and that will typically be hundreds of engineers writing it for for several years. So in the in the the reason I was like 250,000,000, that is about a 100 times more than I've heard for code base. And we're talking about, like, clearly, this is written for production. You're not, like, just goofing around and pushing some scaffolding here and there and and right. And and that when you said first party, it's easy to to have a lot lot of code when you take on an open source component, but no. [00:24:36] S1: Nope. Nope. That was all Google or authored. Yep. And so we have some insight into how things grow. [00:24:46] S0: Okay. So the largest refactoring history at that point in human history might have... You're not extricating. [00:24:51] S1: I don't think so. I mean, I can't prove it, but I I would be surprised.
00:24:51 · Titus Winters, Gergely Orosz
[00:24:51] S1: I don't think so. I mean, I can't prove it, but I I would be surprised. [00:24:56] S0: So how... Okay. When you have this type... You were in the middle of this 10 to 12 people. You have this massive massive task that seems giant and almost... How do you go about it? How do you say, like, how do you break it down? How do you plan for it? How do you speed it up? Because you clearly... If you just did it by hand, this would have not fully worked. Right. So one of the
00:25:17 · Titus Winters
[00:25:17] S1: things that had enabled any of this and that came out of having been on that, the low level c plus plus infrastructure team to start with, was, okay, if we own the interfaces that are kind of the nuts and bolts that everything at Google is written out of, like way below the database level, just like caches, containers, screen manipulation routines, command line flags, like that sort of thing. If we own those things, maybe it's valuable to make them better. But how do you do that when there's 50,000 call sites or a 100,000 edits that need to be made to change that API? And so luckily, was friends and org chart cousins eventually with the compiler team, especially Clang LLVM people. And so they started doing stuff in the AST transformation sort of state. AST. Abstract syntax tree. So like you take a parse tree. And, you know, it's Google, like MapReduce. So if you do a MapReduce over the code base where the mapper parses all of the translation units of your C plus plus like all of the build targets, and produces... Here is the parse tree. Then you can match the parse tree for specific... It's not text match. It's I guarantee you are calling this particular function in this particular context, and then you can apply some code to rewrite based on that. And so we had compiler assisted semi automated refactoring to generate all of the text edits. Turns out one of the things that was really hard at first was we could generate the edit, but we couldn't get the white space in the right places. No way. Yeah. Seriously. So Clang format exists because I wished for it. This is like the auto formatter for many, like, c plus plus sorts of languages at this point. But it's like, I literally cannot accomplish this refactoring until someone figures out a way to, like, hack together a auto white space for c plus plus And so you you do that. You generate one, you know, big edit. It is too large to fit in the code base because for any given code base, there's kind of a natural background level of how much change is happening. And if you're trying to edit 10,000 files, 20,000 files, it is likely the case that you will never be able to test and resync before something has gone out of sync. Yeah. Right? Someone else edited something. Yeah. Exactly. Merge conflicts, like parallel changes, that kind of stuff. Yeah. Exactly.
00:27:40 · Titus Winters, Gergely Orosz
You know, you do not want at a, you know, 10,000 plus person company to be like, everyone stop touching the repo for an hour. Let me do this thing. Like, that's not gonna scale. Right? And so the tech is you break all of... Like, you figure out how to do your refactoring so that the old API and the new one can coexist in spelling. Yep. And then you paralyze, like, just cut by directory structure. Like, alright. I'll send you this batch of updates, this batch of updates, and you wind up, you know, going like that. And then once everything is clean, then there's a race to make sure no one copy and pastes the old thing. And then you delete the old thing, the new thing is the default, you move on with life. You just do that a few 100 times, and your code base is in much better shape. [00:28:28] S0: Yeah. Wow. And and then you're you're doing this all all the while while other engineers, they're just doing their thing. They're using these things they don't even know about. They don't wanna know about you. I mean Exactly. They might be aware of you, but they shouldn't have to change their Yep. Just, as you said, slow down or Yep. But then we
00:28:28 · Gergely Orosz, Titus Winters
[00:28:28] S0: Yeah. Wow. And and then you're you're doing this all all the while while other engineers, they're just doing their thing. They're using these things they don't even know about. They don't wanna know about you. I mean Exactly. They might be aware of you, but they shouldn't have to change their Yep. Just, as you said, slow down or Yep. But then we [00:28:43] S1: did get amazing feedback on some of this. Like, at first, people were like, what are you doing? How... Why are you changing my code? It's like, I'm not okay with this. And very quickly, they're like, oh, this is great. Like, I go away. Please please move My code just gets better. Like, and I don't need to worry about looking up, like, is this the right way to do a thing? Because if it was the wrong way to do a thing, you would have gotten rid of it already. Like, when you have someone that is going through and just janitering the code base like that, copy and paste becomes the correct answer for how do I do this? Like, find someone else that did it once, you copy it, you move on with life, and you have evidence that it's probably a reasonable answer. Because if it was wrong, we would have fixed it already.
00:29:26 · Gergely Orosz
[00:29:26] S0: It's very interesting because today, obviously, if you did a big refactoring, the obvious tool that you would reach for is is AI as we've seen with so many companies having lots of success with... Well, in in the sense that AI is capable of the challenge that you had is you had this massive asset code base, rewriting it by hand would have not been feasible, but at Google, you came up with a very clever semi automated approach, was based on pretty hardcore language parsing and and coming up with new ways where you could formally, maybe informally or or at least prove that this thing will be correct Yeah. Which is what most teams would not have the availability capacity, even expertise to do. What everyone has the expert... The capability to do right now is like, oh, AI, go and do this and and do tests and all this. However, where I wanted to get to, like, rewriting with AI still feels so much more janky than what you did, which was so much more deterministic. Yes. And I wonder if this technique that you used at Google might at some point spread or or people will just put up with like AI kind of, you know, like, it might or might not work.