The Pragmatic Engineer
00:20:58 · Gergely Orosz, Titus Winters
[00:20:58] S0: Oh oh, got it. Yeah. So like a like a way more and the Google could theoretically use it, something That like [00:21:03] S1: became not the hottest topic, and they they kinda dropped that idea after a year or two. But the low level components, like, if you wanted to share these things that were written internally in the mono repo, what would it take to do that effectively? And Abseil was basically, here is the layer of common c plus plus idiom. This is the vernacular that everything at Google is written out of. Let's open source that so that there's a common substrate for all Google tech. At the same time, the question was, some of this vernacular is gross and tech deady. Should we clean that up before we commit to it forever? And the answer was yes, to their great credit. So the Abseil open source project is the public projection of the stuff that got cleaned up. Most of the work was... That was... I I would defend pretty strongly. That was probably the largest refactoring task in human history. No way. Yeah. We had, like, ten ten or 12 people working full time with world class tools that no one else understands yet, just cleaning up and making sure that idiomatic code externally would be compatible with idiomatic code internally. [00:22:21] S0: And obviously, when you were talking about cleaning up, you're talking about cleaning up utilities and components that are in the middle of some of the highest used services in the world. Google Search Yep. Yep. Gmail, Maps, you name it. [00:22:34] S1: All of But
00:22:34 · Titus Winters, Gergely Orosz
[00:22:34] S1: All of But [00:22:35] S0: like, if if if you would break something, you would probably see at some point an outage in one [00:22:39] S1: of those services. It was... At the time, I think it was 200 or 250,000,000 lines of c plus plus code with [00:22:45] S0: 250,000,000 lines of c [00:22:47] S1: First plus plus party. Not counting external dependencies. And Google had that kind of I think that was the public number at the time. 10,000 engineers probably at the time making edits to that code base in a month. And so our job was... We joked about... That's a New York reference. We joked about it as our job is the logical equivalent of Google has a new technology for the skyscraper foundations. We're replacing the foundation under the Empire State Building. We're not allowed to close the building or block traffic, and we discovered nuclear waste in the basement. It's okay, guys. We got this. It was magically. It was a great team. I just wanna put the
00:23:28 · Gergely Orosz, Titus Winters
[00:23:28] S0: two fifty million lines of code in perspective because today, with AI, AI can generate, you know, like, we have crazy numbers. It it can generate tens of thousands of lines of code in an hour, etcetera, and and you can get to millions if if you want to, honestly, just an agent can do it. However, pre AI, Uber's mobile application was about a million lines of code on iOS and on Android and that was each written by 200 engineers for for years. BUN runtime at Anthropic just rewrote something like, I think it was like 500 or 900,000 lines of code and that was written by a small team. I find it rare to find examples of a project that has more than pre AI, more than let's say 3,000,000 lines of code and that will typically be hundreds of engineers writing it for for several years. So in the in the the reason I was like 250,000,000, that is about a 100 times more than I've heard for code base. And we're talking about, like, clearly, this is written for production. You're not, like, just goofing around and pushing some scaffolding here and there and and right. And and that when you said first party, it's easy to to have a lot lot of code when you take on an open source component, but no. [00:24:36] S1: Nope. Nope. That was all Google or authored. Yep. And so we have some insight into how things grow. [00:24:46] S0: Okay. So the largest refactoring history at that point in human history might have... You're not extricating. [00:24:51] S1: I don't think so. I mean, I can't prove it, but I I would be surprised.
00:24:51 · Titus Winters, Gergely Orosz
[00:24:51] S1: I don't think so. I mean, I can't prove it, but I I would be surprised. [00:24:56] S0: So how... Okay. When you have this type... You were in the middle of this 10 to 12 people. You have this massive massive task that seems giant and almost... How do you go about it? How do you say, like, how do you break it down? How do you plan for it? How do you speed it up? Because you clearly... If you just did it by hand, this would have not fully worked. Right. So one of the
00:25:17 · Titus Winters
[00:25:17] S1: things that had enabled any of this and that came out of having been on that, the low level c plus plus infrastructure team to start with, was, okay, if we own the interfaces that are kind of the nuts and bolts that everything at Google is written out of, like way below the database level, just like caches, containers, screen manipulation routines, command line flags, like that sort of thing. If we own those things, maybe it's valuable to make them better. But how do you do that when there's 50,000 call sites or a 100,000 edits that need to be made to change that API? And so luckily, was friends and org chart cousins eventually with the compiler team, especially Clang LLVM people. And so they started doing stuff in the AST transformation sort of state. AST. Abstract syntax tree. So like you take a parse tree. And, you know, it's Google, like MapReduce. So if you do a MapReduce over the code base where the mapper parses all of the translation units of your C plus plus like all of the build targets, and produces... Here is the parse tree. Then you can match the parse tree for specific... It's not text match. It's I guarantee you are calling this particular function in this particular context, and then you can apply some code to rewrite based on that. And so we had compiler assisted semi automated refactoring to generate all of the text edits. Turns out one of the things that was really hard at first was we could generate the edit, but we couldn't get the white space in the right places. No way. Yeah. Seriously. So Clang format exists because I wished for it. This is like the auto formatter for many, like, c plus plus sorts of languages at this point. But it's like, I literally cannot accomplish this refactoring until someone figures out a way to, like, hack together a auto white space for c plus plus And so you you do that. You generate one, you know, big edit. It is too large to fit in the code base because for any given code base, there's kind of a natural background level of how much change is happening. And if you're trying to edit 10,000 files, 20,000 files, it is likely the case that you will never be able to test and resync before something has gone out of sync. Yeah. Right? Someone else edited something. Yeah. Exactly. Merge conflicts, like parallel changes, that kind of stuff. Yeah. Exactly.
00:27:40 · Titus Winters, Gergely Orosz
You know, you do not want at a, you know, 10,000 plus person company to be like, everyone stop touching the repo for an hour. Let me do this thing. Like, that's not gonna scale. Right? And so the tech is you break all of... Like, you figure out how to do your refactoring so that the old API and the new one can coexist in spelling. Yep. And then you paralyze, like, just cut by directory structure. Like, alright. I'll send you this batch of updates, this batch of updates, and you wind up, you know, going like that. And then once everything is clean, then there's a race to make sure no one copy and pastes the old thing. And then you delete the old thing, the new thing is the default, you move on with life. You just do that a few 100 times, and your code base is in much better shape. [00:28:28] S0: Yeah. Wow. And and then you're you're doing this all all the while while other engineers, they're just doing their thing. They're using these things they don't even know about. They don't wanna know about you. I mean Exactly. They might be aware of you, but they shouldn't have to change their Yep. Just, as you said, slow down or Yep. But then we