The Pragmatic Engineer
00:18:56 · Titus Winters, Gergely Orosz
[00:18:56] S1: Yeah. The other part of that is, like, the only time that a human is gonna look at your code if you're an undergrad is to grade you, is to judge you. Right? Whereas, when you, like, become a more experienced engineer, right, like, you're asking for feedback regularly, but it's just a check-in of, are we going in the right direction? Does this make sense to you? And the more of that feedback that you can get that isn't judgment, the more effective you will be. And like, oh, I hate that we've given all of the new grads and all the junior engineers exactly the wrong instinct. [00:19:32] S0: So going back to to Google, what is Abseil? How how do you pronounce it?
00:19:32 · Gergely Orosz, Titus Winters
[00:19:32] S0: So going back to to Google, what is Abseil? How how do you pronounce it? [00:19:39] S1: Abseil. Abseil. Yep. So Abseil was starting about ten years ago. Shortly after the initial pivot to cloud, there was a motion to, hey. We have all of these amazing developer tools in addition to all this amazing production stuff. Maybe we should try to externalize the developer tools as well. And at the same time, there was also a little bit of Google shifting into Alphabet. So for both the developer tooling aspects at one level of, like, how do we share developer tools amongst different companies within the Alphabet umbrella, and how do we share high value internal interfaces and APIs and tech components? How do we share those when it's not one single corporate... One single company? And also not one mono repo. Right? Increasingly. Yeah. Exactly. And so there was a bunch of work that was kind of the normal, like, okay, what does it take to make these developer tools, like, multi tenant and, like, you know, slightly less completely tied into that ecosystem? [00:20:43] S0: And and and just multi tenant for for those who are who are not as familiar with, what does multi tenant mean inside of Google? [00:20:51] S1: Well, so this would have been multitenant of other companies within the Alphabet umbrella, and then maybe extend that out to cloud customers. [00:20:58] S0: Oh oh, got it. Yeah. So like a like a way more and the Google could theoretically use it, something That like
00:20:58 · Gergely Orosz, Titus Winters
[00:20:58] S0: Oh oh, got it. Yeah. So like a like a way more and the Google could theoretically use it, something That like [00:21:03] S1: became not the hottest topic, and they they kinda dropped that idea after a year or two. But the low level components, like, if you wanted to share these things that were written internally in the mono repo, what would it take to do that effectively? And Abseil was basically, here is the layer of common c plus plus idiom. This is the vernacular that everything at Google is written out of. Let's open source that so that there's a common substrate for all Google tech. At the same time, the question was, some of this vernacular is gross and tech deady. Should we clean that up before we commit to it forever? And the answer was yes, to their great credit. So the Abseil open source project is the public projection of the stuff that got cleaned up. Most of the work was... That was... I I would defend pretty strongly. That was probably the largest refactoring task in human history. No way. Yeah. We had, like, ten ten or 12 people working full time with world class tools that no one else understands yet, just cleaning up and making sure that idiomatic code externally would be compatible with idiomatic code internally. [00:22:21] S0: And obviously, when you were talking about cleaning up, you're talking about cleaning up utilities and components that are in the middle of some of the highest used services in the world. Google Search Yep. Yep. Gmail, Maps, you name it. [00:22:34] S1: All of But
00:22:34 · Titus Winters, Gergely Orosz
[00:22:34] S1: All of But [00:22:35] S0: like, if if if you would break something, you would probably see at some point an outage in one [00:22:39] S1: of those services. It was... At the time, I think it was 200 or 250,000,000 lines of c plus plus code with [00:22:45] S0: 250,000,000 lines of c [00:22:47] S1: First plus plus party. Not counting external dependencies. And Google had that kind of I think that was the public number at the time. 10,000 engineers probably at the time making edits to that code base in a month. And so our job was... We joked about... That's a New York reference. We joked about it as our job is the logical equivalent of Google has a new technology for the skyscraper foundations. We're replacing the foundation under the Empire State Building. We're not allowed to close the building or block traffic, and we discovered nuclear waste in the basement. It's okay, guys. We got this. It was magically. It was a great team. I just wanna put the
00:23:28 · Gergely Orosz, Titus Winters
[00:23:28] S0: two fifty million lines of code in perspective because today, with AI, AI can generate, you know, like, we have crazy numbers. It it can generate tens of thousands of lines of code in an hour, etcetera, and and you can get to millions if if you want to, honestly, just an agent can do it. However, pre AI, Uber's mobile application was about a million lines of code on iOS and on Android and that was each written by 200 engineers for for years. BUN runtime at Anthropic just rewrote something like, I think it was like 500 or 900,000 lines of code and that was written by a small team. I find it rare to find examples of a project that has more than pre AI, more than let's say 3,000,000 lines of code and that will typically be hundreds of engineers writing it for for several years. So in the in the the reason I was like 250,000,000, that is about a 100 times more than I've heard for code base. And we're talking about, like, clearly, this is written for production. You're not, like, just goofing around and pushing some scaffolding here and there and and right. And and that when you said first party, it's easy to to have a lot lot of code when you take on an open source component, but no. [00:24:36] S1: Nope. Nope. That was all Google or authored. Yep. And so we have some insight into how things grow. [00:24:46] S0: Okay. So the largest refactoring history at that point in human history might have... You're not extricating. [00:24:51] S1: I don't think so. I mean, I can't prove it, but I I would be surprised.
00:24:51 · Titus Winters, Gergely Orosz
[00:24:51] S1: I don't think so. I mean, I can't prove it, but I I would be surprised. [00:24:56] S0: So how... Okay. When you have this type... You were in the middle of this 10 to 12 people. You have this massive massive task that seems giant and almost... How do you go about it? How do you say, like, how do you break it down? How do you plan for it? How do you speed it up? Because you clearly... If you just did it by hand, this would have not fully worked. Right. So one of the