The Pragmatic Engineer
00:38:03 · Titus Winters, Gergely Orosz
[00:38:03] S1: was a little bit the tail end of, hey. Maybe we'll ship the developer tools. Maybe we'll need to explain why Google's developer tools are the way they are, because otherwise, it'll be a little bit of like aliens came down, and they're trying to give you something that doesn't make any sense in your world. Part of the reason that we had this hubris to tackle that sort of task was because we had been doing this refactoring at scale everywhere. And when you run into everything that any engineer thought was a clever solution to their problem, it meant that the the my cadre of of weirdos have run into so many, oh, yep. I can see how you thought that was a good idea. Don't do that. Like, more than more than anyone else out there. [00:38:52] S0: Well, you... You're talking some of the most kind of battle scarred people in the industry just because you're working on these large scale systems for such a long period of time. Right. [00:39:03] S1: And it's not the production battle scars. It's the, like, testing and, like, local code design battle scars. Like, everyone else in the industry has, like, all of this production experience, and, like, my job ends at merge. Like, I know more about development and API design and test and keep the code base as an artifact working than most. But it does mean... I don't know. I'm kind of an outsider in that sense.
00:39:03 · Titus Winters, Gergely Orosz
[00:39:03] S1: And it's not the production battle scars. It's the, like, testing and, like, local code design battle scars. Like, everyone else in the industry has, like, all of this production experience, and, like, my job ends at merge. Like, I know more about development and API design and test and keep the code base as an artifact working than most. But it does mean... I don't know. I'm kind of an outsider in that sense. [00:39:30] S0: Well, interesting enough, I mean, the book is very, very thorough on how it gets to production, but it it it does not cover the operational part, which I guess is... That's the essence. But there is an SRE book, which... Well, then the the... This this book would be, like, twice as long. How did you pull the book together? Because you wrote some parts of it, Other folks inside of Google wrote different chapters of it.
00:39:30 · Gergely Orosz, Titus Winters
[00:39:30] S0: Well, interesting enough, I mean, the book is very, very thorough on how it gets to production, but it it it does not cover the operational part, which I guess is... That's the essence. But there is an SRE book, which... Well, then the the... This this book would be, like, twice as long. How did you pull the book together? Because you wrote some parts of it, Other folks inside of Google wrote different chapters of it. [00:39:53] S1: So it's very much in the spirit of the SRE books. Like, try to get the expert on... Or an expert on given topics to to contribute something useful. And then the three names on the front of the book, myself, Hyrum Wright, and Tom Manshreck. Hyrum and I were kind of the technical leadership for that, and Tom has... Is plenty technical, but has has been our tech writer friend for many, many years. Mhmm. So he handled a lot of the, like, editorial, structural, organizational stuff. But Hyrum and I had been the tag team of... Hyrum generated millions of edits and sent them to me for review, and I had the permissions to review almost everything in the code base. And so everyone knew us as those two guys that clean everything up. So we had those scars, and so we led that. And then the the trick was finding the experts to to write something high quality. Some of those were great from the start. A lot of that took a, you know, [00:40:54] S0: bit of ghostwriting Yeah.
00:40:54 · Gergely Orosz, Titus Winters
[00:40:54] S0: bit of ghostwriting Yeah. [00:40:56] S1: The whole task was actually almost two years. Mhmm. Certainly not full time. Like, we were at most probably putting in quarter to half time for most of that time. But, yeah. Like, we wrangled together way too much text from way too many people, and I don't think I realized just how much we overshot until I saw a printed copy of that book. [00:41:19] S0: I was [00:41:19] S1: like, wow. That's huge. Yeah. [00:41:22] S0: And and even today, I I do recommend the book because it gives a really good understanding of how the Google at the time worked. Obviously, there's been evolving since but how a company going from zero to building important production systems came up with a bunch stuff that might not make as much sense. And you talk about things like like mono repo, like the the stringent code review process, how there's a readability process at Google, and maybe we can talk about that. As as part... Google is the only company I know where for code review back in the day, they didn't just have code review like an engineer who owns this part of the system and understands, reviews it, says it's okay, but Google had a readability review. What is readability review?
00:42:04 · Titus Winters
[00:42:04] S1: So for any change that's going in, someone that is involved, either the author or the reviewer or author or a reviewer, has to have two different permission bits. First is that ownership thing. So someone has to be in the list of people that is authorized to change these files. And the second is someone has to actually be certified in we believe you understand how to write idiomatic code in this code base in that language. And this is a step towards trying to make sure that everything is relatively consistent and that there's common patterns and that you're not reinventing the wheel because the strategic value of consistency is incredible. And so the readability process is someone has to be certified for that. How do you get certified for that? And initially, that came as employee number three sat down with every new employee and walked them through their first change. That stopped scaling, so they got a bunch of other people to try to proxy Bing Craig, and they walk you through your first big substantive change. And over time, it got to the point where most change was not like, I am writing a new component from scratch. It's an existing product. Like, you're making little edits here and there, you're adding a feature or whatever. So finding something that was meaty enough to demonstrate in one go that you knew everything was kind of hard, and the whole thing felt a little bit more like a hazing ritual. And so over the years, I think the... I think people in Java started it. It moved from that sort of one shot to being explicitly kind of a mentoring program of you do your normal work, and after your team has signed off on it, then we will round robin assign someone from the pool of Java mentors to go through and see, is there anything substantively interesting worth commenting on here to of up the quality of the code and make sure that it matches the rest of the the norms. And you might notice, that means if you're on a team that already has great standards, there's probably very little to comment on. Right? And you'll probably graduate pretty quick. Yeah. And if you're not, and you choose not to learn any of the lessons, you will probably graduate very slowly, if at all. And so the whole thing kind of grew into that.
00:44:23 · Titus Winters, Gergely Orosz
And we standard that sort of mentorship approach into all the other readability languages, and that's been the norm for a bunch of years. I haven't checked up on it since, you know, for at least a couple years. [00:44:36] S0: But but it's very interesting because it it feels to me like this turns code review into into just education in the depth of a programming language. Yep. Was this by design, or was it just a byproduct? [00:44:47] S1: I think it's absolutely by design. Certainly, at the parts of it where I was pulling any of the strings was definitely by design. But, yeah, when I left Google, you know, almost three years ago, readability was kind... Like, I was kind of the overarching owner for the readability program. I would argue that might be the largest educational program in tech or maybe in the world because there were tens of thousands of people participating in that, like, at any given moment. [00:45:14] S0: And and you just participated by by joining Google, and then you had no choice. It wasn't, like, voluntary. [00:45:20] S1: Right. And so, certainly, some people were super pissy about it.