The Pragmatic Engineer
00:39:30 · Gergely Orosz, Titus Winters
[00:39:30] S0: Well, interesting enough, I mean, the book is very, very thorough on how it gets to production, but it it it does not cover the operational part, which I guess is... That's the essence. But there is an SRE book, which... Well, then the the... This this book would be, like, twice as long. How did you pull the book together? Because you wrote some parts of it, Other folks inside of Google wrote different chapters of it. [00:39:53] S1: So it's very much in the spirit of the SRE books. Like, try to get the expert on... Or an expert on given topics to to contribute something useful. And then the three names on the front of the book, myself, Hyrum Wright, and Tom Manshreck. Hyrum and I were kind of the technical leadership for that, and Tom has... Is plenty technical, but has has been our tech writer friend for many, many years. Mhmm. So he handled a lot of the, like, editorial, structural, organizational stuff. But Hyrum and I had been the tag team of... Hyrum generated millions of edits and sent them to me for review, and I had the permissions to review almost everything in the code base. And so everyone knew us as those two guys that clean everything up. So we had those scars, and so we led that. And then the the trick was finding the experts to to write something high quality. Some of those were great from the start. A lot of that took a, you know, [00:40:54] S0: bit of ghostwriting Yeah.
00:40:54 · Gergely Orosz, Titus Winters
[00:40:54] S0: bit of ghostwriting Yeah. [00:40:56] S1: The whole task was actually almost two years. Mhmm. Certainly not full time. Like, we were at most probably putting in quarter to half time for most of that time. But, yeah. Like, we wrangled together way too much text from way too many people, and I don't think I realized just how much we overshot until I saw a printed copy of that book. [00:41:19] S0: I was [00:41:19] S1: like, wow. That's huge. Yeah. [00:41:22] S0: And and even today, I I do recommend the book because it gives a really good understanding of how the Google at the time worked. Obviously, there's been evolving since but how a company going from zero to building important production systems came up with a bunch stuff that might not make as much sense. And you talk about things like like mono repo, like the the stringent code review process, how there's a readability process at Google, and maybe we can talk about that. As as part... Google is the only company I know where for code review back in the day, they didn't just have code review like an engineer who owns this part of the system and understands, reviews it, says it's okay, but Google had a readability review. What is readability review?
00:42:04 · Titus Winters
[00:42:04] S1: So for any change that's going in, someone that is involved, either the author or the reviewer or author or a reviewer, has to have two different permission bits. First is that ownership thing. So someone has to be in the list of people that is authorized to change these files. And the second is someone has to actually be certified in we believe you understand how to write idiomatic code in this code base in that language. And this is a step towards trying to make sure that everything is relatively consistent and that there's common patterns and that you're not reinventing the wheel because the strategic value of consistency is incredible. And so the readability process is someone has to be certified for that. How do you get certified for that? And initially, that came as employee number three sat down with every new employee and walked them through their first change. That stopped scaling, so they got a bunch of other people to try to proxy Bing Craig, and they walk you through your first big substantive change. And over time, it got to the point where most change was not like, I am writing a new component from scratch. It's an existing product. Like, you're making little edits here and there, you're adding a feature or whatever. So finding something that was meaty enough to demonstrate in one go that you knew everything was kind of hard, and the whole thing felt a little bit more like a hazing ritual. And so over the years, I think the... I think people in Java started it. It moved from that sort of one shot to being explicitly kind of a mentoring program of you do your normal work, and after your team has signed off on it, then we will round robin assign someone from the pool of Java mentors to go through and see, is there anything substantively interesting worth commenting on here to of up the quality of the code and make sure that it matches the rest of the the norms. And you might notice, that means if you're on a team that already has great standards, there's probably very little to comment on. Right? And you'll probably graduate pretty quick. Yeah. And if you're not, and you choose not to learn any of the lessons, you will probably graduate very slowly, if at all. And so the whole thing kind of grew into that.
00:44:23 · Titus Winters, Gergely Orosz
And we standard that sort of mentorship approach into all the other readability languages, and that's been the norm for a bunch of years. I haven't checked up on it since, you know, for at least a couple years. [00:44:36] S0: But but it's very interesting because it it feels to me like this turns code review into into just education in the depth of a programming language. Yep. Was this by design, or was it just a byproduct? [00:44:47] S1: I think it's absolutely by design. Certainly, at the parts of it where I was pulling any of the strings was definitely by design. But, yeah, when I left Google, you know, almost three years ago, readability was kind... Like, I was kind of the overarching owner for the readability program. I would argue that might be the largest educational program in tech or maybe in the world because there were tens of thousands of people participating in that, like, at any given moment. [00:45:14] S0: And and you just participated by by joining Google, and then you had no choice. It wasn't, like, voluntary. [00:45:20] S1: Right. And so, certainly, some people were super pissy about it.
00:45:20 · Titus Winters, Gergely Orosz
[00:45:20] S1: Right. And so, certainly, some people were super pissy about it. [00:45:23] S0: But I guess there was also a goal where it wasn't just about, like, okay. So step one is you need to address the comments that's a readability expert puts on your. Step two, you probably say like, I'd like to get fewer comments on them if you're... If you have any sense of pride because it's kind of embarrassing. And then step three is like, you know what? Okay. I now get zero comments. I actually wanna become a readability expert Mhmm. Myself, and then that's a different level. And then step four is now you are giving feedback. And and I guess through that feedback, maybe you meet people, maybe you start to form some, I don't know, mentorship, friendships. Who knows? [00:45:57] S1: Yeah. And you see other parts of the code base. You see stuff that other people are doing. You learn about other tech that might be happening. Like, it was a huge, like, subtle, below the waterline, like, information sharing program. [00:46:09] S0: In the book, you write about the importance of a team, and I I I'll quote you. In the realm of programming, lone craftspeople are extremely rare, and even when they do exist, they don't perform superhuman achievements vacuum. Their world changing accomplishment is almost always the result of a spark of inspiration followed by a heroic team effort. A great team makes brilliant use of its superstars, but the whole is always greater than the sum of the parts. But creating a superstar team is fiendishly difficult.
00:46:09 · Gergely Orosz, Titus Winters
[00:46:09] S0: In the book, you write about the importance of a team, and I I I'll quote you. In the realm of programming, lone craftspeople are extremely rare, and even when they do exist, they don't perform superhuman achievements vacuum. Their world changing accomplishment is almost always the result of a spark of inspiration followed by a heroic team effort. A great team makes brilliant use of its superstars, but the whole is always greater than the sum of the parts. But creating a superstar team is fiendishly difficult. [00:46:37] S1: Yeah. Don't think I wrote that chapter. [00:46:40] S0: It's in the book that you are a co author of. So I wanted to reflect on this especially with AI because now I'm starting to hear a lot more people say, yeah, in the past, we talked about a 10x engineer up to up to one... Up to 22, 23. [00:46:57] S1: We started on that. [00:46:58] S0: But I am now starting to hear people say like, oh, there's now the 100x engineer, which is this engineer who is a great engineer, uses AI amazingly well, and we are also hearing stories of individuals doing heroic efforts by themselves and with a bunch of AI agents. So I just wanted to ask you how the full... Concept of the team holds up as we have more AI tools and the concept of a standout team.