The Pragmatic Engineer
00:34:03 · Titus Winters
[00:34:03] S1: Yeah. No. I I think it makes perfect sense. Like, Let's try to learn our hard lessons only once, codify those rules somewhere deterministic, apply them equally and appropriately and fairly, and leave the interesting parts of code review. Make sure that the parts of code review that we're focusing on are only the things that humans need to be doing. Right? Which was true then. I don't think everyone invested in linting and static analysis that quite to that extent they should have. And then it's equally true now. Right? Like, we can't review everything, but, like, there are places that require... Like, this human attention is is well spent.
00:34:45 · Gergely Orosz
[00:34:45] S0: Titus just made the case for learning the hard lessons once, then writing them as rules, and saving the human attention for the things that really need such. That is pretty much the same way you want permissions for AI agents to work, which brings us to our seasonal sponsor WorkOS. WorkOS provides the auth infrastructure that b to b and AI native companies need to sell to enterprise customers with SSO SCIM RBAC and now authorization for AI agents. WorkOS Airlock is intent based access control. Most access control only asks who you are. Airlock also looks at what the agent is trying to do at the moment it tries to do it. Clear cut rules like no agent deletes cloud resources are run as plain code. For the fuzzier calls, an AI just evaluates the policy at runtime. You write those policies in plain English and Claude can pull the ones you already have out of Notion or Jira and import them for you. Your agents, whether that's Claude or Devin, connect to a single access point, and your MCP servers are behind this access point. Airlock is a new layer on top of your existing permissions which means that an agent can never get more access than the person who invoked the agent. When something risky comes up Airlock can escalate to a human for approval, say over Slack. It's the same idea Titus described: Write the rules once and spend human attention only on the calls that need it. You can request early access at workhorse.com/airlock. I also want to mention our presenting sponsor turbopuffer, a ridiculously scalable, fast and cheap hybrid search engine built on top of object storage. How often do you get to see a rewrite playing out in real time? With the TPUF team, we finally can. They are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. The new system called TurboPuffer v three is not yet in prod, but it already passes CI. And when the rewrite started in September, it was not nearly as performant as the current production version v two. The tfifth team had been perf maxing query plans until they are fast or faster than v two, doing all this in public all the way up to the v three production launch. And it's so cool to see their progress.
00:36:41 · Gergely Orosz
The new system went from being 10 to 100 times slower across most metrics to being several times faster on things like cold vector search, order by attitude, or hybrid search. The v three system is still slower in areas like full text search, but only one to two times slower versus a 100 times slower when this perf maxing started. And the t puff team is sharing exactly how they optimize things in their development logs with things like group count aggregation, compressed column encodings, batch scoring, and many more. A lot of these wins are basic foundational work right now, but the team will be sharing some very cool systems work over the next few weeks. But hang on, why is the team doing a full rewrite? It's because search is changing and so is turbopuffer. Tpuff started as a vector database but it is no longer a vector database. It has evolved into a high performance database for search. The V3 discards a vector primary index and all indexes will share the same columnar storage system. Every query will run on vectorized query engines which is something you don't usually see in search databases but which will result in both very fast search and just generally a very fast database. The TurboPuffer team is keeping a work log of their journey and you can follow along at turbopuffer.com/v3. It's a blast to see how it's being built. Speaking of hard lessons and documenting them, we talk about Software Engineering at Google, the book that you're quite well known for? How did the idea come? How did it come together? What was your motivation to start with? So this
00:38:03 · Titus Winters, Gergely Orosz
[00:38:03] S1: was a little bit the tail end of, hey. Maybe we'll ship the developer tools. Maybe we'll need to explain why Google's developer tools are the way they are, because otherwise, it'll be a little bit of like aliens came down, and they're trying to give you something that doesn't make any sense in your world. Part of the reason that we had this hubris to tackle that sort of task was because we had been doing this refactoring at scale everywhere. And when you run into everything that any engineer thought was a clever solution to their problem, it meant that the the my cadre of of weirdos have run into so many, oh, yep. I can see how you thought that was a good idea. Don't do that. Like, more than more than anyone else out there. [00:38:52] S0: Well, you... You're talking some of the most kind of battle scarred people in the industry just because you're working on these large scale systems for such a long period of time. Right. [00:39:03] S1: And it's not the production battle scars. It's the, like, testing and, like, local code design battle scars. Like, everyone else in the industry has, like, all of this production experience, and, like, my job ends at merge. Like, I know more about development and API design and test and keep the code base as an artifact working than most. But it does mean... I don't know. I'm kind of an outsider in that sense.
00:39:03 · Titus Winters, Gergely Orosz
[00:39:03] S1: And it's not the production battle scars. It's the, like, testing and, like, local code design battle scars. Like, everyone else in the industry has, like, all of this production experience, and, like, my job ends at merge. Like, I know more about development and API design and test and keep the code base as an artifact working than most. But it does mean... I don't know. I'm kind of an outsider in that sense. [00:39:30] S0: Well, interesting enough, I mean, the book is very, very thorough on how it gets to production, but it it it does not cover the operational part, which I guess is... That's the essence. But there is an SRE book, which... Well, then the the... This this book would be, like, twice as long. How did you pull the book together? Because you wrote some parts of it, Other folks inside of Google wrote different chapters of it.
00:39:30 · Gergely Orosz, Titus Winters
[00:39:30] S0: Well, interesting enough, I mean, the book is very, very thorough on how it gets to production, but it it it does not cover the operational part, which I guess is... That's the essence. But there is an SRE book, which... Well, then the the... This this book would be, like, twice as long. How did you pull the book together? Because you wrote some parts of it, Other folks inside of Google wrote different chapters of it. [00:39:53] S1: So it's very much in the spirit of the SRE books. Like, try to get the expert on... Or an expert on given topics to to contribute something useful. And then the three names on the front of the book, myself, Hyrum Wright, and Tom Manshreck. Hyrum and I were kind of the technical leadership for that, and Tom has... Is plenty technical, but has has been our tech writer friend for many, many years. Mhmm. So he handled a lot of the, like, editorial, structural, organizational stuff. But Hyrum and I had been the tag team of... Hyrum generated millions of edits and sent them to me for review, and I had the permissions to review almost everything in the code base. And so everyone knew us as those two guys that clean everything up. So we had those scars, and so we led that. And then the the trick was finding the experts to to write something high quality. Some of those were great from the start. A lot of that took a, you know, [00:40:54] S0: bit of ghostwriting Yeah.