The Pragmatic Engineer
00:00:00 · Gergely Orosz
[00:00:00] S0: How do you scale a product that millions of people use daily when the architecture underneath it is starting to break? Today's guest did this both at Slack and GitHub. Maude Lemaire is a principal engineer and the author of the book Refactoring at Scale. She spent eight years at Slack, worked on database scaling at GitHub, and improved time-to-verse token at Cursor. Today we cover... Load testing with 500,000 WebSocket connections, a job where using headless Chrome instances were too resource intensive and too expensive. Why GitHub had only 16 database charts for all pull requests and issues, and why scatter queries were a massive headache inside a large Rails monolith like GitHub's. Why platform and product engineers should rotate between teams, and more. If you want to understand what scaling up looks like inside of products you probably use every day, this episode is for you. This episode is presented by turbopuffer, Vector and Full-Tech Search built on object storage. It's fast, cheap, and extremely scalable. This episode is brought to you by Antithesis, the platform to release with confidence. In today's episode, we'll talk about rewriting massive production systems as they scale up, which is a kind of engineering challenge that's rebuilding a 747 mid-air. But why not just use CodeReview when using AI coding agents? It's because CodeReview is an increasingly naive and ineffective approach to do so, which is why Antithesis goes well beyond this practice. Antithesis runs your whole system in a hostile simulation. This simulation includes both targeted testing and fast testing. By running this simulation, it finds every bug before your users do. And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it much easier to fix issues. The first thing I thought when I heard about antithesis is that automated bug discovery and fully deterministic testing sounds like science fiction, but actually it's hardcore engineering under the hood. Jane Street, Supa-based Rapid Railway and Datadog ship agent-written code with full confidence because they know it's been verified by antithesis.
00:01:52 · Gergely Orosz, Maude Lemaire, Gergely Orosz
And to give a sense of how Antithesis help with large production systems, Datadog use Antithesis to verify their new architecture when rewriting the Datadog event platform, a system that collects more than 100 trillion events per day. Today, Antithesis is one of many correcting strategies being used as part of Datadog's harness for their coding agents. To see more case studies and details, head to antisysys.com slash pragmatic. Maude, welcome to the podcast. [00:02:16] S1: Thank you for having me. [00:02:17] S2: It is so nice to have you here. [00:02:19] S1: This is a bucket list item, very much so. [00:02:22] S2: Well, this is the first in the podcast. Usually I start with a question. I'm not going to start with a question. I will show you a photo and I will ask, what am I seeing here? I'm seeing a baby who looks like how many months? [00:02:37] S1: Maybe about a year old. Maybe a little before a year old. [00:02:41] S2: Sitting in front of a computer. We're also putting this on the screen if you're watching it on video. Same haircut. Same haircut. So my question is, were you born with computers? [00:02:54] S1: Um, very nearly. There are pictures of me much younger than that, actually, maybe like a couple months old. I truly like I'm barely keeping my head up where my dad has me sitting in his lap, like resting against him. And he's I don't know what he's doing. He's probably like grading some kind of assignment or because he was teaching. He taught computer science for 17 years before we moved to the U.S. [00:03:19] S2: Mm hmm.
00:03:19 · Gergely Orosz, Maude Lemaire
[00:03:19] S2: Mm hmm. [00:03:19] S1: And yeah, there's a bunch of pictures of me like holding ASCII tables, which now in hindsight is hilarious. But I ended up doing this sort of by accident. You'd think that all of the variables and the environment would have led to like a very logical sort of linear career trajectory. And that is not at all. So anyway, he had to think about that like [00:03:45] S2: eons before. [00:03:47] S1: But I thought it sounded really boring. I was like, no, no, no. And I really liked my bio classes. I thought I was going to become a genetics researcher. That was my plan. And my parents always told me to chill the heck out because I had plans. I had, when I started university, I started in microbiology. I signed up for molecular biology, organic chemistry, all the things. I had already accrued enough AP credits for all the Americans listening that I was technically like a second year college student rather than a first year. Oh, yeah. [00:04:19] S2: Oh, wow. [00:04:20] S1: Rather than a first year college student. [00:04:22] S2: So you wanted to speed run college. [00:04:25] S1: Oh, yeah. I was ready. [00:04:26] S2: Skip the fun years. [00:04:27] S1: Yeah. [00:04:28] S2: Get straight to work.
00:04:28 · Gergely Orosz, Maude Lemaire
[00:04:28] S2: Get straight to work. [00:04:29] S1: Which I kept doing that. I had like a 19 credit semester at some point. I was like, why did I do that? That was such a poor decision. But I mean, I had fun doing it. So I guess, you know, my brand of fun is, I guess, different from other people's brand of fun. And yeah, I was going to do that. And then like three weeks in, there were a couple of things that really bothered me. One, it's not because you're good at something that you like doing it. And I think I conflated the two for a really long time. I mean, to be fair, when you're 18, what do you know? But I conflated being really good at memorization with liking memorization. And I don't know if you've ever taken a molecular biology class, but it is a significant amount of memorization. And you're learning the Krebs cycle. So I was going to switch into organic chemistry, into chemical engineering. And to do that, that's a different faculty at McGill. I mean, it's a very big school. There's faculty of science. I was in the faculty of science. There's faculty of engineering. And I was looking at all of the requirements for first-year engineering students. And there was a programming class. There was an intro to computer science. So I swapped out one of my bio courses the next semester with the intro to computer science class. [00:05:37] S2: Computer science, yep.
00:05:37 · Gergely Orosz, Maude Lemaire
[00:05:37] S2: Computer science, yep. [00:05:38] S1: And I told my mom I swapped it out for a computer science class. Please do not tell dad. Please do not tell him that I swapped it out for a computer science. Like, I don't. I'm scared I'm not going to like it. And then he's going to be sad because he loves [00:05:51] S2: this so much. [00:05:52] S1: This is like true love for him. And she was like, sure. OK, I'm pretty sure she told him like two minutes later because that's what my mom does. I love her. But like, man. I think she knew that it was not going to be, you know, a wreck. And sure enough, eventually, like I tell them I'm taking this class and three weeks and I'm like, oh, I really like this. Complete opposite from the molecular biology class where I was like, I really liked it. It was like, oh, with the power, I'm sure you've had this feeling. It's like, I only have a computer and I type into it and then it does things. I feel like a god. [00:06:33] S2: And then your first job, you did internships, but your first job, which followed the internship, was at Rent the Runway. [00:06:39] S1: It was, yeah. [00:06:40] S2: It's a pretty well-known company. Thanks to anyone who's entering management will probably know of the book, The Manager's Path by Camille Fournier, and she was at Rent the Runway.
00:06:40 · Gergely Orosz, Maude Lemaire
[00:06:40] S2: It's a pretty well-known company. Thanks to anyone who's entering management will probably know of the book, The Manager's Path by Camille Fournier, and she was at Rent the Runway. [00:06:50] S1: She was, yeah. And actually, that was a big part of why I wanted to go back after my internship there was because I was You didn't see very many women in those CTO type positions at the time. Also, like, I mean, I, my mom very nearly became a fashion designer. And so we grew up watching Project Runway from the very beginning. That was like the one thing I was allowed to like stay up late to watch was Project Runway. And so it combined two things that I loved, combined, you know, programming, computer science. I was mostly doing front end stuff at the time and in fashion, I could just like Rent stuff for dirt cheap and like walk around in a Monique Olivier like ball gown if I wanted to. It was it was so cool. I think the there were so many women on the team in general, I think partly because of the space that the company was in. But I also just for folks who've had the opportunity to be in a room with Camille, she just commands this presence. And I had so much respect for that. And I mean, even if I was many levels removed from her, I was like, I want to learn from this woman. I really admire this presence that she has and this confidence. And so it was an easy sell. It was a very easy sell to go back. But then she left a couple of months [00:08:12] S2: after I came [00:08:13] S1: back. But she was still instrumental in helping me land a job at Slack a year later.