The Pragmatic Engineer
00:44:12 · Gergely Orosz
[00:44:12] S2: It sounds to me that with load testing, with all I've heard from you is like, you know, step one is understand how your product will be used, what will likely top it over, which, because once, you know, you're not going to just randomly load test, you'll actually can test target it for that. And then Building something that simulates it good enough, knowing the resource constraints. So in this case, don't boot up a whole slack in a headless Chrome, but just mock, not mock, but build out that critical path that you care about. And if you combine this to your domain knowledge and your expertise, knowing how the product works and knowing how, you know, you knew how it used to fall over, you can now be reasonably confident that if it falls over, it will not be this way. It might be a different way, but I think it's nice that we're talking about the cost of testing, especially like we've talked about the cost of LLM, but like there is cost considerations and also repeatability, how lightweight it is, how much easier it is to control it and to control it. Yeah, it was really lightweight. It just did what it needed to do. We evolved the system whereby you could create routines to send to these fake users to run through over time. At first, it was just a very sort of silly JSON blob of like,
00:45:19 · Maude Lemaire
[00:45:19] S1: call this thing, then call this thing, then call this thing. I wrote a mini recorder tool where I could, via a slash command in Slack, say slash start recording, go through a whole flow in Slack. It would record every single API request I made across both to the normal web app. Backend and to our edge cache, it would record every single outgoing and incoming WebSocket message. And then I would sort of sort through it all to just the pieces that I wanted to simulate and then inject that into another tool that I wrote that turned that into a script that I could then send to the simulated clients. Because the other half of the battle was figuring out what are all of the APIs that get called when I do this one product flow, which is not a small number. And you wanted to be sure that you captured the entire thing. So that's where the recorder tool came in pretty handy. And we used that over and over and over again. And I would, again, to go back to the product thing, because I'm going to keep hammering the product thing. I think like the best product managers are mindful of like performance and reliability are important aspects of any given feature. And the best PMs know that and care about that and optimize for that. And I was trying the last like several years that I was there, I was always trying to push more tools into the hands of PMs and say like, hey, you want to add your use case to be automatically tested with Koi Pond with our load testing system? Here's the recorder tool. Go record the flow that you want us to test. Send us back the file. We'll plop it into the system and we'll run it at the frequency that you want it to run at. Because they knew, you know, oh, files do this weird thing when we like pop open a canvas and it like does this thing. And like, I don't know. I don't use canvases the way that like most of our users probably use them. You do. You're the PM. So like go record it and then bring it back to us and then we'll just shove it into the system we wrote.
00:47:35 · Gergely Orosz, Gergely Orosz, Maude Lemaire
[00:47:35] S2: Speaking of reliability, after Slack, you joined GitHub as a principal software engineer, and this was 2024. So this was at a time where GitHub Copilot was already very big. In fact, Cursor and other competitors were already as tools. [00:47:51] S0: The load on GitHub services already started to increase, and GitHub's reliability started to visibly become shakier, not as bad as it [00:47:59] S2: is these days. What did you observe inside of GitHub? Can you tell us the vibe when you joined? If you can give us a bit of an understanding and appreciation of what GitHub has been built on, this massive Ruby monolith. And when you joined, what was it like? [00:48:15] S1: It was a really interesting time, particularly because one of my friends from Rent the Runway, actually, we're going way back, had been at GitHub for many years at that point. And we'd sort of been keeping tabs on each other's careers. And she told me, Maude, we are going to run out of space. GitHub is still primarily on data centers, and it's increasingly less so over time. [00:48:41] S2: They used to have their own data centers. Now they're moving to Azure. This is public information they shared.
00:48:41 · Gergely Orosz, Maude Lemaire
[00:48:41] S2: They used to have their own data centers. Now they're moving to Azure. This is public information they shared. [00:48:47] S1: Yes. Yeah. The move to Azure is front and center, but still a significant or at least at the time when we were talking, a still significant amount of load was going through the data centers that like they had whole teams managing the metal. And she was like, we're going to run out of space. The space is running out. And we either well, we both have to figure out how to move to cloud. And we also have to figure out how to do less because we're doing a bunch of dumb stuff, as everyone does, that we shouldn't be doing. And we can make the system a lot more efficient with what we have to give bias more runway. I was like, oh, this sounds like a very interesting problem. Would love to work on this problem. But I was also seven months pregnant at the time. And so what that ended up being was I showed up. I was seven months pregnant, which is actually kind of, if you can swing it, a really great time way to join a company because you are you sit there and you're like, cool, I will just drop a bunch of opinions like, you know, nothing very strongly held, but I will just drop a bunch of opinions on people. I will own nothing for the next two months and then I'm going to go have a baby and then I'll come back. [00:50:01] S2: So what happened?
00:50:01 · Gergely Orosz, Maude Lemaire
[00:50:01] S2: So what happened? [00:50:02] S1: Let's see what happened. And in my first couple of weeks, I was sort of floating around some spaces. And having done a bunch of database migrations at Slack and some of the stuff that we didn't talk about, but was moving from this team-sharded, tenant-based model to a more dynamically-sharded model with Vitess, where it turns out, no, a team was too big. [00:50:27] S2: So you mentioned Vitess. What is Vitess? [00:50:30] S1: Vitess is essentially a wrapper on top of MySQL that lets you shard more dynamically. [00:50:36] S2: And so you have a- So instead of just having MySQL run on one database, and when you outgrow it, good luck. You can re-shard, redistribute your data a lot more seamlessly than you would be able to otherwise. And you can choose different sharding keys more easily as well.
00:51:09 · Maude Lemaire
[00:51:09] S1: Before, it's like everything was started by team. There's still data that is started by team, where team is a logical unit that makes the most sense for fetching that data. But actually, it turns out that the smallest, most logical unit was channel. And so we ended up resharding a bunch of our data to be fetchable by channel ID rather than by team ID. And then that allows for much better horizontal sharding structure as well. And the idea being that there is no channel that will ever be big enough, maybe one day it will be, the bots keep posting in the channels maybe, that will have enough data associated with it that it can never run on a single ultra-large database, right? It's a much better unit from a scaling perspective. So Vitess has had some of its nascency embedded in GitHub, actually. And it was really interesting because when I was interviewing at GitHub, I spoke with a bunch of the principal engineers that I would later end up working with. And they were sort of trying to figure out, OK, we have Vitess, but I don't think we're actually using it to its potential to where we should be using it. And so they were asking me a bunch of questions about like, are Vitess set up and what we were doing? Caveating this with, I did not ever operate Vitess or deal with the internals. I was more of a, I'm a product platform person. And so I just like find that like, no, we should not be starting this thing by this thing anymore. Let's migrate it. And so that was my perspective on it, which was also mostly their perspective on it as well. And I came in and I started poking at the Vitess setup and I was like, what are we doing? What are we doing here? I was used to thousands of shards per sharding key. So say four channels, we have a bajillion. Because there's a bajillion channels and you want to sort of spread load evenly throughout those. And when I arrived, I was looking at the Vitess setup and there was, I mean, I don't know if this is still the case, but there were 16 shards for all of pull requests and issues.