The Pragmatic Engineer
00:40:44 · Maude Lemaire, Gergely Orosz
[00:40:44] S1: It is not a good time. I mean, the problem with the headless Chrome instance, right, is just running Slack and 200 Chrome tabs and whatever on your laptop is kind of a time. And now you're running LLMs on top of it, but it's an experience in really pushing your laptop to the limit. And now you're running per pod, attempting to run 10,000 headless Chrome instances of Slack, which is already Very memory and CPU, mostly memory intensive application. The amount of money we were spending in these EC2 instances... [00:41:19] S2: The problem seems to be that per node, per VM node, depending on the size, you will be able to run, you know, you'll know, like, I don't know, like 10, 20, 50, 100. I don't know. You figure out how much, but then you pay for every one of those VMs, those nodes. You have a number of them. Yeah. I mean, look, like Slack raised a bunch of VC money, so I guess. [00:41:38] S1: Sure, sure. Yeah. [00:41:41] S2: And also from customers, of course. [00:41:43] S1: Yeah, yeah, yeah. [00:41:44] S2: Of course. That is wicked expensive. [00:41:46] S1: It is. So it was 37 cents. Per headless Chrome instance, per five-hour tranche of load testing. [00:41:54] S2: Yeah, we're talking about $150,000 for that one test. [00:41:57] S1: Yeah, yeah, yeah. Which like, okay, that's like financially not sensible. Like, it just didn't work.
00:41:57 · Maude Lemaire, Gergely Orosz
[00:41:57] S1: Yeah, yeah, yeah. Which like, okay, that's like financially not sensible. Like, it just didn't work. [00:42:04] S2: But I guess if you compare it, like, of course... I'm playing devil's advocate. $150,000 is great. But if you have a customer coming up with 500,000 seats, if they would get a discount, they would pay $5 million per month. You would think there is a point of investing to make sure that it works for them and they don't complain. I don't know where that comes in. But yes, the $150,000 still feels pretty excessive for that one test. [00:42:30] S1: Oh, yeah. Yeah. I think for that one, I mean, that's only a five-hour chunk of time. And that's assuming that you're able... Yeah, that's only five hours. [00:42:38] S2: Okay. Yeah. That's getting a problem. [00:42:41] S1: And that's assuming you're testing all of the feature tests that you wanted to run during that five hours. Which was not the case. That was not the case because we were spending half the time trying to bring pods back up because they were crashing. [00:42:55] S2: Okay. Did you figure out a better way to do it in the end? Yes.
00:42:55 · Gergely Orosz, Maude Lemaire
[00:42:55] S2: Okay. Did you figure out a better way to do it in the end? Yes. [00:42:57] S1: Yeah, we did. Lower fidelity, but definitely a better way. We gave up on the whole headless Chrome instance situation, which was obviously like the highest fidelity and best option if you wanted to make sure that it was truly testing how Slack would work. But instead, we spun up a bunch of little Go processes that would each establish a WebSocket connection and run through a series of API requests that made it look like They were actual clients. And so they would still receive everything over the WebSocket, send everything over the WebSocket, as you would expect, and still have a connection to the regular API that the client was making requests to and also our edge cache. Edgecache at all. But there's a bunch of data that needs to be really snappy to load when you load up Slack. Like when you go to switch channels, the channel list is in an Edgecache closer to where you are because all of the main API requests still hit US East 1. Which is great. I mean, we are so boned if US East 1 goes down. All those main requests still hit US East 1. So we still had to also test the latency associated with making requests to those edge caches. And so we also established that request pattern there.
00:44:12 · Gergely Orosz
[00:44:12] S2: It sounds to me that with load testing, with all I've heard from you is like, you know, step one is understand how your product will be used, what will likely top it over, which, because once, you know, you're not going to just randomly load test, you'll actually can test target it for that. And then Building something that simulates it good enough, knowing the resource constraints. So in this case, don't boot up a whole slack in a headless Chrome, but just mock, not mock, but build out that critical path that you care about. And if you combine this to your domain knowledge and your expertise, knowing how the product works and knowing how, you know, you knew how it used to fall over, you can now be reasonably confident that if it falls over, it will not be this way. It might be a different way, but I think it's nice that we're talking about the cost of testing, especially like we've talked about the cost of LLM, but like there is cost considerations and also repeatability, how lightweight it is, how much easier it is to control it and to control it. Yeah, it was really lightweight. It just did what it needed to do. We evolved the system whereby you could create routines to send to these fake users to run through over time. At first, it was just a very sort of silly JSON blob of like,
00:45:19 · Maude Lemaire
[00:45:19] S1: call this thing, then call this thing, then call this thing. I wrote a mini recorder tool where I could, via a slash command in Slack, say slash start recording, go through a whole flow in Slack. It would record every single API request I made across both to the normal web app. Backend and to our edge cache, it would record every single outgoing and incoming WebSocket message. And then I would sort of sort through it all to just the pieces that I wanted to simulate and then inject that into another tool that I wrote that turned that into a script that I could then send to the simulated clients. Because the other half of the battle was figuring out what are all of the APIs that get called when I do this one product flow, which is not a small number. And you wanted to be sure that you captured the entire thing. So that's where the recorder tool came in pretty handy. And we used that over and over and over again. And I would, again, to go back to the product thing, because I'm going to keep hammering the product thing. I think like the best product managers are mindful of like performance and reliability are important aspects of any given feature. And the best PMs know that and care about that and optimize for that. And I was trying the last like several years that I was there, I was always trying to push more tools into the hands of PMs and say like, hey, you want to add your use case to be automatically tested with Koi Pond with our load testing system? Here's the recorder tool. Go record the flow that you want us to test. Send us back the file. We'll plop it into the system and we'll run it at the frequency that you want it to run at. Because they knew, you know, oh, files do this weird thing when we like pop open a canvas and it like does this thing. And like, I don't know. I don't use canvases the way that like most of our users probably use them. You do. You're the PM. So like go record it and then bring it back to us and then we'll just shove it into the system we wrote.
00:47:35 · Gergely Orosz, Gergely Orosz, Maude Lemaire
[00:47:35] S2: Speaking of reliability, after Slack, you joined GitHub as a principal software engineer, and this was 2024. So this was at a time where GitHub Copilot was already very big. In fact, Cursor and other competitors were already as tools. [00:47:51] S0: The load on GitHub services already started to increase, and GitHub's reliability started to visibly become shakier, not as bad as it [00:47:59] S2: is these days. What did you observe inside of GitHub? Can you tell us the vibe when you joined? If you can give us a bit of an understanding and appreciation of what GitHub has been built on, this massive Ruby monolith. And when you joined, what was it like? [00:48:15] S1: It was a really interesting time, particularly because one of my friends from Rent the Runway, actually, we're going way back, had been at GitHub for many years at that point. And we'd sort of been keeping tabs on each other's careers. And she told me, Maude, we are going to run out of space. GitHub is still primarily on data centers, and it's increasingly less so over time. [00:48:41] S2: They used to have their own data centers. Now they're moving to Azure. This is public information they shared.