The Pragmatic Engineer
00:37:42 · Gergely Orosz
[00:37:42] S2: And then back to load testing. So how Did you simulate the load that you would be expecting to get? In this case, you knew that a big customer would be coming with 500,000 or a million users. Yeah. And you didn't know if Slack could handle it, but you want to figure out before testing and production, before they all came in, did you build a custom load testing tool? How did you go about it?
00:38:04 · Maude Lemaire, Gergely Orosz
[00:38:04] S1: Yeah. So we'd had a couple of like half conceived load testing options to date. Mostly that tested just API throughput and making sure that we could withstand the underlying load primarily on the database. And so if you are loading up a channel that issues a particular API request, I think it's conversations.view unless they last changed it. And that makes a particular database request a SQL read on that channel shard now because everything started differently. But it makes a particular request. We wanted to be able to We wanted to know that we could handle that RPS. The problem with that is, and any of the load testing solutions before this particular moment in time, was none of it had active WebSocket connections. And the two particular cases that we just talked about, the WebSocket is the problem in some ways. I mean, we've never really had latency issues with sending messages through the WebSocket. Like those stateful services have One of the co-founders was like the mastermind behind all of these and like the amount of time and energy he sank into making them as performant as humanly possible is commendable. They were never really a bottleneck. It was either how the client handled it or the re-downstream event caused by you seeing and receiving this message that was the further bottleneck. And so we needed a system that had an active WebSocket connection in order to be able to adequately test the whole flow. It's not just sending, like, as though you're reading a channel. [00:39:40] S2: It's not just an API, I get it.
00:39:40 · Gergely Orosz, Maude Lemaire
[00:39:40] S2: It's not just an API, I get it. [00:39:41] S1: So we really, and it's not just, like, flooding a database with, like... [00:39:45] S2: But when you say you needed an active WebSocket connection to test for 500,000 loads, you needed 500,000 active WebSocket connections. [00:39:51] S1: We needed 500,000 active WebSocket connections. Yes, we did. [00:39:54] S2: Okay. [00:39:55] S1: Yeah, yeah, which is not easy to do. And so we had a solution before this customer came in, which I'm 90% sure it's public knowledge now that it's Amazon, but we had a solution that would spin up Slack as a headless Chrome instance in like a bunch of Kubernetes pods and try to run a tiny little JavaScript script and like click buttons, like pretend to be a Slack user. [00:40:23] S2: Pretend to be a user, yeah. [00:40:24] S1: Which was sort of fine, but I don't know if you've tried to wrangle more than 50,000 headless Chrome instances in Kubernetes. [00:40:38] S2: I don't think many of us have. I don't think many people listening or viewing have. [00:40:44] S1: It is not a good time. I mean, the problem with the headless Chrome instance, right, is just running Slack and 200 Chrome tabs and whatever on your laptop is kind of a time. And now you're running LLMs on top of it, but it's an experience in really pushing your laptop to the limit. And now you're running per pod, attempting to run 10,000 headless Chrome instances of Slack, which is already Very memory and CPU, mostly memory intensive application. The amount of money we were spending in these EC2 instances...
00:40:44 · Maude Lemaire, Gergely Orosz
[00:40:44] S1: It is not a good time. I mean, the problem with the headless Chrome instance, right, is just running Slack and 200 Chrome tabs and whatever on your laptop is kind of a time. And now you're running LLMs on top of it, but it's an experience in really pushing your laptop to the limit. And now you're running per pod, attempting to run 10,000 headless Chrome instances of Slack, which is already Very memory and CPU, mostly memory intensive application. The amount of money we were spending in these EC2 instances... [00:41:19] S2: The problem seems to be that per node, per VM node, depending on the size, you will be able to run, you know, you'll know, like, I don't know, like 10, 20, 50, 100. I don't know. You figure out how much, but then you pay for every one of those VMs, those nodes. You have a number of them. Yeah. I mean, look, like Slack raised a bunch of VC money, so I guess. [00:41:38] S1: Sure, sure. Yeah. [00:41:41] S2: And also from customers, of course. [00:41:43] S1: Yeah, yeah, yeah. [00:41:44] S2: Of course. That is wicked expensive. [00:41:46] S1: It is. So it was 37 cents. Per headless Chrome instance, per five-hour tranche of load testing. [00:41:54] S2: Yeah, we're talking about $150,000 for that one test. [00:41:57] S1: Yeah, yeah, yeah. Which like, okay, that's like financially not sensible. Like, it just didn't work.
00:41:57 · Maude Lemaire, Gergely Orosz
[00:41:57] S1: Yeah, yeah, yeah. Which like, okay, that's like financially not sensible. Like, it just didn't work. [00:42:04] S2: But I guess if you compare it, like, of course... I'm playing devil's advocate. $150,000 is great. But if you have a customer coming up with 500,000 seats, if they would get a discount, they would pay $5 million per month. You would think there is a point of investing to make sure that it works for them and they don't complain. I don't know where that comes in. But yes, the $150,000 still feels pretty excessive for that one test. [00:42:30] S1: Oh, yeah. Yeah. I think for that one, I mean, that's only a five-hour chunk of time. And that's assuming that you're able... Yeah, that's only five hours. [00:42:38] S2: Okay. Yeah. That's getting a problem. [00:42:41] S1: And that's assuming you're testing all of the feature tests that you wanted to run during that five hours. Which was not the case. That was not the case because we were spending half the time trying to bring pods back up because they were crashing. [00:42:55] S2: Okay. Did you figure out a better way to do it in the end? Yes.
00:42:55 · Gergely Orosz, Maude Lemaire
[00:42:55] S2: Okay. Did you figure out a better way to do it in the end? Yes. [00:42:57] S1: Yeah, we did. Lower fidelity, but definitely a better way. We gave up on the whole headless Chrome instance situation, which was obviously like the highest fidelity and best option if you wanted to make sure that it was truly testing how Slack would work. But instead, we spun up a bunch of little Go processes that would each establish a WebSocket connection and run through a series of API requests that made it look like They were actual clients. And so they would still receive everything over the WebSocket, send everything over the WebSocket, as you would expect, and still have a connection to the regular API that the client was making requests to and also our edge cache. Edgecache at all. But there's a bunch of data that needs to be really snappy to load when you load up Slack. Like when you go to switch channels, the channel list is in an Edgecache closer to where you are because all of the main API requests still hit US East 1. Which is great. I mean, we are so boned if US East 1 goes down. All those main requests still hit US East 1. So we still had to also test the latency associated with making requests to those edge caches. And so we also established that request pattern there.