The Pragmatic Engineer
00:23:45 · Gergely Orosz, Maude Lemaire
[00:23:45] S2: Oh, we didn't just do a gradual rollout, because with these things, you don't really do a gradual rollout. [00:23:51] S1: Internally, we were not that big at the time. Everyone was at gather hour. I was like, oh, this seems safe. We've been shadow reading. It's fine. Yeah, I turned it on. I was like, oh, there's some data that's not quite right. And I had been so used Essentially running the script that would delete all of the backing data and rerun the backfill, or at least chunks of it, because I had been seeing discrepancies. And honestly, it was so much easier to just nuke the data, re-backfill, than it was to try to pinpoint and scalpel, fix the call sites. So I forgot that the flag was still on to everyone. And then I promptly deleted... Which [00:24:38] S2: flag was on? [00:24:39] S1: The flag to have everyone read their channel memberships from the new table. [00:24:43] S2: Okay. [00:24:43] S1: Yeah. I forgot that it was on for Slack. Not everyone externally, like just us. And I was like, oh, there's some discrepancies. Oh, right. But I fixed those. Let me just nuke the table for our team and backfill, which like took 25 minutes to do. Like it was not. [00:25:00] S2: Oh, but did everyone lose their messages until then? [00:25:03] S1: Oh, yeah. People had completely blank sidebars for about half an hour. [00:25:07] S2: Inside of all of those. Inside all of Slack. And Slack always uses Slack for everything.
00:25:07 · Gergely Orosz, Maude Lemaire
[00:25:07] S2: Inside of all of those. Inside all of Slack. And Slack always uses Slack for everything. [00:25:12] S1: For everything. But everyone was at gather hours. So there was this only one guy, Kevin, who had been sort of helping me coach through this, who DM'd me. And he was like, what's going on with my sidebar? I was like, I'm so sorry. Oh, my gosh. Let me flip the flag. So I re-flipped to read from the old data while it was backfilling, finished backfilling, and then I swapped it back. [00:25:33] S2: So what did you learn at Slack about load testing? And I'm asking because right now load is going up on all systems. Oh my gosh, yes. We see this. AI agents generate more pull requests, more code. They put more load on the systems. I assume we will have the point where we generate more data here and there. But at Slack, you have this interesting challenge where a customer with 500 I'm more interested in that, and as you built load testing, something called Koi Pond on this, but when you think of load testing, of checking how a service will be able to handle this type of load before it hits, what are the techniques that you've used and would still use today? [00:26:26] S1: Tested production. [00:26:28] S2: But how do you think the production is something where you don't have the load just yet, right?
00:26:28 · Gergely Orosz, Maude Lemaire
[00:26:28] S2: But how do you think the production is something where you don't have the load just yet, right? [00:26:32] S1: Yeah, no, I mean, I think there's a lot of different ways to do it. The simplest way that I think a lot of people can manage to do it, it depends on what kind of usage you are worried about with your product. And nailing down those usage patterns first and where you're really concerned it's going to fall over. Is the determining factor in how you figure out how to simulate load. [00:26:56] S2: In Slack specifically, what was usage patterned? [00:26:59] S1: There were a couple of key ways that it had historically fallen over. Thundering herds, where I mean, I think most people have some kind of thundering herd problem at some point where you say. [00:27:09] S2: Can we spell out what the thundering herd is just for those who have not had a herd. Suddenly herd, suddenly everyone shows up. [00:27:17] S1: Suddenly, like, that is a thundering herd. A thing happens, and then suddenly everyone shows up and does the same thing at the same time. [00:27:24] S2: I guess a good example is if your service goes down, like you're, I don't know, you're running, let's say, imaginary, like, Google search and somehow, like, All your machines die for a bug. And then you have like a thousand machines. Obviously, Google has more, but we can just imagine. And then you start to bring up machine one, machine two, machine three. But each of them gets hit by the complete traffic and they go down again, right? [00:27:47] S1: Yes. Yes. [00:27:47] S2: This is a classic example.
00:27:47 · Gergely Orosz, Maude Lemaire
[00:27:47] S2: This is a classic example. [00:27:49] S1: Yeah. Yeah. The classic example in Slack is, say, CEO says, oh, great quarter, everyone. You all get Friday off. Posts this message in the big company channel that has like 100,000 active users in it. That notification gets sent out. You see that notification. You are very motivated to plop a really good custom emoji on that notification. You click that notification. You immediately land in the channel. In the channel, you're fetching channel history. And then on top of that, you are adding an emoji reaction because you're really pumped about getting Friday off. But so is everyone else also getting, putting custom emojis on that message because they're also pumped. [00:28:27] S2: In the same five seconds.
00:28:27 · Gergely Orosz, Maude Lemaire
[00:28:27] S2: In the same five seconds. [00:28:28] S1: Yes, in the same five seconds. So the message that came in to tell you that you're getting Friday off came in over the WebSocket connection and it was fanned out to all 100,000 active users. But then all 100,000 within a couple seconds, right, of those people show up to that channel, issue a channel read. And then are sending emoji reactions. Every single emoji reaction fans out also over the WebSocket. So one custom emoji turns into 100,000 custom emojis being sent to everyone who's connected. And if you add 10, well, then you can do the math and it just like balloons and balloons and balloons. It's like prime example for like pure explosion. And as much as I think we we had done some work because we knew that emoji fan out was like actually really expensive to rate the importance priority order, the types of messages that get sent over the WebSocket. And so like message like actual text messages were the most important. And then emojis. You were in a very high load situation, would just get dropped on the floor, which is the thing to do, obviously. [00:29:33] S2: I'm sorry to say, but the next generation, I'm not sure if Gen Z or Gen Apple might disagree that emojis are more important than text messages. [00:29:41] S1: I need to know if someone reacted to my message. [00:29:45] S2: No, but I hear it makes sense of prioritizing, especially based on size, right?
00:29:52 · Maude Lemaire
[00:29:52] S1: Yes, absolutely. And there's some amount of and this is where like I think product thinking is you can't go around thinking you're going to load test strictly from an engineering perspective. You have to be putting your product hat on. You absolutely have to because that's where. Understanding that messages with text are more important than emojis, that's more of a product call than it is an engineering call. I'm coming to you and I'm saying, we have to drop something. Let's reason together about what makes sense to drop and understand it. And sometimes it's not what you would expect. Yeah. And I think, I mean, this example I keep coming back to because I had multiple conversations with folks over at Slack about it. Like we should just limit the total number of emojis that are allowed on a message, period. So we currently limit, or I think last I checked, Slack still currently limits. You can only have 23 distinct custom emojis as a reaction on a message, but you can have as many like plus ones on those emojis as you would like. Why? The 10,001th emoji does not matter. We should just cut it off at some point. And I think, like, whoever, I mean, I think part of the problem is, like, no one really owned custom emojis. It would have been the messaging team, I guess. But, like, it's one of those things that's just so ubiquitous. Is that a word to the Slack experience? And, like, kind of no one owns it. So who would have made the call on, like, what is the proper limit to put on this? But Thundering Herds was definitely super top of mind. As one of those issues.