The Pragmatic Engineer
00:26:28 · Gergely Orosz, Maude Lemaire
[00:26:28] S2: But how do you think the production is something where you don't have the load just yet, right? [00:26:32] S1: Yeah, no, I mean, I think there's a lot of different ways to do it. The simplest way that I think a lot of people can manage to do it, it depends on what kind of usage you are worried about with your product. And nailing down those usage patterns first and where you're really concerned it's going to fall over. Is the determining factor in how you figure out how to simulate load. [00:26:56] S2: In Slack specifically, what was usage patterned? [00:26:59] S1: There were a couple of key ways that it had historically fallen over. Thundering herds, where I mean, I think most people have some kind of thundering herd problem at some point where you say. [00:27:09] S2: Can we spell out what the thundering herd is just for those who have not had a herd. Suddenly herd, suddenly everyone shows up. [00:27:17] S1: Suddenly, like, that is a thundering herd. A thing happens, and then suddenly everyone shows up and does the same thing at the same time. [00:27:24] S2: I guess a good example is if your service goes down, like you're, I don't know, you're running, let's say, imaginary, like, Google search and somehow, like, All your machines die for a bug. And then you have like a thousand machines. Obviously, Google has more, but we can just imagine. And then you start to bring up machine one, machine two, machine three. But each of them gets hit by the complete traffic and they go down again, right? [00:27:47] S1: Yes. Yes. [00:27:47] S2: This is a classic example.
00:27:47 · Gergely Orosz, Maude Lemaire
[00:27:47] S2: This is a classic example. [00:27:49] S1: Yeah. Yeah. The classic example in Slack is, say, CEO says, oh, great quarter, everyone. You all get Friday off. Posts this message in the big company channel that has like 100,000 active users in it. That notification gets sent out. You see that notification. You are very motivated to plop a really good custom emoji on that notification. You click that notification. You immediately land in the channel. In the channel, you're fetching channel history. And then on top of that, you are adding an emoji reaction because you're really pumped about getting Friday off. But so is everyone else also getting, putting custom emojis on that message because they're also pumped. [00:28:27] S2: In the same five seconds.
00:28:27 · Gergely Orosz, Maude Lemaire
[00:28:27] S2: In the same five seconds. [00:28:28] S1: Yes, in the same five seconds. So the message that came in to tell you that you're getting Friday off came in over the WebSocket connection and it was fanned out to all 100,000 active users. But then all 100,000 within a couple seconds, right, of those people show up to that channel, issue a channel read. And then are sending emoji reactions. Every single emoji reaction fans out also over the WebSocket. So one custom emoji turns into 100,000 custom emojis being sent to everyone who's connected. And if you add 10, well, then you can do the math and it just like balloons and balloons and balloons. It's like prime example for like pure explosion. And as much as I think we we had done some work because we knew that emoji fan out was like actually really expensive to rate the importance priority order, the types of messages that get sent over the WebSocket. And so like message like actual text messages were the most important. And then emojis. You were in a very high load situation, would just get dropped on the floor, which is the thing to do, obviously. [00:29:33] S2: I'm sorry to say, but the next generation, I'm not sure if Gen Z or Gen Apple might disagree that emojis are more important than text messages. [00:29:41] S1: I need to know if someone reacted to my message. [00:29:45] S2: No, but I hear it makes sense of prioritizing, especially based on size, right?
00:29:52 · Maude Lemaire
[00:29:52] S1: Yes, absolutely. And there's some amount of and this is where like I think product thinking is you can't go around thinking you're going to load test strictly from an engineering perspective. You have to be putting your product hat on. You absolutely have to because that's where. Understanding that messages with text are more important than emojis, that's more of a product call than it is an engineering call. I'm coming to you and I'm saying, we have to drop something. Let's reason together about what makes sense to drop and understand it. And sometimes it's not what you would expect. Yeah. And I think, I mean, this example I keep coming back to because I had multiple conversations with folks over at Slack about it. Like we should just limit the total number of emojis that are allowed on a message, period. So we currently limit, or I think last I checked, Slack still currently limits. You can only have 23 distinct custom emojis as a reaction on a message, but you can have as many like plus ones on those emojis as you would like. Why? The 10,001th emoji does not matter. We should just cut it off at some point. And I think, like, whoever, I mean, I think part of the problem is, like, no one really owned custom emojis. It would have been the messaging team, I guess. But, like, it's one of those things that's just so ubiquitous. Is that a word to the Slack experience? And, like, kind of no one owns it. So who would have made the call on, like, what is the proper limit to put on this? But Thundering Herds was definitely super top of mind. As one of those issues.
00:31:27 · Gergely Orosz, Maude Lemaire
[00:31:27] S2: I wanted to ask, are event floods and thundering curves the same thing? [00:31:32] S1: No, they're different. [00:31:33] S2: What are event floods? [00:31:35] S1: So event floods are when your client is personally overwhelmed by the events coming over the WebSocket connection. And so any I mean, that could be from that specific example. That's sort of a thundering herd and maybe an event flood event in that you're getting hammered with emoji reactions after the fact as well. But an event flood is just say, you know, This message is happening, the CEO you get Friday off, but it's happening in 16 different channels at the same time that you personally happen to be in. Not everyone else is in super, super busy channels, but maybe you happen to be. This happened to... There was this guy who was our like main Slack guy at IBM. His name is Jeff Robke. He eventually became a PM at Slack, which I think is like the most interesting thing ever because this guy was on the receiving end of so many event floods. And I felt so bad for Jeff all the time. I mean, I was in some channels. I'm like, I'm like, tell Jeff, I'm sorry. [00:32:38] S2: But I still don't fully get like what the event flood experience is. Can you tell me like what? Yeah. So one of the things that would happen to
00:32:38 · Gergely Orosz, Maude Lemaire
[00:32:38] S2: But I still don't fully get like what the event flood experience is. Can you tell me like what? Yeah. So one of the things that would happen to [00:32:44] S1: him, right, is that before some of the re-architecture work whereby you didn't have a bunch of distinct WebSocket connections for each team that you were logged in, Jeff was an admin at IBM. So he was in... Hundreds of different workspaces. At one point, I think the maximum number of workspaces you could be logged into was around like 30 something. [00:33:07] S2: And you're saying he was in this because IBM as an enterprise had this concept of like the marketing team, the finance team, the thing. Maybe they had an external customer team. And as the admin, he was in all of. [00:33:20] S1: And they had so many teams and he was in all of them. And so if you were getting a message in an enterprise level DM or a channel that is shared across all 30 of those workspaces that he was currently logged into, well, he would get that message 30 times. There is no deduplication of that whatsoever. [00:33:40] S2: So his client, his machine, his notifications would be both overwhelmed from maybe a little bit less in processing. That's fine. But like from a UI, from being unable to see what's going on, annoyance point of view. [00:33:56] S1: Yeah. I mean, also his client was just dying all the time, which was not good. [00:34:01] S2: Yeah. [00:34:02] S1: Yeah. [00:34:03] S2: Interesting. This is the first time I've heard about event floods.