The Pragmatic Engineer
00:28:27 · Gergely Orosz, Maude Lemaire
[00:28:27] S2: In the same five seconds. [00:28:28] S1: Yes, in the same five seconds. So the message that came in to tell you that you're getting Friday off came in over the WebSocket connection and it was fanned out to all 100,000 active users. But then all 100,000 within a couple seconds, right, of those people show up to that channel, issue a channel read. And then are sending emoji reactions. Every single emoji reaction fans out also over the WebSocket. So one custom emoji turns into 100,000 custom emojis being sent to everyone who's connected. And if you add 10, well, then you can do the math and it just like balloons and balloons and balloons. It's like prime example for like pure explosion. And as much as I think we we had done some work because we knew that emoji fan out was like actually really expensive to rate the importance priority order, the types of messages that get sent over the WebSocket. And so like message like actual text messages were the most important. And then emojis. You were in a very high load situation, would just get dropped on the floor, which is the thing to do, obviously. [00:29:33] S2: I'm sorry to say, but the next generation, I'm not sure if Gen Z or Gen Apple might disagree that emojis are more important than text messages. [00:29:41] S1: I need to know if someone reacted to my message. [00:29:45] S2: No, but I hear it makes sense of prioritizing, especially based on size, right?
00:29:52 · Maude Lemaire
[00:29:52] S1: Yes, absolutely. And there's some amount of and this is where like I think product thinking is you can't go around thinking you're going to load test strictly from an engineering perspective. You have to be putting your product hat on. You absolutely have to because that's where. Understanding that messages with text are more important than emojis, that's more of a product call than it is an engineering call. I'm coming to you and I'm saying, we have to drop something. Let's reason together about what makes sense to drop and understand it. And sometimes it's not what you would expect. Yeah. And I think, I mean, this example I keep coming back to because I had multiple conversations with folks over at Slack about it. Like we should just limit the total number of emojis that are allowed on a message, period. So we currently limit, or I think last I checked, Slack still currently limits. You can only have 23 distinct custom emojis as a reaction on a message, but you can have as many like plus ones on those emojis as you would like. Why? The 10,001th emoji does not matter. We should just cut it off at some point. And I think, like, whoever, I mean, I think part of the problem is, like, no one really owned custom emojis. It would have been the messaging team, I guess. But, like, it's one of those things that's just so ubiquitous. Is that a word to the Slack experience? And, like, kind of no one owns it. So who would have made the call on, like, what is the proper limit to put on this? But Thundering Herds was definitely super top of mind. As one of those issues.
00:31:27 · Gergely Orosz, Maude Lemaire
[00:31:27] S2: I wanted to ask, are event floods and thundering curves the same thing? [00:31:32] S1: No, they're different. [00:31:33] S2: What are event floods? [00:31:35] S1: So event floods are when your client is personally overwhelmed by the events coming over the WebSocket connection. And so any I mean, that could be from that specific example. That's sort of a thundering herd and maybe an event flood event in that you're getting hammered with emoji reactions after the fact as well. But an event flood is just say, you know, This message is happening, the CEO you get Friday off, but it's happening in 16 different channels at the same time that you personally happen to be in. Not everyone else is in super, super busy channels, but maybe you happen to be. This happened to... There was this guy who was our like main Slack guy at IBM. His name is Jeff Robke. He eventually became a PM at Slack, which I think is like the most interesting thing ever because this guy was on the receiving end of so many event floods. And I felt so bad for Jeff all the time. I mean, I was in some channels. I'm like, I'm like, tell Jeff, I'm sorry. [00:32:38] S2: But I still don't fully get like what the event flood experience is. Can you tell me like what? Yeah. So one of the things that would happen to
00:32:38 · Gergely Orosz, Maude Lemaire
[00:32:38] S2: But I still don't fully get like what the event flood experience is. Can you tell me like what? Yeah. So one of the things that would happen to [00:32:44] S1: him, right, is that before some of the re-architecture work whereby you didn't have a bunch of distinct WebSocket connections for each team that you were logged in, Jeff was an admin at IBM. So he was in... Hundreds of different workspaces. At one point, I think the maximum number of workspaces you could be logged into was around like 30 something. [00:33:07] S2: And you're saying he was in this because IBM as an enterprise had this concept of like the marketing team, the finance team, the thing. Maybe they had an external customer team. And as the admin, he was in all of. [00:33:20] S1: And they had so many teams and he was in all of them. And so if you were getting a message in an enterprise level DM or a channel that is shared across all 30 of those workspaces that he was currently logged into, well, he would get that message 30 times. There is no deduplication of that whatsoever. [00:33:40] S2: So his client, his machine, his notifications would be both overwhelmed from maybe a little bit less in processing. That's fine. But like from a UI, from being unable to see what's going on, annoyance point of view. [00:33:56] S1: Yeah. I mean, also his client was just dying all the time, which was not good. [00:34:01] S2: Yeah. [00:34:02] S1: Yeah. [00:34:03] S2: Interesting. This is the first time I've heard about event floods.
00:34:03 · Gergely Orosz, Maude Lemaire
[00:34:03] S2: Interesting. This is the first time I've heard about event floods. [00:34:06] S1: Yeah. And they were very user specific, but they were particularly painful for power users. This happened to power users all the time. And power users were also some of the biggest champions of Slack. And so it's a double edged sword of like, I'm sorry you get the absolute worst experience of our product, but also you love it, I guess. And you are trying to make sure that people adopt it more. It's really, really rough.
00:34:30 · Gergely Orosz
[00:34:30] S0: Mo just made the point that the heaviest users of Slack got the worst of the event floods. Today, the heaviest users of Git happen to be your agents that often push code in parallel, which brings us to our seasoned sponsor, Entire. A few weeks ago, I told you that Entire was building a Git forge. Well, it's finally here. You can now host private repos directly on Entire in the US, Europe, India, and Australia. And Entire has just reinvented a pull request for the agentic era. The problem with PRs is how agents write it faster than how any of us can review it. So Entire just replaced it with something called trails. A trail collects everything about the work in one place, the intent, the code changes, and the agent sessions. A tail can be spread across multiple repos, so the change to your backend and your web app gets reviewed once. With a trail, you can configure customized evals like code review, security reviews, slot reviews, and others. They look like a heads-up display in an airplane and score your work on every push. And you can see how each agent from passes or fails your user to find gates and then decide when you merge based on these. The point is that you control when you allow your agents to be allowed to merge code like merging only after a human review or after all gates pass. Oh, and one more thing. Entire ship continuous memory. This turns your repo into your brain so your agents can search past coding sessions to find the context behind earlier changes. Basically, answer the why behind the code changes. Memories get shared across the team and across the repo. If you want to see what an AI-native GitForge looks like, head to entire.io slash pragmatic and check out Trails. I'd also like to mention our presenting sponsor, TurboPuffer, who are rebuilding their storage engine to make search faster and cheaper at the kind of scale we're talking about in this episode. The new system, called TurboPuffer V3, is not yet in prod, but it already passes CI. And when the rewrite started in September, it was not nearly as performant as the current production version, V2. The TPUF team has been perf-maxing query plans until they are as fast or faster than V2, doing all of this in public all the way up to the V3 production launch. And it's so cool to see their progress.