The Pragmatic Engineer
00:32:38 · Gergely Orosz, Maude Lemaire
[00:32:38] S2: But I still don't fully get like what the event flood experience is. Can you tell me like what? Yeah. So one of the things that would happen to [00:32:44] S1: him, right, is that before some of the re-architecture work whereby you didn't have a bunch of distinct WebSocket connections for each team that you were logged in, Jeff was an admin at IBM. So he was in... Hundreds of different workspaces. At one point, I think the maximum number of workspaces you could be logged into was around like 30 something. [00:33:07] S2: And you're saying he was in this because IBM as an enterprise had this concept of like the marketing team, the finance team, the thing. Maybe they had an external customer team. And as the admin, he was in all of. [00:33:20] S1: And they had so many teams and he was in all of them. And so if you were getting a message in an enterprise level DM or a channel that is shared across all 30 of those workspaces that he was currently logged into, well, he would get that message 30 times. There is no deduplication of that whatsoever. [00:33:40] S2: So his client, his machine, his notifications would be both overwhelmed from maybe a little bit less in processing. That's fine. But like from a UI, from being unable to see what's going on, annoyance point of view. [00:33:56] S1: Yeah. I mean, also his client was just dying all the time, which was not good. [00:34:01] S2: Yeah. [00:34:02] S1: Yeah. [00:34:03] S2: Interesting. This is the first time I've heard about event floods.
00:34:03 · Gergely Orosz, Maude Lemaire
[00:34:03] S2: Interesting. This is the first time I've heard about event floods. [00:34:06] S1: Yeah. And they were very user specific, but they were particularly painful for power users. This happened to power users all the time. And power users were also some of the biggest champions of Slack. And so it's a double edged sword of like, I'm sorry you get the absolute worst experience of our product, but also you love it, I guess. And you are trying to make sure that people adopt it more. It's really, really rough.
00:34:30 · Gergely Orosz
[00:34:30] S0: Mo just made the point that the heaviest users of Slack got the worst of the event floods. Today, the heaviest users of Git happen to be your agents that often push code in parallel, which brings us to our seasoned sponsor, Entire. A few weeks ago, I told you that Entire was building a Git forge. Well, it's finally here. You can now host private repos directly on Entire in the US, Europe, India, and Australia. And Entire has just reinvented a pull request for the agentic era. The problem with PRs is how agents write it faster than how any of us can review it. So Entire just replaced it with something called trails. A trail collects everything about the work in one place, the intent, the code changes, and the agent sessions. A tail can be spread across multiple repos, so the change to your backend and your web app gets reviewed once. With a trail, you can configure customized evals like code review, security reviews, slot reviews, and others. They look like a heads-up display in an airplane and score your work on every push. And you can see how each agent from passes or fails your user to find gates and then decide when you merge based on these. The point is that you control when you allow your agents to be allowed to merge code like merging only after a human review or after all gates pass. Oh, and one more thing. Entire ship continuous memory. This turns your repo into your brain so your agents can search past coding sessions to find the context behind earlier changes. Basically, answer the why behind the code changes. Memories get shared across the team and across the repo. If you want to see what an AI-native GitForge looks like, head to entire.io slash pragmatic and check out Trails. I'd also like to mention our presenting sponsor, TurboPuffer, who are rebuilding their storage engine to make search faster and cheaper at the kind of scale we're talking about in this episode. The new system, called TurboPuffer V3, is not yet in prod, but it already passes CI. And when the rewrite started in September, it was not nearly as performant as the current production version, V2. The TPUF team has been perf-maxing query plans until they are as fast or faster than V2, doing all of this in public all the way up to the V3 production launch. And it's so cool to see their progress.
00:36:35 · Gergely Orosz
The new system went from being 10 to 100 times slower across most The V3 system is still slower in areas like hot full-text search, but only 1-2 times slower versus 100 times slower when the perfmaxing started. And the TPUF team is sharing exactly how they optimize things in their development log with things like vectorized query execution, group count aggregation, compressed column encodings, and many more. But hang on, why is the team doing a full rewrite? It's because search is changing, and so is turbopuffer. TBuff started as a vector database, but it's no longer a VectorDB. It has evolved into a high-performance database for search. The V3 discards the Vector Primary Index, and all indexes will share the same columnar storage system. Every query will run on vectorized query engines, which is something that you don't usually see in search databases, but which will both result in very fast search and just generally a very fast database. The TurboPuffer team is keeping a work log of their journey and you can follow along at turbopuffer.com slash v3. It's a blast to see how it's being built. And with this, let's get back to mode and how Slack load tested for a customer expecting to have 500,000 users.
00:37:42 · Gergely Orosz
[00:37:42] S2: And then back to load testing. So how Did you simulate the load that you would be expecting to get? In this case, you knew that a big customer would be coming with 500,000 or a million users. Yeah. And you didn't know if Slack could handle it, but you want to figure out before testing and production, before they all came in, did you build a custom load testing tool? How did you go about it?
00:38:04 · Maude Lemaire, Gergely Orosz
[00:38:04] S1: Yeah. So we'd had a couple of like half conceived load testing options to date. Mostly that tested just API throughput and making sure that we could withstand the underlying load primarily on the database. And so if you are loading up a channel that issues a particular API request, I think it's conversations.view unless they last changed it. And that makes a particular database request a SQL read on that channel shard now because everything started differently. But it makes a particular request. We wanted to be able to We wanted to know that we could handle that RPS. The problem with that is, and any of the load testing solutions before this particular moment in time, was none of it had active WebSocket connections. And the two particular cases that we just talked about, the WebSocket is the problem in some ways. I mean, we've never really had latency issues with sending messages through the WebSocket. Like those stateful services have One of the co-founders was like the mastermind behind all of these and like the amount of time and energy he sank into making them as performant as humanly possible is commendable. They were never really a bottleneck. It was either how the client handled it or the re-downstream event caused by you seeing and receiving this message that was the further bottleneck. And so we needed a system that had an active WebSocket connection in order to be able to adequately test the whole flow. It's not just sending, like, as though you're reading a channel. [00:39:40] S2: It's not just an API, I get it.