The Pragmatic Engineer
00:34:03 · Gergely Orosz, Maude Lemaire
[00:34:03] S2: Interesting. This is the first time I've heard about event floods. [00:34:06] S1: Yeah. And they were very user specific, but they were particularly painful for power users. This happened to power users all the time. And power users were also some of the biggest champions of Slack. And so it's a double edged sword of like, I'm sorry you get the absolute worst experience of our product, but also you love it, I guess. And you are trying to make sure that people adopt it more. It's really, really rough.
00:34:30 · Gergely Orosz
[00:34:30] S0: Mo just made the point that the heaviest users of Slack got the worst of the event floods. Today, the heaviest users of Git happen to be your agents that often push code in parallel, which brings us to our seasoned sponsor, Entire. A few weeks ago, I told you that Entire was building a Git forge. Well, it's finally here. You can now host private repos directly on Entire in the US, Europe, India, and Australia. And Entire has just reinvented a pull request for the agentic era. The problem with PRs is how agents write it faster than how any of us can review it. So Entire just replaced it with something called trails. A trail collects everything about the work in one place, the intent, the code changes, and the agent sessions. A tail can be spread across multiple repos, so the change to your backend and your web app gets reviewed once. With a trail, you can configure customized evals like code review, security reviews, slot reviews, and others. They look like a heads-up display in an airplane and score your work on every push. And you can see how each agent from passes or fails your user to find gates and then decide when you merge based on these. The point is that you control when you allow your agents to be allowed to merge code like merging only after a human review or after all gates pass. Oh, and one more thing. Entire ship continuous memory. This turns your repo into your brain so your agents can search past coding sessions to find the context behind earlier changes. Basically, answer the why behind the code changes. Memories get shared across the team and across the repo. If you want to see what an AI-native GitForge looks like, head to entire.io slash pragmatic and check out Trails. I'd also like to mention our presenting sponsor, TurboPuffer, who are rebuilding their storage engine to make search faster and cheaper at the kind of scale we're talking about in this episode. The new system, called TurboPuffer V3, is not yet in prod, but it already passes CI. And when the rewrite started in September, it was not nearly as performant as the current production version, V2. The TPUF team has been perf-maxing query plans until they are as fast or faster than V2, doing all of this in public all the way up to the V3 production launch. And it's so cool to see their progress.
00:36:35 · Gergely Orosz
The new system went from being 10 to 100 times slower across most The V3 system is still slower in areas like hot full-text search, but only 1-2 times slower versus 100 times slower when the perfmaxing started. And the TPUF team is sharing exactly how they optimize things in their development log with things like vectorized query execution, group count aggregation, compressed column encodings, and many more. But hang on, why is the team doing a full rewrite? It's because search is changing, and so is turbopuffer. TBuff started as a vector database, but it's no longer a VectorDB. It has evolved into a high-performance database for search. The V3 discards the Vector Primary Index, and all indexes will share the same columnar storage system. Every query will run on vectorized query engines, which is something that you don't usually see in search databases, but which will both result in very fast search and just generally a very fast database. The TurboPuffer team is keeping a work log of their journey and you can follow along at turbopuffer.com slash v3. It's a blast to see how it's being built. And with this, let's get back to mode and how Slack load tested for a customer expecting to have 500,000 users.
00:37:42 · Gergely Orosz
[00:37:42] S2: And then back to load testing. So how Did you simulate the load that you would be expecting to get? In this case, you knew that a big customer would be coming with 500,000 or a million users. Yeah. And you didn't know if Slack could handle it, but you want to figure out before testing and production, before they all came in, did you build a custom load testing tool? How did you go about it?
00:38:04 · Maude Lemaire, Gergely Orosz
[00:38:04] S1: Yeah. So we'd had a couple of like half conceived load testing options to date. Mostly that tested just API throughput and making sure that we could withstand the underlying load primarily on the database. And so if you are loading up a channel that issues a particular API request, I think it's conversations.view unless they last changed it. And that makes a particular database request a SQL read on that channel shard now because everything started differently. But it makes a particular request. We wanted to be able to We wanted to know that we could handle that RPS. The problem with that is, and any of the load testing solutions before this particular moment in time, was none of it had active WebSocket connections. And the two particular cases that we just talked about, the WebSocket is the problem in some ways. I mean, we've never really had latency issues with sending messages through the WebSocket. Like those stateful services have One of the co-founders was like the mastermind behind all of these and like the amount of time and energy he sank into making them as performant as humanly possible is commendable. They were never really a bottleneck. It was either how the client handled it or the re-downstream event caused by you seeing and receiving this message that was the further bottleneck. And so we needed a system that had an active WebSocket connection in order to be able to adequately test the whole flow. It's not just sending, like, as though you're reading a channel. [00:39:40] S2: It's not just an API, I get it.
00:39:40 · Gergely Orosz, Maude Lemaire
[00:39:40] S2: It's not just an API, I get it. [00:39:41] S1: So we really, and it's not just, like, flooding a database with, like... [00:39:45] S2: But when you say you needed an active WebSocket connection to test for 500,000 loads, you needed 500,000 active WebSocket connections. [00:39:51] S1: We needed 500,000 active WebSocket connections. Yes, we did. [00:39:54] S2: Okay. [00:39:55] S1: Yeah, yeah, which is not easy to do. And so we had a solution before this customer came in, which I'm 90% sure it's public knowledge now that it's Amazon, but we had a solution that would spin up Slack as a headless Chrome instance in like a bunch of Kubernetes pods and try to run a tiny little JavaScript script and like click buttons, like pretend to be a Slack user. [00:40:23] S2: Pretend to be a user, yeah. [00:40:24] S1: Which was sort of fine, but I don't know if you've tried to wrangle more than 50,000 headless Chrome instances in Kubernetes. [00:40:38] S2: I don't think many of us have. I don't think many people listening or viewing have. [00:40:44] S1: It is not a good time. I mean, the problem with the headless Chrome instance, right, is just running Slack and 200 Chrome tabs and whatever on your laptop is kind of a time. And now you're running LLMs on top of it, but it's an experience in really pushing your laptop to the limit. And now you're running per pod, attempting to run 10,000 headless Chrome instances of Slack, which is already Very memory and CPU, mostly memory intensive application. The amount of money we were spending in these EC2 instances...