The Pragmatic Engineer
00:36:35 · Gergely Orosz
The new system went from being 10 to 100 times slower across most The V3 system is still slower in areas like hot full-text search, but only 1-2 times slower versus 100 times slower when the perfmaxing started. And the TPUF team is sharing exactly how they optimize things in their development log with things like vectorized query execution, group count aggregation, compressed column encodings, and many more. But hang on, why is the team doing a full rewrite? It's because search is changing, and so is turbopuffer. TBuff started as a vector database, but it's no longer a VectorDB. It has evolved into a high-performance database for search. The V3 discards the Vector Primary Index, and all indexes will share the same columnar storage system. Every query will run on vectorized query engines, which is something that you don't usually see in search databases, but which will both result in very fast search and just generally a very fast database. The TurboPuffer team is keeping a work log of their journey and you can follow along at turbopuffer.com slash v3. It's a blast to see how it's being built. And with this, let's get back to mode and how Slack load tested for a customer expecting to have 500,000 users.
00:37:42 · Gergely Orosz
[00:37:42] S2: And then back to load testing. So how Did you simulate the load that you would be expecting to get? In this case, you knew that a big customer would be coming with 500,000 or a million users. Yeah. And you didn't know if Slack could handle it, but you want to figure out before testing and production, before they all came in, did you build a custom load testing tool? How did you go about it?
00:38:04 · Maude Lemaire, Gergely Orosz
[00:38:04] S1: Yeah. So we'd had a couple of like half conceived load testing options to date. Mostly that tested just API throughput and making sure that we could withstand the underlying load primarily on the database. And so if you are loading up a channel that issues a particular API request, I think it's conversations.view unless they last changed it. And that makes a particular database request a SQL read on that channel shard now because everything started differently. But it makes a particular request. We wanted to be able to We wanted to know that we could handle that RPS. The problem with that is, and any of the load testing solutions before this particular moment in time, was none of it had active WebSocket connections. And the two particular cases that we just talked about, the WebSocket is the problem in some ways. I mean, we've never really had latency issues with sending messages through the WebSocket. Like those stateful services have One of the co-founders was like the mastermind behind all of these and like the amount of time and energy he sank into making them as performant as humanly possible is commendable. They were never really a bottleneck. It was either how the client handled it or the re-downstream event caused by you seeing and receiving this message that was the further bottleneck. And so we needed a system that had an active WebSocket connection in order to be able to adequately test the whole flow. It's not just sending, like, as though you're reading a channel. [00:39:40] S2: It's not just an API, I get it.
00:39:40 · Gergely Orosz, Maude Lemaire
[00:39:40] S2: It's not just an API, I get it. [00:39:41] S1: So we really, and it's not just, like, flooding a database with, like... [00:39:45] S2: But when you say you needed an active WebSocket connection to test for 500,000 loads, you needed 500,000 active WebSocket connections. [00:39:51] S1: We needed 500,000 active WebSocket connections. Yes, we did. [00:39:54] S2: Okay. [00:39:55] S1: Yeah, yeah, which is not easy to do. And so we had a solution before this customer came in, which I'm 90% sure it's public knowledge now that it's Amazon, but we had a solution that would spin up Slack as a headless Chrome instance in like a bunch of Kubernetes pods and try to run a tiny little JavaScript script and like click buttons, like pretend to be a Slack user. [00:40:23] S2: Pretend to be a user, yeah. [00:40:24] S1: Which was sort of fine, but I don't know if you've tried to wrangle more than 50,000 headless Chrome instances in Kubernetes. [00:40:38] S2: I don't think many of us have. I don't think many people listening or viewing have. [00:40:44] S1: It is not a good time. I mean, the problem with the headless Chrome instance, right, is just running Slack and 200 Chrome tabs and whatever on your laptop is kind of a time. And now you're running LLMs on top of it, but it's an experience in really pushing your laptop to the limit. And now you're running per pod, attempting to run 10,000 headless Chrome instances of Slack, which is already Very memory and CPU, mostly memory intensive application. The amount of money we were spending in these EC2 instances...
00:40:44 · Maude Lemaire, Gergely Orosz
[00:40:44] S1: It is not a good time. I mean, the problem with the headless Chrome instance, right, is just running Slack and 200 Chrome tabs and whatever on your laptop is kind of a time. And now you're running LLMs on top of it, but it's an experience in really pushing your laptop to the limit. And now you're running per pod, attempting to run 10,000 headless Chrome instances of Slack, which is already Very memory and CPU, mostly memory intensive application. The amount of money we were spending in these EC2 instances... [00:41:19] S2: The problem seems to be that per node, per VM node, depending on the size, you will be able to run, you know, you'll know, like, I don't know, like 10, 20, 50, 100. I don't know. You figure out how much, but then you pay for every one of those VMs, those nodes. You have a number of them. Yeah. I mean, look, like Slack raised a bunch of VC money, so I guess. [00:41:38] S1: Sure, sure. Yeah. [00:41:41] S2: And also from customers, of course. [00:41:43] S1: Yeah, yeah, yeah. [00:41:44] S2: Of course. That is wicked expensive. [00:41:46] S1: It is. So it was 37 cents. Per headless Chrome instance, per five-hour tranche of load testing. [00:41:54] S2: Yeah, we're talking about $150,000 for that one test. [00:41:57] S1: Yeah, yeah, yeah. Which like, okay, that's like financially not sensible. Like, it just didn't work.
00:41:57 · Maude Lemaire, Gergely Orosz
[00:41:57] S1: Yeah, yeah, yeah. Which like, okay, that's like financially not sensible. Like, it just didn't work. [00:42:04] S2: But I guess if you compare it, like, of course... I'm playing devil's advocate. $150,000 is great. But if you have a customer coming up with 500,000 seats, if they would get a discount, they would pay $5 million per month. You would think there is a point of investing to make sure that it works for them and they don't complain. I don't know where that comes in. But yes, the $150,000 still feels pretty excessive for that one test. [00:42:30] S1: Oh, yeah. Yeah. I think for that one, I mean, that's only a five-hour chunk of time. And that's assuming that you're able... Yeah, that's only five hours. [00:42:38] S2: Okay. Yeah. That's getting a problem. [00:42:41] S1: And that's assuming you're testing all of the feature tests that you wanted to run during that five hours. Which was not the case. That was not the case because we were spending half the time trying to bring pods back up because they were crashing. [00:42:55] S2: Okay. Did you figure out a better way to do it in the end? Yes.