The Pragmatic Engineer
00:21:35 · Gergely Orosz, Maude Lemaire
[00:21:35] S2: And you had a minimal abstraction layer, if you will. [00:21:39] S1: Yeah. Which already was like a little bit spicy. But I tried to make sure that like the function name was like real darn clear as to what it was doing. And I got the spicier folks to be OK with this approach. So I felt that felt like a win. And so I migrated everything over. Everything was feature flagged. Basically, I had I don't know how many feature flags at some point, but I had so many distinct ones just to make sure that I was doing it as carefully as humanly possible. Even then, you know, I still like definitely borked some things along The way I accidentally when I thought that we were basically there and ready to start flipping on to prod traffic, I'd run all the backfills because obviously like I was moving everything to a new table. I was doing we were doing double writes for a very long [00:22:30] S2: time to make sure the data will be up to date. [00:22:33] S1: Yes. So that's where we started. We started doing double writes, making sure that that all looked sensible. Did you [00:22:40] S2: also do shadowing? [00:22:42] S1: Yes. [00:22:43] S2: Shadowing meaning that you're doing whatever you're doing and you're doing a new operation and comparing that as the same result, right?
00:22:43 · Gergely Orosz, Maude Lemaire
[00:22:43] S2: Shadowing meaning that you're doing whatever you're doing and you're doing a new operation and comparing that as the same result, right? [00:22:50] S1: Yeah. We did a bunch of that as well. But obviously we had to make sure all the data was there first. So we did the double writing, did all the backfills. And then we were shadow reading and comparing if there were any differences, throwing away the new call, obviously, the new results, and keeping reading from the old ones. I was about to flip all of Slack to reading from the new table. [00:23:15] S2: Which, at this point, you confirmed works, so that should have been a little risk change. [00:23:18] S1: Yes. Yeah, yeah, yeah. All the data looked right. I had a bunch of data warehouse dashboards that showed me how many differences existed between the two sets of tables within Reason. And then I was about to turn it on to all of Slack. This was like Thursdays back when we had gather hour, which was like after work hours, everyone would sort of hang out if they wanted, have appetizers, whatever. And I turned it on and then I realized, oh, wait, no, something's not quite right. Yeah. [00:23:45] S2: Oh, we didn't just do a gradual rollout, because with these things, you don't really do a gradual rollout.
00:23:45 · Gergely Orosz, Maude Lemaire
[00:23:45] S2: Oh, we didn't just do a gradual rollout, because with these things, you don't really do a gradual rollout. [00:23:51] S1: Internally, we were not that big at the time. Everyone was at gather hour. I was like, oh, this seems safe. We've been shadow reading. It's fine. Yeah, I turned it on. I was like, oh, there's some data that's not quite right. And I had been so used Essentially running the script that would delete all of the backing data and rerun the backfill, or at least chunks of it, because I had been seeing discrepancies. And honestly, it was so much easier to just nuke the data, re-backfill, than it was to try to pinpoint and scalpel, fix the call sites. So I forgot that the flag was still on to everyone. And then I promptly deleted... Which [00:24:38] S2: flag was on? [00:24:39] S1: The flag to have everyone read their channel memberships from the new table. [00:24:43] S2: Okay. [00:24:43] S1: Yeah. I forgot that it was on for Slack. Not everyone externally, like just us. And I was like, oh, there's some discrepancies. Oh, right. But I fixed those. Let me just nuke the table for our team and backfill, which like took 25 minutes to do. Like it was not. [00:25:00] S2: Oh, but did everyone lose their messages until then? [00:25:03] S1: Oh, yeah. People had completely blank sidebars for about half an hour. [00:25:07] S2: Inside of all of those. Inside all of Slack. And Slack always uses Slack for everything.
00:25:07 · Gergely Orosz, Maude Lemaire
[00:25:07] S2: Inside of all of those. Inside all of Slack. And Slack always uses Slack for everything. [00:25:12] S1: For everything. But everyone was at gather hours. So there was this only one guy, Kevin, who had been sort of helping me coach through this, who DM'd me. And he was like, what's going on with my sidebar? I was like, I'm so sorry. Oh, my gosh. Let me flip the flag. So I re-flipped to read from the old data while it was backfilling, finished backfilling, and then I swapped it back. [00:25:33] S2: So what did you learn at Slack about load testing? And I'm asking because right now load is going up on all systems. Oh my gosh, yes. We see this. AI agents generate more pull requests, more code. They put more load on the systems. I assume we will have the point where we generate more data here and there. But at Slack, you have this interesting challenge where a customer with 500 I'm more interested in that, and as you built load testing, something called Koi Pond on this, but when you think of load testing, of checking how a service will be able to handle this type of load before it hits, what are the techniques that you've used and would still use today? [00:26:26] S1: Tested production. [00:26:28] S2: But how do you think the production is something where you don't have the load just yet, right?
00:26:28 · Gergely Orosz, Maude Lemaire
[00:26:28] S2: But how do you think the production is something where you don't have the load just yet, right? [00:26:32] S1: Yeah, no, I mean, I think there's a lot of different ways to do it. The simplest way that I think a lot of people can manage to do it, it depends on what kind of usage you are worried about with your product. And nailing down those usage patterns first and where you're really concerned it's going to fall over. Is the determining factor in how you figure out how to simulate load. [00:26:56] S2: In Slack specifically, what was usage patterned? [00:26:59] S1: There were a couple of key ways that it had historically fallen over. Thundering herds, where I mean, I think most people have some kind of thundering herd problem at some point where you say. [00:27:09] S2: Can we spell out what the thundering herd is just for those who have not had a herd. Suddenly herd, suddenly everyone shows up. [00:27:17] S1: Suddenly, like, that is a thundering herd. A thing happens, and then suddenly everyone shows up and does the same thing at the same time. [00:27:24] S2: I guess a good example is if your service goes down, like you're, I don't know, you're running, let's say, imaginary, like, Google search and somehow, like, All your machines die for a bug. And then you have like a thousand machines. Obviously, Google has more, but we can just imagine. And then you start to bring up machine one, machine two, machine three. But each of them gets hit by the complete traffic and they go down again, right? [00:27:47] S1: Yes. Yes. [00:27:47] S2: This is a classic example.
00:27:47 · Gergely Orosz, Maude Lemaire
[00:27:47] S2: This is a classic example. [00:27:49] S1: Yeah. Yeah. The classic example in Slack is, say, CEO says, oh, great quarter, everyone. You all get Friday off. Posts this message in the big company channel that has like 100,000 active users in it. That notification gets sent out. You see that notification. You are very motivated to plop a really good custom emoji on that notification. You click that notification. You immediately land in the channel. In the channel, you're fetching channel history. And then on top of that, you are adding an emoji reaction because you're really pumped about getting Friday off. But so is everyone else also getting, putting custom emojis on that message because they're also pumped. [00:28:27] S2: In the same five seconds.