The Pragmatic Engineer
00:18:07 · Maude Lemaire
[00:18:07] S1: All the time. Yeah, I think it came about from calculating mentions. So calculating mentions to get the badge icon was also a very complex query. Whereby you had to join across all these tables, very latency sensitive. And we had gone through the process of doing as much as we possibly could at that time to ease the load. And in one of my teammates doing some work on mentions, he was like, we got to join these tables together. There's only so much we can do before we have to really start digging the hole. So I was like, sure, yeah, I will endeavor to do this very scary thing that no one else wants to do. That seems totally fine. I have so much appreciation for this in hindsight, too, in that if you look at Flamework on GitHub, which is the PHP framework that Cal Henderson open sourced after writing Flickr, it will explain many things about what Slack was like early on. Yes, I highly recommend it. It's a good, I think it's still there. It's still there. It's a good read in terms of like, oh, this explains so many things. So we had, everything was super procedural for lack of better term in terms of how code was organized. We had sort of like API entry points and it was like chat.login or whatever, right? And it was like, okay, this is the API and this is all the code for the API with some helper functions in there. There were a wild amount of Functions that were like many, many, many, many, many lines long. But there was just like straight SQL queries like embedded inside the code. Like an ORM did not exist in this world. Whatever you want to call your data access layer. We did not have one. And there were actually there was a lot of opinions about whether we should even do anything to hide the SQL queries. And Yeah, now I mean I totally understand why they had that stance and why we did the super minimal approach that we ended up doing because it is so easy to write a SQL query that will like shoot you in the foot over and over and over again. Unless you're forced to actually understand what it does in the context that it is being called. But we had like 400 something call sites that called into any number of these channel tables.
00:20:38 · Gergely Orosz, Maude Lemaire
[00:20:38] S2: And when you say call sites, these are SQL instructions in the code. [00:20:42] S1: Just like select star from Teams channels where user ID blank. [00:20:48] S2: Yeah. [00:20:48] S1: We're team ID blank, user ID blank. And so you just copy paste the pattern that exists, right? That's always the problem with migrations is like, You have to make it so clear that like this is the old pattern. Stop doing this because there is still while it's going on so many examples of it in the code. So I had to write some I started off by going through doing some kind of regex approach where I was finding all the places that we were querying for this. Thankfully, the table names were like pretty easy to find and identify. Got very lucky there. And then I just one for one took the call and just pasted it into another file. [00:21:23] S2: Yep. [00:21:23] S1: That is all I did. [00:21:24] S2: You created a layer that did the same thing. Yeah. Just a class, basically. [00:21:28] S1: Yeah. I didn't optimize anything. I literally just copy pasted and now it was calling into a function instead of straight up calling SQL. [00:21:35] S2: And you had a minimal abstraction layer, if you will.
00:21:35 · Gergely Orosz, Maude Lemaire
[00:21:35] S2: And you had a minimal abstraction layer, if you will. [00:21:39] S1: Yeah. Which already was like a little bit spicy. But I tried to make sure that like the function name was like real darn clear as to what it was doing. And I got the spicier folks to be OK with this approach. So I felt that felt like a win. And so I migrated everything over. Everything was feature flagged. Basically, I had I don't know how many feature flags at some point, but I had so many distinct ones just to make sure that I was doing it as carefully as humanly possible. Even then, you know, I still like definitely borked some things along The way I accidentally when I thought that we were basically there and ready to start flipping on to prod traffic, I'd run all the backfills because obviously like I was moving everything to a new table. I was doing we were doing double writes for a very long [00:22:30] S2: time to make sure the data will be up to date. [00:22:33] S1: Yes. So that's where we started. We started doing double writes, making sure that that all looked sensible. Did you [00:22:40] S2: also do shadowing? [00:22:42] S1: Yes. [00:22:43] S2: Shadowing meaning that you're doing whatever you're doing and you're doing a new operation and comparing that as the same result, right?
00:22:43 · Gergely Orosz, Maude Lemaire
[00:22:43] S2: Shadowing meaning that you're doing whatever you're doing and you're doing a new operation and comparing that as the same result, right? [00:22:50] S1: Yeah. We did a bunch of that as well. But obviously we had to make sure all the data was there first. So we did the double writing, did all the backfills. And then we were shadow reading and comparing if there were any differences, throwing away the new call, obviously, the new results, and keeping reading from the old ones. I was about to flip all of Slack to reading from the new table. [00:23:15] S2: Which, at this point, you confirmed works, so that should have been a little risk change. [00:23:18] S1: Yes. Yeah, yeah, yeah. All the data looked right. I had a bunch of data warehouse dashboards that showed me how many differences existed between the two sets of tables within Reason. And then I was about to turn it on to all of Slack. This was like Thursdays back when we had gather hour, which was like after work hours, everyone would sort of hang out if they wanted, have appetizers, whatever. And I turned it on and then I realized, oh, wait, no, something's not quite right. Yeah. [00:23:45] S2: Oh, we didn't just do a gradual rollout, because with these things, you don't really do a gradual rollout.
00:23:45 · Gergely Orosz, Maude Lemaire
[00:23:45] S2: Oh, we didn't just do a gradual rollout, because with these things, you don't really do a gradual rollout. [00:23:51] S1: Internally, we were not that big at the time. Everyone was at gather hour. I was like, oh, this seems safe. We've been shadow reading. It's fine. Yeah, I turned it on. I was like, oh, there's some data that's not quite right. And I had been so used Essentially running the script that would delete all of the backing data and rerun the backfill, or at least chunks of it, because I had been seeing discrepancies. And honestly, it was so much easier to just nuke the data, re-backfill, than it was to try to pinpoint and scalpel, fix the call sites. So I forgot that the flag was still on to everyone. And then I promptly deleted... Which [00:24:38] S2: flag was on? [00:24:39] S1: The flag to have everyone read their channel memberships from the new table. [00:24:43] S2: Okay. [00:24:43] S1: Yeah. I forgot that it was on for Slack. Not everyone externally, like just us. And I was like, oh, there's some discrepancies. Oh, right. But I fixed those. Let me just nuke the table for our team and backfill, which like took 25 minutes to do. Like it was not. [00:25:00] S2: Oh, but did everyone lose their messages until then? [00:25:03] S1: Oh, yeah. People had completely blank sidebars for about half an hour. [00:25:07] S2: Inside of all of those. Inside all of Slack. And Slack always uses Slack for everything.
00:25:07 · Gergely Orosz, Maude Lemaire
[00:25:07] S2: Inside of all of those. Inside all of Slack. And Slack always uses Slack for everything. [00:25:12] S1: For everything. But everyone was at gather hours. So there was this only one guy, Kevin, who had been sort of helping me coach through this, who DM'd me. And he was like, what's going on with my sidebar? I was like, I'm so sorry. Oh, my gosh. Let me flip the flag. So I re-flipped to read from the old data while it was backfilling, finished backfilling, and then I swapped it back. [00:25:33] S2: So what did you learn at Slack about load testing? And I'm asking because right now load is going up on all systems. Oh my gosh, yes. We see this. AI agents generate more pull requests, more code. They put more load on the systems. I assume we will have the point where we generate more data here and there. But at Slack, you have this interesting challenge where a customer with 500 I'm more interested in that, and as you built load testing, something called Koi Pond on this, but when you think of load testing, of checking how a service will be able to handle this type of load before it hits, what are the techniques that you've used and would still use today? [00:26:26] S1: Tested production. [00:26:28] S2: But how do you think the production is something where you don't have the load just yet, right?