The Pragmatic Engineer
00:15:44 · Maude Lemaire
[00:15:44] S1: But there still needed to be like an overarching body. So we created an enterprise, which was really just another row in the team table that just said, is enterprise one? And... And then all of the teams associated with that enterprise had an enterprise ID. And you still within IBM would want to say, you know, say you were working on one of the engineering teams. I was working on a different engineering team. We want to collaborate. I want to be able to DM you. We're at the same darn company. I should be able to send you a message. We pretended that these teams within a company were their own companies. But then we had to create this other entity that overarched everything where you could still cross teams and And that got very complicated very fast. But then the part that became really terrible was everyone across all of IBM, whether they were on the marketing team, the engineering team, whatever, were all querying the enterprise shard for all of their DMs and the big company wide channels, all of that stuff, all at the same time. And then within that data, when you're booting up Slack, you have your channels that show up in the left sidebar. You'd think that it was just like, cool, get, you know, for user ID, get all channel membership entries. That was not the case. And I think it came from a good place in that DMs seemed like a separate thing from a regular public channel, right? They seem different. And so that was a separate table. And a public channel is a public channel. And so that was a separate table. But then private channels seemed different. And I think from a product perspective, you can totally see it. And from a product security perspective, you can also totally see that. You want more scrutiny when you are querying for private channels. Yep. And so that was a separate table as well. And then group DMs came about later and it seemed like they were just sort of like a private channel. And so they were just glued into the private channel table. So when you were making that query to populate your sidebar, you were actually doing a big, big old join union.
00:17:48 · Gergely Orosz
[00:17:48] S2: So what happened once a team had the idea or figured out that these tables don't make sense, but now you're talking about A lot of customers, including the likes of IBM, just putting a bunch of load on it. They're using it. How do you migrate from these three tables that are actively being used into one?
00:18:07 · Maude Lemaire
[00:18:07] S1: All the time. Yeah, I think it came about from calculating mentions. So calculating mentions to get the badge icon was also a very complex query. Whereby you had to join across all these tables, very latency sensitive. And we had gone through the process of doing as much as we possibly could at that time to ease the load. And in one of my teammates doing some work on mentions, he was like, we got to join these tables together. There's only so much we can do before we have to really start digging the hole. So I was like, sure, yeah, I will endeavor to do this very scary thing that no one else wants to do. That seems totally fine. I have so much appreciation for this in hindsight, too, in that if you look at Flamework on GitHub, which is the PHP framework that Cal Henderson open sourced after writing Flickr, it will explain many things about what Slack was like early on. Yes, I highly recommend it. It's a good, I think it's still there. It's still there. It's a good read in terms of like, oh, this explains so many things. So we had, everything was super procedural for lack of better term in terms of how code was organized. We had sort of like API entry points and it was like chat.login or whatever, right? And it was like, okay, this is the API and this is all the code for the API with some helper functions in there. There were a wild amount of Functions that were like many, many, many, many, many lines long. But there was just like straight SQL queries like embedded inside the code. Like an ORM did not exist in this world. Whatever you want to call your data access layer. We did not have one. And there were actually there was a lot of opinions about whether we should even do anything to hide the SQL queries. And Yeah, now I mean I totally understand why they had that stance and why we did the super minimal approach that we ended up doing because it is so easy to write a SQL query that will like shoot you in the foot over and over and over again. Unless you're forced to actually understand what it does in the context that it is being called. But we had like 400 something call sites that called into any number of these channel tables.
00:20:38 · Gergely Orosz, Maude Lemaire
[00:20:38] S2: And when you say call sites, these are SQL instructions in the code. [00:20:42] S1: Just like select star from Teams channels where user ID blank. [00:20:48] S2: Yeah. [00:20:48] S1: We're team ID blank, user ID blank. And so you just copy paste the pattern that exists, right? That's always the problem with migrations is like, You have to make it so clear that like this is the old pattern. Stop doing this because there is still while it's going on so many examples of it in the code. So I had to write some I started off by going through doing some kind of regex approach where I was finding all the places that we were querying for this. Thankfully, the table names were like pretty easy to find and identify. Got very lucky there. And then I just one for one took the call and just pasted it into another file. [00:21:23] S2: Yep. [00:21:23] S1: That is all I did. [00:21:24] S2: You created a layer that did the same thing. Yeah. Just a class, basically. [00:21:28] S1: Yeah. I didn't optimize anything. I literally just copy pasted and now it was calling into a function instead of straight up calling SQL. [00:21:35] S2: And you had a minimal abstraction layer, if you will.
00:21:35 · Gergely Orosz, Maude Lemaire
[00:21:35] S2: And you had a minimal abstraction layer, if you will. [00:21:39] S1: Yeah. Which already was like a little bit spicy. But I tried to make sure that like the function name was like real darn clear as to what it was doing. And I got the spicier folks to be OK with this approach. So I felt that felt like a win. And so I migrated everything over. Everything was feature flagged. Basically, I had I don't know how many feature flags at some point, but I had so many distinct ones just to make sure that I was doing it as carefully as humanly possible. Even then, you know, I still like definitely borked some things along The way I accidentally when I thought that we were basically there and ready to start flipping on to prod traffic, I'd run all the backfills because obviously like I was moving everything to a new table. I was doing we were doing double writes for a very long [00:22:30] S2: time to make sure the data will be up to date. [00:22:33] S1: Yes. So that's where we started. We started doing double writes, making sure that that all looked sensible. Did you [00:22:40] S2: also do shadowing? [00:22:42] S1: Yes. [00:22:43] S2: Shadowing meaning that you're doing whatever you're doing and you're doing a new operation and comparing that as the same result, right?
00:22:43 · Gergely Orosz, Maude Lemaire
[00:22:43] S2: Shadowing meaning that you're doing whatever you're doing and you're doing a new operation and comparing that as the same result, right? [00:22:50] S1: Yeah. We did a bunch of that as well. But obviously we had to make sure all the data was there first. So we did the double writing, did all the backfills. And then we were shadow reading and comparing if there were any differences, throwing away the new call, obviously, the new results, and keeping reading from the old ones. I was about to flip all of Slack to reading from the new table. [00:23:15] S2: Which, at this point, you confirmed works, so that should have been a little risk change. [00:23:18] S1: Yes. Yeah, yeah, yeah. All the data looked right. I had a bunch of data warehouse dashboards that showed me how many differences existed between the two sets of tables within Reason. And then I was about to turn it on to all of Slack. This was like Thursdays back when we had gather hour, which was like after work hours, everyone would sort of hang out if they wanted, have appetizers, whatever. And I turned it on and then I realized, oh, wait, no, something's not quite right. Yeah. [00:23:45] S2: Oh, we didn't just do a gradual rollout, because with these things, you don't really do a gradual rollout.