The Pragmatic Engineer
00:14:03 · Gergely Orosz, Maude Lemaire
[00:14:03] S2: But you're saying that there was a time where this charting worked, where a large company might just have its own database instance. And for a while, they... It would work. It would work. They didn't fill it up. Yeah. And smaller companies would share, they would be multiple on the same... [00:14:16] S1: Multi-tenant on the same database instance. But you, as a customer, your data was not spread across multiple database charts. It was just on this one. It was also how we distributed messages. So for the WebSocket connection, when you were connecting... Logging into Slack, establishing that WebSocket connection, you were establishing that WebSocket connection with a server that was associated with your company. And so if everyone at your company was very busy, very busy chatting on Slack, you all were on that one same server. And again, like if we had really, really busy customers, eventually we could just say like, oh, well, this is your instance of The chat server. And that was mostly fine for a very long time. Enterprise started complicating that in some special ways in that we, I mean, again, startups, you like sort of make decisions that are fine in the moment, but then sort of don't scale. We sort of made the assumption that like inside really big companies, it was smaller companies. [00:15:20] S2: Mm hmm. [00:15:21] S1: And so like marketing at, say, IBM is its own company. So you sign into the marketing workspace at IBM or the engineering works. I mean, engineering actually is too big. It's some subset of engineering.
00:15:21 · Maude Lemaire, Gergely Orosz
[00:15:21] S1: And so like marketing at, say, IBM is its own company. So you sign into the marketing workspace at IBM or the engineering works. I mean, engineering actually is too big. It's some subset of engineering. [00:15:35] S2: It sounds like it's a little bit of wishful thinking that you could just turn this large problem into smaller problems, right? And then you can just make your current architecture work.
00:15:44 · Maude Lemaire
[00:15:44] S1: But there still needed to be like an overarching body. So we created an enterprise, which was really just another row in the team table that just said, is enterprise one? And... And then all of the teams associated with that enterprise had an enterprise ID. And you still within IBM would want to say, you know, say you were working on one of the engineering teams. I was working on a different engineering team. We want to collaborate. I want to be able to DM you. We're at the same darn company. I should be able to send you a message. We pretended that these teams within a company were their own companies. But then we had to create this other entity that overarched everything where you could still cross teams and And that got very complicated very fast. But then the part that became really terrible was everyone across all of IBM, whether they were on the marketing team, the engineering team, whatever, were all querying the enterprise shard for all of their DMs and the big company wide channels, all of that stuff, all at the same time. And then within that data, when you're booting up Slack, you have your channels that show up in the left sidebar. You'd think that it was just like, cool, get, you know, for user ID, get all channel membership entries. That was not the case. And I think it came from a good place in that DMs seemed like a separate thing from a regular public channel, right? They seem different. And so that was a separate table. And a public channel is a public channel. And so that was a separate table. But then private channels seemed different. And I think from a product perspective, you can totally see it. And from a product security perspective, you can also totally see that. You want more scrutiny when you are querying for private channels. Yep. And so that was a separate table as well. And then group DMs came about later and it seemed like they were just sort of like a private channel. And so they were just glued into the private channel table. So when you were making that query to populate your sidebar, you were actually doing a big, big old join union.
00:17:48 · Gergely Orosz
[00:17:48] S2: So what happened once a team had the idea or figured out that these tables don't make sense, but now you're talking about A lot of customers, including the likes of IBM, just putting a bunch of load on it. They're using it. How do you migrate from these three tables that are actively being used into one?
00:18:07 · Maude Lemaire
[00:18:07] S1: All the time. Yeah, I think it came about from calculating mentions. So calculating mentions to get the badge icon was also a very complex query. Whereby you had to join across all these tables, very latency sensitive. And we had gone through the process of doing as much as we possibly could at that time to ease the load. And in one of my teammates doing some work on mentions, he was like, we got to join these tables together. There's only so much we can do before we have to really start digging the hole. So I was like, sure, yeah, I will endeavor to do this very scary thing that no one else wants to do. That seems totally fine. I have so much appreciation for this in hindsight, too, in that if you look at Flamework on GitHub, which is the PHP framework that Cal Henderson open sourced after writing Flickr, it will explain many things about what Slack was like early on. Yes, I highly recommend it. It's a good, I think it's still there. It's still there. It's a good read in terms of like, oh, this explains so many things. So we had, everything was super procedural for lack of better term in terms of how code was organized. We had sort of like API entry points and it was like chat.login or whatever, right? And it was like, okay, this is the API and this is all the code for the API with some helper functions in there. There were a wild amount of Functions that were like many, many, many, many, many lines long. But there was just like straight SQL queries like embedded inside the code. Like an ORM did not exist in this world. Whatever you want to call your data access layer. We did not have one. And there were actually there was a lot of opinions about whether we should even do anything to hide the SQL queries. And Yeah, now I mean I totally understand why they had that stance and why we did the super minimal approach that we ended up doing because it is so easy to write a SQL query that will like shoot you in the foot over and over and over again. Unless you're forced to actually understand what it does in the context that it is being called. But we had like 400 something call sites that called into any number of these channel tables.
00:20:38 · Gergely Orosz, Maude Lemaire
[00:20:38] S2: And when you say call sites, these are SQL instructions in the code. [00:20:42] S1: Just like select star from Teams channels where user ID blank. [00:20:48] S2: Yeah. [00:20:48] S1: We're team ID blank, user ID blank. And so you just copy paste the pattern that exists, right? That's always the problem with migrations is like, You have to make it so clear that like this is the old pattern. Stop doing this because there is still while it's going on so many examples of it in the code. So I had to write some I started off by going through doing some kind of regex approach where I was finding all the places that we were querying for this. Thankfully, the table names were like pretty easy to find and identify. Got very lucky there. And then I just one for one took the call and just pasted it into another file. [00:21:23] S2: Yep. [00:21:23] S1: That is all I did. [00:21:24] S2: You created a layer that did the same thing. Yeah. Just a class, basically. [00:21:28] S1: Yeah. I didn't optimize anything. I literally just copy pasted and now it was calling into a function instead of straight up calling SQL. [00:21:35] S2: And you had a minimal abstraction layer, if you will.