The Pragmatic Engineer
00:11:33 · Gergely Orosz, Maude Lemaire
[00:11:33] S2: That's 100k plus people, if I recall. [00:11:36] S1: Yeah. So one of the distinctions actually was it was sort of fine to have. Tens of thousands of folks in one channel at one time. And we were sort of spreading them out across different teams within the same Slack organization. Obviously, they could DM each other among some other things, but that sort of worked. But IBM was sort of falling over on a regular basis. And I think... I was not part of these conversations, but I get the impression in hindsight that when we agreed to GA, then they agreed that they would start scaling and adding more users to Slack. Once we reach GA, we can't hold the reins on you adding more people as much. So we hit GA and they started adding more people pretty aggressively. We ended up having to spin an actual team up that was just the enterprise performance team. I ended up being able to join that team because I just hadn't to be working on odd random performance things. I was definitely the most junior person on the team. But I love that. I feel like I've spent so much of my career to date being the most junior person ever. I mean, less so in the last couple of years, but I that's how I've learned the most. And it's really just like pushed me out of my comfort zone over and over again. And it's just like it's the most it's the most fun.
00:13:02 · Gergely Orosz, Maude Lemaire
[00:13:02] S2: And can we do a bit of a Slack in the 2016-2017 Slack Architecture 101? Now, this is, as we know, Slack has changed a lot. This is not how it will look like today at all. But when we think about Slack workspaces, then channels, private groups, DMs, behind the scenes, how was it built up today? In terms of databases, in terms of what happened when you sent a message to someone or to a channel in terms of fan outs or rights? [00:13:34] S1: So the pattern before Enterprise and still with Enterprise was everything belonged to a company, a workspace, a team unit. And all of the data also was sharded by team. And so if you were on a really big team, well, we just... would give you a bigger database. And eventually, you run out of the largest size of database that you can throw money at Amazon for. [00:14:03] S2: But you're saying that there was a time where this charting worked, where a large company might just have its own database instance. And for a while, they... It would work. It would work. They didn't fill it up. Yeah. And smaller companies would share, they would be multiple on the same...
00:14:03 · Gergely Orosz, Maude Lemaire
[00:14:03] S2: But you're saying that there was a time where this charting worked, where a large company might just have its own database instance. And for a while, they... It would work. It would work. They didn't fill it up. Yeah. And smaller companies would share, they would be multiple on the same... [00:14:16] S1: Multi-tenant on the same database instance. But you, as a customer, your data was not spread across multiple database charts. It was just on this one. It was also how we distributed messages. So for the WebSocket connection, when you were connecting... Logging into Slack, establishing that WebSocket connection, you were establishing that WebSocket connection with a server that was associated with your company. And so if everyone at your company was very busy, very busy chatting on Slack, you all were on that one same server. And again, like if we had really, really busy customers, eventually we could just say like, oh, well, this is your instance of The chat server. And that was mostly fine for a very long time. Enterprise started complicating that in some special ways in that we, I mean, again, startups, you like sort of make decisions that are fine in the moment, but then sort of don't scale. We sort of made the assumption that like inside really big companies, it was smaller companies. [00:15:20] S2: Mm hmm. [00:15:21] S1: And so like marketing at, say, IBM is its own company. So you sign into the marketing workspace at IBM or the engineering works. I mean, engineering actually is too big. It's some subset of engineering.
00:15:21 · Maude Lemaire, Gergely Orosz
[00:15:21] S1: And so like marketing at, say, IBM is its own company. So you sign into the marketing workspace at IBM or the engineering works. I mean, engineering actually is too big. It's some subset of engineering. [00:15:35] S2: It sounds like it's a little bit of wishful thinking that you could just turn this large problem into smaller problems, right? And then you can just make your current architecture work.
00:15:44 · Maude Lemaire
[00:15:44] S1: But there still needed to be like an overarching body. So we created an enterprise, which was really just another row in the team table that just said, is enterprise one? And... And then all of the teams associated with that enterprise had an enterprise ID. And you still within IBM would want to say, you know, say you were working on one of the engineering teams. I was working on a different engineering team. We want to collaborate. I want to be able to DM you. We're at the same darn company. I should be able to send you a message. We pretended that these teams within a company were their own companies. But then we had to create this other entity that overarched everything where you could still cross teams and And that got very complicated very fast. But then the part that became really terrible was everyone across all of IBM, whether they were on the marketing team, the engineering team, whatever, were all querying the enterprise shard for all of their DMs and the big company wide channels, all of that stuff, all at the same time. And then within that data, when you're booting up Slack, you have your channels that show up in the left sidebar. You'd think that it was just like, cool, get, you know, for user ID, get all channel membership entries. That was not the case. And I think it came from a good place in that DMs seemed like a separate thing from a regular public channel, right? They seem different. And so that was a separate table. And a public channel is a public channel. And so that was a separate table. But then private channels seemed different. And I think from a product perspective, you can totally see it. And from a product security perspective, you can also totally see that. You want more scrutiny when you are querying for private channels. Yep. And so that was a separate table as well. And then group DMs came about later and it seemed like they were just sort of like a private channel. And so they were just glued into the private channel table. So when you were making that query to populate your sidebar, you were actually doing a big, big old join union.
00:17:48 · Gergely Orosz
[00:17:48] S2: So what happened once a team had the idea or figured out that these tables don't make sense, but now you're talking about A lot of customers, including the likes of IBM, just putting a bunch of load on it. They're using it. How do you migrate from these three tables that are actively being used into one?