The Pragmatic Engineer
00:09:58 · Maude Lemaire, Gergely Orosz
[00:09:58] S1: Yeah, yeah. [00:09:59] S2: And... What were the problems you had to work on that you got to work? [00:10:04] S1: Yes. Yeah. Yeah. I very much got the performance itch from Runway. It just wasn't. I just had to work on it, you know, on my own time, but I still enjoyed doing it. And then when I showed up at Slack, I my onboarding buddy was just like very senior engineer who was working on performance issues specifically at the time. And I burned through some of my starter stuff. And I was like, what are you working on? What are you working on over there, Moisha? And he was like, oh, let me send you some tickets. Like, do you want to work on performance? [00:10:34] S2: I was like, yes, yes, I do. [00:10:37] S1: So he sent me a couple things, like relatively easy stuff. In startup land, when you... Need to get to the next place and just get that next customer or build that next feature that you think is going to capture. Like you make a lot of short term decisions that, you know, probably inside your soul you will revisit at some point. And a lot of the performance work that we were doing at the time was just a lot of revisiting of shortcuts. I mean, like well calculated shortcuts, but like shortcuts that we knew were going to like bite us in the butt eventually down the line. So a lot of that work I was doing. And this was before the like enterprise, the Slack enterprise product was launched. But it started a couple months before GA. IBM was already trialing the product. They were sort of on beta there. They are a large company. Lots of people work there.
00:10:37 · Maude Lemaire, Gergely Orosz
[00:10:37] S1: So he sent me a couple things, like relatively easy stuff. In startup land, when you... Need to get to the next place and just get that next customer or build that next feature that you think is going to capture. Like you make a lot of short term decisions that, you know, probably inside your soul you will revisit at some point. And a lot of the performance work that we were doing at the time was just a lot of revisiting of shortcuts. I mean, like well calculated shortcuts, but like shortcuts that we knew were going to like bite us in the butt eventually down the line. So a lot of that work I was doing. And this was before the like enterprise, the Slack enterprise product was launched. But it started a couple months before GA. IBM was already trialing the product. They were sort of on beta there. They are a large company. Lots of people work there. [00:11:33] S2: That's 100k plus people, if I recall.
00:11:33 · Gergely Orosz, Maude Lemaire
[00:11:33] S2: That's 100k plus people, if I recall. [00:11:36] S1: Yeah. So one of the distinctions actually was it was sort of fine to have. Tens of thousands of folks in one channel at one time. And we were sort of spreading them out across different teams within the same Slack organization. Obviously, they could DM each other among some other things, but that sort of worked. But IBM was sort of falling over on a regular basis. And I think... I was not part of these conversations, but I get the impression in hindsight that when we agreed to GA, then they agreed that they would start scaling and adding more users to Slack. Once we reach GA, we can't hold the reins on you adding more people as much. So we hit GA and they started adding more people pretty aggressively. We ended up having to spin an actual team up that was just the enterprise performance team. I ended up being able to join that team because I just hadn't to be working on odd random performance things. I was definitely the most junior person on the team. But I love that. I feel like I've spent so much of my career to date being the most junior person ever. I mean, less so in the last couple of years, but I that's how I've learned the most. And it's really just like pushed me out of my comfort zone over and over again. And it's just like it's the most it's the most fun.
00:13:02 · Gergely Orosz, Maude Lemaire
[00:13:02] S2: And can we do a bit of a Slack in the 2016-2017 Slack Architecture 101? Now, this is, as we know, Slack has changed a lot. This is not how it will look like today at all. But when we think about Slack workspaces, then channels, private groups, DMs, behind the scenes, how was it built up today? In terms of databases, in terms of what happened when you sent a message to someone or to a channel in terms of fan outs or rights? [00:13:34] S1: So the pattern before Enterprise and still with Enterprise was everything belonged to a company, a workspace, a team unit. And all of the data also was sharded by team. And so if you were on a really big team, well, we just... would give you a bigger database. And eventually, you run out of the largest size of database that you can throw money at Amazon for. [00:14:03] S2: But you're saying that there was a time where this charting worked, where a large company might just have its own database instance. And for a while, they... It would work. It would work. They didn't fill it up. Yeah. And smaller companies would share, they would be multiple on the same...
00:14:03 · Gergely Orosz, Maude Lemaire
[00:14:03] S2: But you're saying that there was a time where this charting worked, where a large company might just have its own database instance. And for a while, they... It would work. It would work. They didn't fill it up. Yeah. And smaller companies would share, they would be multiple on the same... [00:14:16] S1: Multi-tenant on the same database instance. But you, as a customer, your data was not spread across multiple database charts. It was just on this one. It was also how we distributed messages. So for the WebSocket connection, when you were connecting... Logging into Slack, establishing that WebSocket connection, you were establishing that WebSocket connection with a server that was associated with your company. And so if everyone at your company was very busy, very busy chatting on Slack, you all were on that one same server. And again, like if we had really, really busy customers, eventually we could just say like, oh, well, this is your instance of The chat server. And that was mostly fine for a very long time. Enterprise started complicating that in some special ways in that we, I mean, again, startups, you like sort of make decisions that are fine in the moment, but then sort of don't scale. We sort of made the assumption that like inside really big companies, it was smaller companies. [00:15:20] S2: Mm hmm. [00:15:21] S1: And so like marketing at, say, IBM is its own company. So you sign into the marketing workspace at IBM or the engineering works. I mean, engineering actually is too big. It's some subset of engineering.
00:15:21 · Maude Lemaire, Gergely Orosz
[00:15:21] S1: And so like marketing at, say, IBM is its own company. So you sign into the marketing workspace at IBM or the engineering works. I mean, engineering actually is too big. It's some subset of engineering. [00:15:35] S2: It sounds like it's a little bit of wishful thinking that you could just turn this large problem into smaller problems, right? And then you can just make your current architecture work.