The Pragmatic Engineer
00:48:41 · Gergely Orosz, Maude Lemaire
[00:48:41] S2: They used to have their own data centers. Now they're moving to Azure. This is public information they shared. [00:48:47] S1: Yes. Yeah. The move to Azure is front and center, but still a significant or at least at the time when we were talking, a still significant amount of load was going through the data centers that like they had whole teams managing the metal. And she was like, we're going to run out of space. The space is running out. And we either well, we both have to figure out how to move to cloud. And we also have to figure out how to do less because we're doing a bunch of dumb stuff, as everyone does, that we shouldn't be doing. And we can make the system a lot more efficient with what we have to give bias more runway. I was like, oh, this sounds like a very interesting problem. Would love to work on this problem. But I was also seven months pregnant at the time. And so what that ended up being was I showed up. I was seven months pregnant, which is actually kind of, if you can swing it, a really great time way to join a company because you are you sit there and you're like, cool, I will just drop a bunch of opinions like, you know, nothing very strongly held, but I will just drop a bunch of opinions on people. I will own nothing for the next two months and then I'm going to go have a baby and then I'll come back. [00:50:01] S2: So what happened?
00:50:01 · Gergely Orosz, Maude Lemaire
[00:50:01] S2: So what happened? [00:50:02] S1: Let's see what happened. And in my first couple of weeks, I was sort of floating around some spaces. And having done a bunch of database migrations at Slack and some of the stuff that we didn't talk about, but was moving from this team-sharded, tenant-based model to a more dynamically-sharded model with Vitess, where it turns out, no, a team was too big. [00:50:27] S2: So you mentioned Vitess. What is Vitess? [00:50:30] S1: Vitess is essentially a wrapper on top of MySQL that lets you shard more dynamically. [00:50:36] S2: And so you have a- So instead of just having MySQL run on one database, and when you outgrow it, good luck. You can re-shard, redistribute your data a lot more seamlessly than you would be able to otherwise. And you can choose different sharding keys more easily as well.
00:51:09 · Maude Lemaire
[00:51:09] S1: Before, it's like everything was started by team. There's still data that is started by team, where team is a logical unit that makes the most sense for fetching that data. But actually, it turns out that the smallest, most logical unit was channel. And so we ended up resharding a bunch of our data to be fetchable by channel ID rather than by team ID. And then that allows for much better horizontal sharding structure as well. And the idea being that there is no channel that will ever be big enough, maybe one day it will be, the bots keep posting in the channels maybe, that will have enough data associated with it that it can never run on a single ultra-large database, right? It's a much better unit from a scaling perspective. So Vitess has had some of its nascency embedded in GitHub, actually. And it was really interesting because when I was interviewing at GitHub, I spoke with a bunch of the principal engineers that I would later end up working with. And they were sort of trying to figure out, OK, we have Vitess, but I don't think we're actually using it to its potential to where we should be using it. And so they were asking me a bunch of questions about like, are Vitess set up and what we were doing? Caveating this with, I did not ever operate Vitess or deal with the internals. I was more of a, I'm a product platform person. And so I just like find that like, no, we should not be starting this thing by this thing anymore. Let's migrate it. And so that was my perspective on it, which was also mostly their perspective on it as well. And I came in and I started poking at the Vitess setup and I was like, what are we doing? What are we doing here? I was used to thousands of shards per sharding key. So say four channels, we have a bajillion. Because there's a bajillion channels and you want to sort of spread load evenly throughout those. And when I arrived, I was looking at the Vitess setup and there was, I mean, I don't know if this is still the case, but there were 16 shards for all of pull requests and issues.
00:53:14 · Gergely Orosz, Maude Lemaire
[00:53:14] S2: Across all of GitHub. [00:53:16] S1: Across all of GitHub. Only 16, which is not very many given the amount of data that we know is associated with that whole space. And my first question was like, wait, why do we only have 16 charts? Oh my gosh, this makes me deeply, deeply nervous. And part of the answer was, we can't have more because we already have so many scatter queries. So those are queries that are coming in. [00:53:43] S2: What are scatter queries? [00:53:45] S1: So those are queries that are coming in where you do not have the primary key in them. So say you are querying for a pull request or an issue, and you don't have the pull request ID, which is how this data is being distributed. Well, you have to find it. So you have to go... You have to find and match the criteria. So say I'm querying for all pull requests authored by me. I don't know that that's an actual scatterer, but this could be in the scenario. Let's just assume. And so because I don't have a pull request ID, well, I'm going to hit every single one of those 16 shards. [00:54:23] S2: Wow, this will be bad in so many levels. [00:54:27] S1: Uh-huh. Because, well, so many people are doing it also. [00:54:29] S2: And I understand then why you don't want to have many shards because if you have more shards, you would need to hit all of them. We're talking like, it's almost like a linear scan.
00:54:29 · Gergely Orosz, Maude Lemaire
[00:54:29] S2: And I understand then why you don't want to have many shards because if you have more shards, you would need to hit all of them. We're talking like, it's almost like a linear scan. [00:54:37] S1: Yeah. Yeah. Oh, no, it's quite, quite, quite inefficient. And again, I don't know that this is an actual example of one of the- This was hypothetical. This is completely hypothetical, but there were still very many significant scatter queries that were hitting almost every single shard before it knew that it could complete the query and return the result. Which, yes, I understand why that was a constraint then. You don't want to add more shards. But then again, you also cannot make the shards much bigger and they're under very heavy load. And so I was ready to apply the playbook that I'd been using for the last several years of like, well, let's [00:55:12] S2: just burn down the scatters. Let's just get rid of the scatter queries. Don't have scatter queries.
00:55:12 · Gergely Orosz, Maude Lemaire
[00:55:12] S2: just burn down the scatters. Let's just get rid of the scatter queries. Don't have scatter queries. [00:55:17] S1: Just don't have scatter queries. That can't be that hard. Well, it turns out that can be very hard when you're dealing with a... Rails, Monolith, and you've got ActiveRecord. And I always sort of thought it was funny, the number of, there were a lot of ex-Twitter folks at Slack early on, and everyone was like, haha, ActiveRecord. And I was like, what are you, like before they did the whole like Scala Java thing, right? And I was like, okay, I guess like we're all, you know, joking about ActiveRecord. I did not understand. Yeah. Until I like came face to face with it and I was like, oh, I really don't like this. This is really terrible. [00:55:54] S2: So like the record is Rails' abstraction layer for the database, right? So you define things in, I guess, code and it kind of maps nicely to database, except it might be harder to fine tune your queries, et cetera.