The Pragmatic Engineer
00:56:08 · Maude Lemaire
[00:56:08] S1: Yeah, I mean, it does a very good job at hiding the underlying data model from you, which, again, pros and cons. I came from a world where, like, SQL was, you know, religiously written out by hand for performance reasons. And then I was being... Thrown into this world where at one point we were trying to figure out, we were doing a bunch of work around domain extraction. And so how do we rip out everything related to, say, pull requests from the Rails monolith and extract that into a separate Go subsystem, take everything that deals with pull requests, just make the requests go through, say, the monolith still, but then just immediately send them over to the other system. Really standard sort of migration procedure. Started doing that. We started with stars and I think that project was eventually abandoned, but I was working, I was pairing with someone who had like a ton of Rails experience. And I was like, okay, you gotta like, because I, as someone who was so used to reading just code that was very sort of straightforward and didn't have so many levels of abstraction. It's a little bit daunting when you're going into a world where now there's abstractions upon abstractions upon abstractions. And I was working with her to try to rip out more of the STARS-related queries inside of the Rails model. We'd found sort of the root caller that created this query that would call into the STARS table. And it was like a five-line function. And I was looking at it with her on Zoom. And I said, wait, Kate, where... Where is the call? Like, where is the sequel being made? Like, what is happening? It's a five-line function. And she's like, Mo, that's the million-dollar question. And I was like, what are we looking at? What is happening? Because I half expected to see... I mean, at this point, I think so many people use Prisma, right? And Prisma, you can sort of... It's not, you know... Plain SQL written out as a string between quotation marks, but you can sort of see the structure of the SQL query being made. You do not get any of that in this case. It just looks like a regular function call. And in that moment is when I realized this is why this has been impossible to date. And why the problem has gotten so bad, because no one sees the impact Just writing a new line of code.
00:58:39 · Gergely Orosz, Maude Lemaire
[00:58:39] S2: And also fixing it now will probably, it's not as easy as just tweaking your abstraction layer. You would probably have to change how code is being written, being called, starting to call more either functions that call SQL, et cetera. Like it's just, you took a technology that worked great when you were small, you were able to scale it as long as you, and they figured out how to grow it as long as things are going linearly. [00:59:04] S1: Yeah, and I think one of the things that blew my mind the most that both gave me a lot of respect for how they've managed to date and also filled me with much dread was when you work inside of GitHub, obviously you have the admin view of GitHub and there's this bar at the top where you can load up a page on GitHub and it'll tell you exactly Wish MySQL queries were run to populate the page, which clusters they hit, how fast they all were, totals around all of this stuff. [00:59:38] S2: Just like the internal employee view. Yeah. [00:59:40] S1: Yeah, yeah, yeah. And what's called the files disambiguated page, which is sort of like the repo homepage that you land on with the readme. That page last I checked before I left. [00:59:51] S2: Before you left like a year ago?
00:59:51 · Gergely Orosz, Maude Lemaire
[00:59:51] S2: Before you left like a year ago? [00:59:52] S1: Actually less than that. But yeah, yeah, yeah. So some amount of time ago, it was over 100 MySQL queries to populate that page across at least three different MySQL clusters. And the thing with ActiveRecord is that it only caches the output automatically if the query is exactly the same. And so if, say, you're fetching user information, like full blanket, like select star from users where user ID equals blank, and you have that full user, but then later on you do select star from users where user ID equals blank and something else, it's not going to reach back into the results that it cashed for that first query because they are not exact same queries. Every single count. Or at least when I last left. Almost every single count you see on a GitHub page is live calculated. None of those are cached. There's a lot of reasons why that is. Mostly product reasons, actually. But that's one of the first things that I was like, but why? And to go back to like my sort of silly emoji example of like the 10,000 in month emoji, no one cares. Why does anyone care, right? And if you have low counts of things, sure, okay, maybe it matters that there's three open pull requests on like this very small repo that like no one really does anything with. But once it gets to like 99, don't just, don't show anything, just say 99 plus. Don't bother counting it. Why are we counting this?
01:01:23 · Gergely Orosz, Maude Lemaire, Gergely Orosz
[01:01:23] S2: Really interesting that So far, we talked about you did interesting things at Slack, at GitHub, you also worked at Copilot Code Review. Afterwards, you worked at Cursor. Again, they really pushed how we're coding with AI tools. But so far, we've not really talked about AI. It just didn't come up naturally. And I wanted to ask... Firstly, what do you think AI so far? You obviously use these tools all the time, increasingly, intensively. What parts of the job does AI not help you with? And what parts of it has it helped and has it changed how you work? [01:02:02] S1: Oh, gosh, yeah. It has not helped me with performance optimizations, basically, at all. At least the investigation part. Maybe the implementation part is a little bit faster. [01:02:14] S2: Once you know what you want to do. [01:02:16] S1: But yeah, figuring out what I want to do. So say I was looking at improving time to first token, time to first visible token. [01:02:24] S0: At Cursor.
01:02:24 · Gergely Orosz, Maude Lemaire, Gergely Orosz
[01:02:24] S0: At Cursor. [01:02:24] S1: At Cursor a couple of months ago where and there's a bunch of factors that feed into how fast you start getting those tokens back. There's a significant amount of time that goes into when you are typing your prompt. And you hit submit. There's a significant amount of time that goes into building all of the context associated with the request that you are about to make to send to the LLM. And then there's some amount of data that we are able to cache, some amount that we are not able to cache, either for product reasons, privacy reasons, but also strictly because of load and, again, cost of these systems. It is not cheap to say cache absolutely everything that you have ever queried cursor for in Redis for 24 hours. And it costs a lot of money. [01:03:15] S2: So we're not going to do that.
01:03:17 · Maude Lemaire
[01:03:17] S1: And then eventually sending that request over to inference proxy, which then sends the request over to the underlying LLMs and providers. Given that there's so much that happens, but also back and forth, a lot of this process is not really as synchronous as the experience leads you to believe. And in the regular flow, if you're lucky enough to be able to use HTTP2, which some poor folks out there are stuck on HTTP1, you establish that gRPC streaming connection with the backend servers, and you're able to split that connection. Yeah. Which is why some of the skills that I ended up crafting run through intensive series of mutation tests on the code to then force the LLM to realize, oh, I didn't actually test any of the edge cases. Let me rewrite those tests. Thank you. So the LLMs right now I find are... Good at doing the really basic level of instrumentation, but not great at figuring out, especially in an asynchronous context, what makes sense to instrument how. I love traces. I think they're amazing. You can get so much really good debugging information from them, particularly from a performance perspective. We had this problem at Cursor where we had like Too many and too little traces. And by that, I mean that we had traces with millions upon millions of spans, which are completely useless. You cannot do anything meaningful with a trace that has that many spans. And then we also did the very classic mistake that basically everyone does, which is we started sampling for cost reasons, but Not in one place, which is, again, really easy to have an LLM do because it's like, oh, like, find me a place where we can send fewer traces to save money. And it's like, cool, I'll just like sample it right before I send it to Datadog or whatever. No, no, no, don't do that.