The Pragmatic Engineer
00:59:51 · Gergely Orosz, Maude Lemaire
[00:59:51] S2: Before you left like a year ago? [00:59:52] S1: Actually less than that. But yeah, yeah, yeah. So some amount of time ago, it was over 100 MySQL queries to populate that page across at least three different MySQL clusters. And the thing with ActiveRecord is that it only caches the output automatically if the query is exactly the same. And so if, say, you're fetching user information, like full blanket, like select star from users where user ID equals blank, and you have that full user, but then later on you do select star from users where user ID equals blank and something else, it's not going to reach back into the results that it cashed for that first query because they are not exact same queries. Every single count. Or at least when I last left. Almost every single count you see on a GitHub page is live calculated. None of those are cached. There's a lot of reasons why that is. Mostly product reasons, actually. But that's one of the first things that I was like, but why? And to go back to like my sort of silly emoji example of like the 10,000 in month emoji, no one cares. Why does anyone care, right? And if you have low counts of things, sure, okay, maybe it matters that there's three open pull requests on like this very small repo that like no one really does anything with. But once it gets to like 99, don't just, don't show anything, just say 99 plus. Don't bother counting it. Why are we counting this?
01:01:23 · Gergely Orosz, Maude Lemaire, Gergely Orosz
[01:01:23] S2: Really interesting that So far, we talked about you did interesting things at Slack, at GitHub, you also worked at Copilot Code Review. Afterwards, you worked at Cursor. Again, they really pushed how we're coding with AI tools. But so far, we've not really talked about AI. It just didn't come up naturally. And I wanted to ask... Firstly, what do you think AI so far? You obviously use these tools all the time, increasingly, intensively. What parts of the job does AI not help you with? And what parts of it has it helped and has it changed how you work? [01:02:02] S1: Oh, gosh, yeah. It has not helped me with performance optimizations, basically, at all. At least the investigation part. Maybe the implementation part is a little bit faster. [01:02:14] S2: Once you know what you want to do. [01:02:16] S1: But yeah, figuring out what I want to do. So say I was looking at improving time to first token, time to first visible token. [01:02:24] S0: At Cursor.
01:02:24 · Gergely Orosz, Maude Lemaire, Gergely Orosz
[01:02:24] S0: At Cursor. [01:02:24] S1: At Cursor a couple of months ago where and there's a bunch of factors that feed into how fast you start getting those tokens back. There's a significant amount of time that goes into when you are typing your prompt. And you hit submit. There's a significant amount of time that goes into building all of the context associated with the request that you are about to make to send to the LLM. And then there's some amount of data that we are able to cache, some amount that we are not able to cache, either for product reasons, privacy reasons, but also strictly because of load and, again, cost of these systems. It is not cheap to say cache absolutely everything that you have ever queried cursor for in Redis for 24 hours. And it costs a lot of money. [01:03:15] S2: So we're not going to do that.
01:03:17 · Maude Lemaire
[01:03:17] S1: And then eventually sending that request over to inference proxy, which then sends the request over to the underlying LLMs and providers. Given that there's so much that happens, but also back and forth, a lot of this process is not really as synchronous as the experience leads you to believe. And in the regular flow, if you're lucky enough to be able to use HTTP2, which some poor folks out there are stuck on HTTP1, you establish that gRPC streaming connection with the backend servers, and you're able to split that connection. Yeah. Which is why some of the skills that I ended up crafting run through intensive series of mutation tests on the code to then force the LLM to realize, oh, I didn't actually test any of the edge cases. Let me rewrite those tests. Thank you. So the LLMs right now I find are... Good at doing the really basic level of instrumentation, but not great at figuring out, especially in an asynchronous context, what makes sense to instrument how. I love traces. I think they're amazing. You can get so much really good debugging information from them, particularly from a performance perspective. We had this problem at Cursor where we had like Too many and too little traces. And by that, I mean that we had traces with millions upon millions of spans, which are completely useless. You cannot do anything meaningful with a trace that has that many spans. And then we also did the very classic mistake that basically everyone does, which is we started sampling for cost reasons, but Not in one place, which is, again, really easy to have an LLM do because it's like, oh, like, find me a place where we can send fewer traces to save money. And it's like, cool, I'll just like sample it right before I send it to Datadog or whatever. No, no, no, don't do that.
01:05:43 · Gergely Orosz, Maude Lemaire
[01:05:43] S2: Please don't do that. [01:05:44] S1: Always do it in the same place. The problem with duplication is just like so much worse when you're dealing with And so one of the first things that I ended up having to do was like, honestly, manually instrument a lot of these flows and actually read through the code myself and understand, okay, this chunk of processing is synchronous. This chunk of processing is asynchronous and put all of those together. And one of the ways that I started doing that was I had it generate a Like big sequence diagram of everything that goes into time to first token. So like from the time you hit submit to when you receive that first visible token and it was huge. And I went to Staples and I had them printed out on like big blueprint paper because I'm still a really analog person. And then from that, I went through and I was reading the code and I was annotating in on my big piece of paper. Like, okay, this is correct or this is incorrect. This piece we could probably do something about. This other piece, no, actually definitely needs to happen before this other piece. One of the other things that I was finding that it was doing really painfully wrong is that it was finding sort of the classic things that could... [01:07:00] S2: What was doing what the LM was doing wrong? [01:07:02] S1: Oh, yeah. So what it was doing wrong was finding... Well, wrong is kind of the wrong word. [01:07:08] S2: Let's be opinionated. I like it.
01:07:08 · Gergely Orosz, Maude Lemaire
[01:07:08] S2: Let's be opinionated. I like it. [01:07:11] S1: Naively, naively what it was doing was, say you have a bunch of things, you're finding a bunch of tasks that don't depend on each other, right? But they're latency sensitive. What's the classic solution? [01:07:24] S2: You paralyze them. [01:07:25] S1: You paralyze them. However, again, product hat, right? What if basically no one is ever doing all four of those things? [01:07:35] S2: And why are you doing all of them? At the same time, you're wasting resources, latency, you're increasing load for no reason. [01:07:43] S1: And so I remember, I think it was Opus that was very excited that it found this opportunity to paralyze a bunch of these tasks. And I was like, oh, on first glance, sure, that seems... Like a good thing to do, except that the problem, even if you paralyze them, the longest one is always going to be the bottleneck regardless. And the longest one was always so much longer than the other three. And the other three tasks were being run such a small 1% of users were doing any of the other three. And so it ended up being a complete waste of time. Like this was not the optimization that it was like selling to me as being like the panacea to improving time to first token. And a lot of the things that were actually quite helpful were really silly things like, I mean, silly, like increasing the TTL on what we stored in Redis where we could within reason.