Unsupervised Learning with Jacob Effron
00:25:43 · Jacob Effron
[00:25:43] S0: It's so interesting. Because a lot of those problem... Like, the the problem you're just talking about now, like, you could imagine, you know, approaching from the the perspective of a model company or, like, there's all these, you know, enterprise agent security companies that are popping up, right, that aren't model builders themselves, but they're like, yeah. Let's put in some, like, you know, some actual, like, deterministic software to to to monitor these things. And it's interesting to, like, think about you approaching a similar set of problems, but more from, like, a model perspective versus, obviously, there's plenty of other folks that are just trying to build software that you can put around an LLO.
00:26:11 · Diogo, Jacob Effron
[00:26:11] S1: So my philosophy is that we should enable those software players to win. Just give them tools to build up people. Exactly. But, ideally, they also tell us... We we don't want their data, but they should tell us, like, hey. You have jaggedness on this particular axis that makes our thing less reliable. Let's work together. And, actually, they could just tell us, like, we are not gonna tell you what we work on at all. Here's the rough shape of this task. Fix it. And I would just be like, hell yeah. Thank you so much for telling us this. Hopefully, it solves your problem and a whole bunch of other people's problems. I hope you produce tons of value in the world. Like, I hope you make, like, a decent amount of money, and I hope that this enables other people to build all sorts of other things because now this jaggedness thing is healed. And that is the relationship I hope to have. And this is why I'm, like, so so jazzed about software. [00:27:00] S0: No. I love it. You you said something there that I I wanted to come back to because it was just a great phase. I think you said that the tyranny of the KV cache in in coding agents. Maybe explain for our listeners a little bit about, like, some of the limitations in coding agents, you know, due to the KV cache and, like, how you think about that problem space.
00:27:14 · Diogo
[00:27:14] S1: I would say almost all design decisions around the UX of coding agents, like the stuff that actually functionally changes stuff, is built around the KV cache. And this is natural because, like, the biggest wins in ML are always, like, very hardware aware. This is how things have gotten so efficient recently or pseudo recently since g p t four's days. G p t four is very expensive. Like, you just need... Really need to design around this, which is great because it's efficient, but it's also quite limited. It it... This is why, like, routing doesn't tend to work. Routing absolutely works if you're making the equivalent of, like, what we call instruction following calls. Like, you know, task in, task out, you know, fuck the cache, throw it away. Routing is amazing for that. Actually, you can do way more advanced things than just routing in that case. There's so many advanced patterns you can do when that is your assumption. Like, you could actually try it with a cheap model, use an extremely cheap judge, and if it's efficient, keep it. If not, try a larger model or, you know, try multiple times and pick the best. Like, there's so many things you can do. But when you have, like, the stateful k v cache, you are, like, really constrained to it being kind of append only, And you have this complicated trade off, kind of like an iPython notebook, where you can never delete to the memory because it's, like, a really weird, you know, agglomerated state. And it 10... Turns out that it's very hard to engineer around this because the models are trained with it. So if you actually start doing fancy stateful operations with the models having free reign over that state, they'll end up looping and stuff like that because they're so tuned to having that KV cache and so efficient to have that KV cache. So this prevents you from doing, like, routing effectively. It prevents sub agents from being really effective because now you have this weird trade off of, like, what state... What context do I pass to my sub agent? Because this requires intelligence in order to know what to pass in. If you pass it all in, this is a huge input token cost instead of a cache read cost. And when... Knowing your subagent makes an output, how do you know what to pass in? And, like, what subset is relevant down the line? And the safest thing to do tends to be to append a lot of stuff.
00:29:19 · Diogo
And now this is the equivalent of, like, from a state management perspective. Imagine your state is like global variables. It's passing one giant dictionary around forever until the models break, and they start getting dumber over time because context drop is a real thing. And this is totally cool, just to be clear. I don't believe that there's, like, anything super tragic about this on its own because it's very useful. I think what's tragic is that's the really only... Only way to do it. And I think there's so many more advanced patterns that you could do if you could assume that I could just put intelligence all over these operations. And that is the kind of thing to me that I think from a dis... Engineering, architecture, design, beauty power perspective is so limiting. And that is what that tier and is because the people who have tried to do those cool algorithms have mostly failed because they've tried to use LLMs for it, and LMs are not optimized for, you know, routing logic within the software or making discrete little decisions. They're optimized for making human pleasing outputs that are not super reliable running as a subprocess. And this then makes people believe this doesn't work. Yeah. And, actually, a lot of what our original shape looks like is a... It's not too dissimilar from langchain. Well, like, there's nothing wrong with langchain's interface. Right? Like, I... Like, some would even say that it was, like, maybe perfect for classifiers on top of LMs. I think... I I I I can't remember back back in the day, but I'm pretty darn sure it's gonna be pretty darn good if someone made, like, a, like, a langchain compatible wrapper on top of our model. The problem is people didn't like it because it made models some... Somewhat dumber because the models are not optimized for that type of pattern. And this is also a form of tyranny where everyone just thinks Anthropic and OpenAI or whoever else is, like, number three right now is, like, doing doing their best and giving us the best intelligence possible when the reality is there's so many other intelligences possible. And, like, there... There's trade offs there. You know? Like, there's truly a lot of trade offs that are made in this design that people can feel but cannot really measure because it's really hard to do so.
00:31:21 · Jacob Effron
[00:31:21] S0: What's, like, an example of the shape of what a coding agent could look like, I guess, with, like, a different intelligence design?
00:31:27 · Diogo
[00:31:27] S1: One feature that I think would be really sick is what if you could have, like, multiple agents sharing the same context? And, you know, like, truly, like, read writing from the same context. That sounds really complicated. Like, there's no way that works right now because of the KV cache, but, like, nothing stops that from happening. And, actually, a very common pattern people have with agents these days is, like, they're just cooking on crazy stuff. They don't know what's going on. And after it does the thing you asked it to do, people are like, make a visualizer or a dashboard or a learning tool. And it's kind of crazy that that has to be, like, an extension of the context that has all of the trial and error in it or starting from scratch looking at a Git commit. Why can't... And rereading everything from scratch. Why can't it share the state, and why can't you have, like, an agent here that's actually basically read only for the code base but allows you to write the visualizer such that as the agent is cooking, you actually get live visualizations of maybe, you know, using, like, a slightly dumber model, like Sonnet instead of Opus, to write out what Opus is doing so you kind of get a sense of what's going on instead of having to read, like, the, know, the the gibbitzing updates or whatever else. I think that would be cool. Another example here is compaction. Compaction makes a lot of sense for a write once k v cache that is append only because, like, oh, it's full. We need to restart it. But compaction is a form of compression, and compression is just not totally solvable. Like, it's very hard to compress without priors for compression. But what happens if you compress based on what problem you were solving before? And now you want to actually ask a problem that is different from that one and would use information that was compressed away. Like like like, that agent is, like, Delta very much losing hand. But what if how the state gets compressed was actually semantic, and it could detect, hey. I'm doing a new task. I should actually think about what the context should be for this new task using an ROI calculation, which should be in code. You know, the code should count...