Unsupervised Learning with Jacob Effron
00:29:19 · Diogo
And now this is the equivalent of, like, from a state management perspective. Imagine your state is like global variables. It's passing one giant dictionary around forever until the models break, and they start getting dumber over time because context drop is a real thing. And this is totally cool, just to be clear. I don't believe that there's, like, anything super tragic about this on its own because it's very useful. I think what's tragic is that's the really only... Only way to do it. And I think there's so many more advanced patterns that you could do if you could assume that I could just put intelligence all over these operations. And that is the kind of thing to me that I think from a dis... Engineering, architecture, design, beauty power perspective is so limiting. And that is what that tier and is because the people who have tried to do those cool algorithms have mostly failed because they've tried to use LLMs for it, and LMs are not optimized for, you know, routing logic within the software or making discrete little decisions. They're optimized for making human pleasing outputs that are not super reliable running as a subprocess. And this then makes people believe this doesn't work. Yeah. And, actually, a lot of what our original shape looks like is a... It's not too dissimilar from langchain. Well, like, there's nothing wrong with langchain's interface. Right? Like, I... Like, some would even say that it was, like, maybe perfect for classifiers on top of LMs. I think... I I I I can't remember back back in the day, but I'm pretty darn sure it's gonna be pretty darn good if someone made, like, a, like, a langchain compatible wrapper on top of our model. The problem is people didn't like it because it made models some... Somewhat dumber because the models are not optimized for that type of pattern. And this is also a form of tyranny where everyone just thinks Anthropic and OpenAI or whoever else is, like, number three right now is, like, doing doing their best and giving us the best intelligence possible when the reality is there's so many other intelligences possible. And, like, there... There's trade offs there. You know? Like, there's truly a lot of trade offs that are made in this design that people can feel but cannot really measure because it's really hard to do so.
00:31:21 · Jacob Effron
[00:31:21] S0: What's, like, an example of the shape of what a coding agent could look like, I guess, with, like, a different intelligence design?
00:31:27 · Diogo
[00:31:27] S1: One feature that I think would be really sick is what if you could have, like, multiple agents sharing the same context? And, you know, like, truly, like, read writing from the same context. That sounds really complicated. Like, there's no way that works right now because of the KV cache, but, like, nothing stops that from happening. And, actually, a very common pattern people have with agents these days is, like, they're just cooking on crazy stuff. They don't know what's going on. And after it does the thing you asked it to do, people are like, make a visualizer or a dashboard or a learning tool. And it's kind of crazy that that has to be, like, an extension of the context that has all of the trial and error in it or starting from scratch looking at a Git commit. Why can't... And rereading everything from scratch. Why can't it share the state, and why can't you have, like, an agent here that's actually basically read only for the code base but allows you to write the visualizer such that as the agent is cooking, you actually get live visualizations of maybe, you know, using, like, a slightly dumber model, like Sonnet instead of Opus, to write out what Opus is doing so you kind of get a sense of what's going on instead of having to read, like, the, know, the the gibbitzing updates or whatever else. I think that would be cool. Another example here is compaction. Compaction makes a lot of sense for a write once k v cache that is append only because, like, oh, it's full. We need to restart it. But compaction is a form of compression, and compression is just not totally solvable. Like, it's very hard to compress without priors for compression. But what happens if you compress based on what problem you were solving before? And now you want to actually ask a problem that is different from that one and would use information that was compressed away. Like like like, that agent is, like, Delta very much losing hand. But what if how the state gets compressed was actually semantic, and it could detect, hey. I'm doing a new task. I should actually think about what the context should be for this new task using an ROI calculation, which should be in code. You know, the code should count...
00:33:22 · Diogo
Compute, you know, expected value of, like, cache reads and cache writes and everything, and recalculate perhaps a new k v cache for this other task and maybe even keep the other one hot. I don't really know. This sounds like cool sci fi. One more. One more, but then I will stop on this topic. But I I... I'm just so excited about this. One thing that I think would be really cool is, hey. I keep cooking on the same code base, and I keep reading the same files and having to reunderstand my system's abstractions over and over and over again. You know, there's a partial solution to this, which is, maybe the agent should write markdown files, but then how do you keep it in sync? Blah blah blah. You know, it's it's a... It's kind of a pain. You know? But what if you could look at the history of old models, but, like, not as, a... Like, I feel like right now it's, like, a little bit of a hack. Like, oh, it's a tool that you can use sometimes if the model deems it as necessary. But what if, like, by its nature, one of the first steps you do is even a extremely cheap MapReduce overall history to find the relevant stuff here? And what if that history could also be shared with your coworkers who are who are using the same code base so you can actually have insights from what they're working on? Like, these are very doable software problems. Right? The the problem is the... It's, like, very unfriendly with a KV cache when all the intelligence needs to be append only. And there's there's so many other cool things.
00:34:40 · Jacob Effron
[00:34:40] S0: I love it. Yeah. Feel the creative juices flowing. I I I guess, you know, another thing I'd I'd be curious about is I feel like you you all obviously have released some some, you know, stuff around images. Right? And, like, I'm... And, you know you know, obviously, multimodal, I imagine, is the focus. I I think most of our listeners probably think of Jev in the context of, like, you know, the... These these decision models or type one things and and and Mhmm. Kind of in in classic, like, LLM replacement. Maybe talk a little bit about, like, the multimodal world, like, how you see the shape of problems there, how, like, Jev models fit into those.
00:35:12 · Diogo
[00:35:12] S1: So there's a couple of, like, branches to go off of here. I will say branch one is that being aware of the reality of AI is so unbelievably important. And by its nature, just like long context tends to degrade intelligence no matter the model, multimodality tends to degrade intelligence too. And and this is, like, well known fact by, like, anyone in ML circles. And, actually, every modality has different rates at which it is degraded, and this is very measurable stuff except when you put them into chatbots because chatbots are optimized for human preference. It becomes a very, very fuzzy thing on whether or not it's better or not. It's very measurable that multimodal models will be less reliable than the text models. And my philosophy has always been automate the easy work before the hard work. Like, will Jev models be used to automate software engineering? Maybe one day. But software engineering, maybe I'm arrogant, seems a lot harder than, like, really easy tasks that LMs fail at. And in order to, like, be calibrated to the world, we should study why they fail at those easy tasks and, like, really honestly, like, go in-depth into studying them, not cheat them, which I think is, like, quite common, but, like, actually, like, dig in and figure out what is missing and why. And sometimes it's not the model. Sometimes it's the prompt. Sometimes it's every other stuff. You know, like, it's a it's a collaboration between software and model. But it's through that that I think we, like, learn the truth. I have, like, stories about how we got to even be anti benchmaxing more. But this is a thing that LMs try to avoid because they will look bad on benchmarks as far as I can tell. And so that is one aspect of multimodality. We have to take into account text tasks will be automated before the image tasks. Thing too is we invested really heavily on text tasks and very much... Like like, we've done, like, a tiny amount of work in comparison to the image tasks. Like, I am, like, kind of jazzed that our model does as well as it does. I I think that that's great. Like, as far as I can tell, our lead in images is not as large as our lead in text, but we do have an overall lead in images. It's my vibe. Could be totally wrong, but that... That's kind of what I've sensed. And I would actually describe the image model as completely insufficient.