Unsupervised Learning with Jacob Effron
00:33:22 · Diogo
Compute, you know, expected value of, like, cache reads and cache writes and everything, and recalculate perhaps a new k v cache for this other task and maybe even keep the other one hot. I don't really know. This sounds like cool sci fi. One more. One more, but then I will stop on this topic. But I I... I'm just so excited about this. One thing that I think would be really cool is, hey. I keep cooking on the same code base, and I keep reading the same files and having to reunderstand my system's abstractions over and over and over again. You know, there's a partial solution to this, which is, maybe the agent should write markdown files, but then how do you keep it in sync? Blah blah blah. You know, it's it's a... It's kind of a pain. You know? But what if you could look at the history of old models, but, like, not as, a... Like, I feel like right now it's, like, a little bit of a hack. Like, oh, it's a tool that you can use sometimes if the model deems it as necessary. But what if, like, by its nature, one of the first steps you do is even a extremely cheap MapReduce overall history to find the relevant stuff here? And what if that history could also be shared with your coworkers who are who are using the same code base so you can actually have insights from what they're working on? Like, these are very doable software problems. Right? The the problem is the... It's, like, very unfriendly with a KV cache when all the intelligence needs to be append only. And there's there's so many other cool things.
00:34:40 · Jacob Effron
[00:34:40] S0: I love it. Yeah. Feel the creative juices flowing. I I I guess, you know, another thing I'd I'd be curious about is I feel like you you all obviously have released some some, you know, stuff around images. Right? And, like, I'm... And, you know you know, obviously, multimodal, I imagine, is the focus. I I think most of our listeners probably think of Jev in the context of, like, you know, the... These these decision models or type one things and and and Mhmm. Kind of in in classic, like, LLM replacement. Maybe talk a little bit about, like, the multimodal world, like, how you see the shape of problems there, how, like, Jev models fit into those.
00:35:12 · Diogo
[00:35:12] S1: So there's a couple of, like, branches to go off of here. I will say branch one is that being aware of the reality of AI is so unbelievably important. And by its nature, just like long context tends to degrade intelligence no matter the model, multimodality tends to degrade intelligence too. And and this is, like, well known fact by, like, anyone in ML circles. And, actually, every modality has different rates at which it is degraded, and this is very measurable stuff except when you put them into chatbots because chatbots are optimized for human preference. It becomes a very, very fuzzy thing on whether or not it's better or not. It's very measurable that multimodal models will be less reliable than the text models. And my philosophy has always been automate the easy work before the hard work. Like, will Jev models be used to automate software engineering? Maybe one day. But software engineering, maybe I'm arrogant, seems a lot harder than, like, really easy tasks that LMs fail at. And in order to, like, be calibrated to the world, we should study why they fail at those easy tasks and, like, really honestly, like, go in-depth into studying them, not cheat them, which I think is, like, quite common, but, like, actually, like, dig in and figure out what is missing and why. And sometimes it's not the model. Sometimes it's the prompt. Sometimes it's every other stuff. You know, like, it's a it's a collaboration between software and model. But it's through that that I think we, like, learn the truth. I have, like, stories about how we got to even be anti benchmaxing more. But this is a thing that LMs try to avoid because they will look bad on benchmarks as far as I can tell. And so that is one aspect of multimodality. We have to take into account text tasks will be automated before the image tasks. Thing too is we invested really heavily on text tasks and very much... Like like, we've done, like, a tiny amount of work in comparison to the image tasks. Like, I am, like, kind of jazzed that our model does as well as it does. I I think that that's great. Like, as far as I can tell, our lead in images is not as large as our lead in text, but we do have an overall lead in images. It's my vibe. Could be totally wrong, but that... That's kind of what I've sensed. And I would actually describe the image model as completely insufficient.
00:37:21 · Diogo, Jacob Effron
Like, I would be truly embarrassed to say that this is prod ready because there's so much jaggedness left in it that I'm I'm jazzed if people find stuff to actually deploy, but I will say that this does not have, like, my seal of approval yet. We need to do so many rounds. Because, like, it it... Like like like, the... It's it's not quite as bad as, like, other people just shipping a shape of something, but it's it's, like, in the grand scheme of it, like, image... Our image model is, like, maybe 10% of the way that our, like, text model is. [00:37:51] S0: As you move from, like, 10% by your characterization to, like, 90, 95%, what are the first kinds of things that you're excited about gem models doing on the image side?
00:38:00 · Diogo
[00:38:00] S1: On image land, I have a soft spot for video games and computer use. Video games because I have, like, a video game background. I... And... Oh, man. I I I... Confession. I've also, like, played around with botting them, not for competitive reasons, but just because it's cool to automate stuff. And Yeah. I think, like, being able to play around with that and, like... I I feel like software is a little bit like warping technology to your will, and being able to, like, expand what you can warp to your will is just very cool in general. So I think that is really cool in video games. Computer use, I think, is gonna be super duper valuable, unbelievably valuable. I I think it's it's... Like, especially combined with, like, lots of other stuff, I think that AI should be so much better for interfaces. It's actually kind of confusing to me how it is so bad. You know, like, why... There's a lot of people who don't like using computers, and they still have to even if they're, like, old and not experts despite that, despite AI being probably better than an assistant that they could hire to do a lot of computer stuff for them. And I would love for that to be reliable enough that people could, like, trust it as an interface. And, you know, I I guess an additional thing that is, like, I have a soft spot for, and this is why I'm wearing my, like, unsloppable shirt, is I would love ways to just stop slop. For me, personally, it might be bad for the Internet. Like like, the Internet is funded by ads, so, like, maybe it's a good thing. I don't know. But I would love if I could have some... Or someone would make me, like, a slop stopper where I just don't need to see it anymore. That would be lovely.
00:39:36 · Jacob Effron
[00:39:36] S0: The one thing I've been curious about from the outside is, like, the extent to which, you know, the advances that you've had at Jev were enabled by, like, all the advances we've seen in in, you know, AI in the last two, three years? Or could you have just done this five years ago? Like, obviously, you have a different approach that that is, in some ways orthogonal to the rest of the field. Like, could you have done this five years ago, or how much of what you built, like, does leverage some of the stuff that's been done, you know, on the LM side?