Unsupervised Learning with Jacob Effron
00:17:48 · Diogo
[00:17:48] S1: I'm gonna end up saying something spicy here, but I would say when I first... When we first launched this, I was really excited for novel shapes to emerge. Really, really excited, and I still think that that will be the case. I think new shapes are freaking awesome. We plan to keep shipping them. I'm a little bit worried that, again, what we talked about... Maybe this was, like, before we got on the record, that ML people especially have, like, kind of, like, lost the actual message I was trying to say. And I'm worried that people are now overfocused on, like, the shapes of tasks rather than the reliability. The shape of task is incredibly... You know, like, it's super easy to do that. When I thought this would take a week, I thought it would take a week to make the shape of the task, which it did. And this is what's happening now with other people making similar models with similar shapes. But the shape really doesn't matter compared to actually it being useful, and that is the thing I think people really, really need to focus on. So there will a 100% be way more shapes than these two. Even if no one else does it, we will do them. I think people are making too many new shapes right now and not caring enough about you... Like, the utility of it. And I'm worried that this will, like, actually burn the goodwill that people... People don't like being, overpromised, underdelivered to. And, like, these things are, like, pure overpromised, underdelivered. And I think that that's, like, not good for the world. You know? Like, if people have, like, creative, playful energy, we don't want them to waste it on stuff that clearly doesn't work. As far as I can tell, people dislike that. And even if they disliked it, I hate it. Like, I... Oh, man. And I think there's going to be a balance. There will definitely be a lot more future shapes to be had. And I just really, really hope that people start giving, like, unbelievable levels of fucks for utility, usefulness, reliability. What would a developer want in this situation rather than ML people who I think focus on, like, I want to flag plant. You know? Like, I want to be the one who names the thing that gets cited. I want to, like, get revenue in the short term because people will play around with it, and I want to do the same thing that every other AI company does.
00:19:57 · Diogo, Jacob Effron
I really hope we don't do this, but overpromise, underdeliver in order to get attention because it feels like a lot of the AI economy ish is really driven by that attention. And that is so, so bad for developers. It's, like, actually, the absolute worst for the developers who do not have, like, the time to test these things themselves because they just get... They're the ones who get swept up and all. This is this is also why I'm anti benchmarks. It's a it's a... It's it's so bad for those people. You know? [00:20:26] S0: You have to play around with some of the stuff yourself to have any sense of of of of how good it actually is. [00:20:31] S1: Yes. And, like, that is tragic. You know? Like, it it makes sense. Like, I actually think that as far as I can tell, people have stopped caring about, like, benchmarks for our class of model. Maybe I'm, like, in the wrong niche for this. But my sense... And we... There could be survivorship bias, is that people already don't care about new releases here because they've all been lied to so many times. And this is good for us as a company, but bad for the field because, actually, if someone makes a genuinely good thing, there should be a way to acknowledge it and shout it out and say, like, hey. This is awesome. But, like, in the world where everyone optimizes for benchmarks and demos and things that don't require liability, it's very hard to communicate that other than this very manual process of evaluating models themselves, which is actually super duper hard because Yeah. Benchmarks also don't capture that.
00:20:31 · Diogo, Jacob Effron
[00:20:31] S1: Yes. And, like, that is tragic. You know? Like, it it makes sense. Like, I actually think that as far as I can tell, people have stopped caring about, like, benchmarks for our class of model. Maybe I'm, like, in the wrong niche for this. But my sense... And we... There could be survivorship bias, is that people already don't care about new releases here because they've all been lied to so many times. And this is good for us as a company, but bad for the field because, actually, if someone makes a genuinely good thing, there should be a way to acknowledge it and shout it out and say, like, hey. This is awesome. But, like, in the world where everyone optimizes for benchmarks and demos and things that don't require liability, it's very hard to communicate that other than this very manual process of evaluating models themselves, which is actually super duper hard because Yeah. Benchmarks also don't capture that. [00:21:24] S0: One thing I'm struck by, you know, is obviously this idea of, like, these shapes of problems. Right? And the chatbots are very good for some shape of a problem you're trying to solve. Coding agents, like you said, very good for some shape of a problem. Obviously, folks have found Jev really helpful in in in different shapes of problems. Are there other problems out there that you're still like, god. There's just nothing out there today that's good for [00:21:42] S1: So many.
00:21:42 · Diogo, Jacob Effron
[00:21:42] S1: So many. [00:21:43] S0: What what are like some that come to mind that like... Of of these future areas? Hi. I'm Anne Maris, and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. I love working on this show, and I have the challenge of hitting 35,000 subscribers by the end of the year. If I don't hit that, I'll be fired, so please subscribe. Thank you. Bye. What what are, like, some that come to mind that, like... Of of these future areas?
00:22:06 · Diogo
[00:22:06] S1: What I realized is that a lot of people are still stuck in, like, fairly basic patterns, and we have, like, a gigantic job to both push the frontiers of model intelligence, reliability, all of these things that we really care about, as well as education to push the limits of software. And it... Like, there's, like, a thing in, like, the intersection of education and pushing the frontier forward, which is like this collaboration between research and product, which is, I think, where all gigantic, like, ML wins come from. And this is, I think, is, like, kind of an unexplored place. Like, if we had more help, we could do so much more here to, like, encourage people in the world or even fund the kinds of things that we think should exist and really allow people to push their front... The frontiers of what software can possibly do. So I'll I'll give a couple of examples of this, and, you know, this is going to be a little bit of a spoiler for a launch... A release we will have sometime in the future. Is that we want to, like, fund these things, and there's so many cool shapes of stuff missing. Like, one of the things people know I'm passionate about is coding agents that are free from the tyranny of the KV cache. I think that so much coding agent is built around the KV cache. And the KV cache is freaking awesome too. We... I don't want to, like, toss it away, but I want to be aware of it and design around it and make different sets of trade offs to make things that were, like, pure sci fi possible. Another thing that I think would be so unbelievably cool is... Do you know what dependent typing is? Dependent typing is a lot about, like, adding constraints to your programs that are, like, higher level than purely the type system. You know? Like, you can make sure that your function returns an int, but dependent typing would allow you to do things like it is an int greater than zero. And then now downstream... Or or downstream applications can use those properties to have their own sets of guarantees, etcetera, etcetera, etcetera, to get higher and higher levels of composability with your software because they're more legible, and they don't require as much, like, unit testing edge cases and also as much human understanding to understand the whole system. And one of the things I think would be so freaking cool would be a dependently...
00:24:13 · Diogo
A semantic dependent typing, where instead of, like, having to represent your outputs as pure types, like int, float, you could actually say specific properties of that in natural language because there's so many properties you could care about in software and then have a type system that uses Jev like models verify that everything works together. I think that that would be so unbelievably cool. Maybe it doesn't matter in a world where everyone uses coding agents, but I think that that... Like like like, we should do stuff like that just because it's us. Awesome. And that's what early Internet energy is about, like, to me. So security is a huge thing as well. Like, as these models get cheaper, faster, all over the place, where can we add extra levels of security kind of to avoid if you guys saw the news about, like, the Cohere, Qwen 3.8 Next Flash Yeah. Debatably makes a sketchy call to, like, phone home. It's it's debatable. I'm not gonna talk about whether or not that's legit or not because I haven't read enough about it. But why can't we just have checks on top of every single tool call that actually are semantic to your organization rather than whatever Anthropic thinks should go into their auto router? You know? Like, maybe in this part of the code base, they shouldn't be making any calls to this other part of the code base at all. And how do you add these checks? You know, you you need, like, extremely fast and cheap security all over the place, which could be super awesome. You know? I'm not saying everyone should build it right now, but I think that we should collaborate with, you know, the making... Pushing capabilities and software in order for, like, thinking