Unsupervised Learning with Jacob Effron
00:16:03 · Diogo, Jacob Effron
Like, it's very counter to how software engineering tends to work. [00:16:07] S0: I guess in your vision of the world, like, Dell, have a place in the world in, like, ten years?
00:16:07 · Jacob Effron, Diogo
[00:16:07] S0: I guess in your vision of the world, like, Dell, have a place in the world in, like, ten years? [00:16:11] S1: Super duper. Absolutely. I I I hope if that they do not have a place, then something strictly better exists. I... I'm not opposed to LMs. I'm not actually opposed to either chatbots or coding agents, completely different things. And I I really don't want to go to a world where they don't exist. The thing is I think people overuse them for a lot of tasks, and I actually hope the market for that increases too as people get better at it. And I'm I'm I'm, like, basically certain that there's tons of use cases for it at basically every single intelligence cutoff that has existed, maybe since GPT four. Like, all of these models had beautiful, awesome use cases that maybe would get Pareto dominated by another model, but, like, they should be able to run-in production. And it takes a lot of nuance and understanding, and it... It's like it's like really not built for engineers. So I love LMs. I hope that I I can continue to, like, chatbot anything that I have, like, I need, like, x... Pseudo expert level answers to that don't need to be correct. That is the use case in my mind for chatbots. And I love being able to, like, spin up, like, throw away software instantly when I want custom dashboards and stuff, because I think it's the best use case for coding agents. And that should stay for the rest of time. I don't think people should be doing that. And we should keep on building our toolbox, so to speak. Like, it's... I think it's just awesome.
00:17:31 · Jacob Effron
[00:17:31] S0: Is your intuition that the toolbox will include, like, many other shapes of models in the future with, you know, other RL acronyms? Or if we're talking ten, fifteen years from now, like, are there, you know, are there a bunch of different, like, tools in this toolbox, or is it kind of like your approach and the LLM approach cover most of the use cases?
00:17:48 · Diogo
[00:17:48] S1: I'm gonna end up saying something spicy here, but I would say when I first... When we first launched this, I was really excited for novel shapes to emerge. Really, really excited, and I still think that that will be the case. I think new shapes are freaking awesome. We plan to keep shipping them. I'm a little bit worried that, again, what we talked about... Maybe this was, like, before we got on the record, that ML people especially have, like, kind of, like, lost the actual message I was trying to say. And I'm worried that people are now overfocused on, like, the shapes of tasks rather than the reliability. The shape of task is incredibly... You know, like, it's super easy to do that. When I thought this would take a week, I thought it would take a week to make the shape of the task, which it did. And this is what's happening now with other people making similar models with similar shapes. But the shape really doesn't matter compared to actually it being useful, and that is the thing I think people really, really need to focus on. So there will a 100% be way more shapes than these two. Even if no one else does it, we will do them. I think people are making too many new shapes right now and not caring enough about you... Like, the utility of it. And I'm worried that this will, like, actually burn the goodwill that people... People don't like being, overpromised, underdelivered to. And, like, these things are, like, pure overpromised, underdelivered. And I think that that's, like, not good for the world. You know? Like, if people have, like, creative, playful energy, we don't want them to waste it on stuff that clearly doesn't work. As far as I can tell, people dislike that. And even if they disliked it, I hate it. Like, I... Oh, man. And I think there's going to be a balance. There will definitely be a lot more future shapes to be had. And I just really, really hope that people start giving, like, unbelievable levels of fucks for utility, usefulness, reliability. What would a developer want in this situation rather than ML people who I think focus on, like, I want to flag plant. You know? Like, I want to be the one who names the thing that gets cited. I want to, like, get revenue in the short term because people will play around with it, and I want to do the same thing that every other AI company does.
00:19:57 · Diogo, Jacob Effron
I really hope we don't do this, but overpromise, underdeliver in order to get attention because it feels like a lot of the AI economy ish is really driven by that attention. And that is so, so bad for developers. It's, like, actually, the absolute worst for the developers who do not have, like, the time to test these things themselves because they just get... They're the ones who get swept up and all. This is this is also why I'm anti benchmarks. It's a it's a... It's it's so bad for those people. You know? [00:20:26] S0: You have to play around with some of the stuff yourself to have any sense of of of of how good it actually is. [00:20:31] S1: Yes. And, like, that is tragic. You know? Like, it it makes sense. Like, I actually think that as far as I can tell, people have stopped caring about, like, benchmarks for our class of model. Maybe I'm, like, in the wrong niche for this. But my sense... And we... There could be survivorship bias, is that people already don't care about new releases here because they've all been lied to so many times. And this is good for us as a company, but bad for the field because, actually, if someone makes a genuinely good thing, there should be a way to acknowledge it and shout it out and say, like, hey. This is awesome. But, like, in the world where everyone optimizes for benchmarks and demos and things that don't require liability, it's very hard to communicate that other than this very manual process of evaluating models themselves, which is actually super duper hard because Yeah. Benchmarks also don't capture that.
00:20:31 · Diogo, Jacob Effron
[00:20:31] S1: Yes. And, like, that is tragic. You know? Like, it it makes sense. Like, I actually think that as far as I can tell, people have stopped caring about, like, benchmarks for our class of model. Maybe I'm, like, in the wrong niche for this. But my sense... And we... There could be survivorship bias, is that people already don't care about new releases here because they've all been lied to so many times. And this is good for us as a company, but bad for the field because, actually, if someone makes a genuinely good thing, there should be a way to acknowledge it and shout it out and say, like, hey. This is awesome. But, like, in the world where everyone optimizes for benchmarks and demos and things that don't require liability, it's very hard to communicate that other than this very manual process of evaluating models themselves, which is actually super duper hard because Yeah. Benchmarks also don't capture that. [00:21:24] S0: One thing I'm struck by, you know, is obviously this idea of, like, these shapes of problems. Right? And the chatbots are very good for some shape of a problem you're trying to solve. Coding agents, like you said, very good for some shape of a problem. Obviously, folks have found Jev really helpful in in in different shapes of problems. Are there other problems out there that you're still like, god. There's just nothing out there today that's good for [00:21:42] S1: So many.