Unsupervised Learning with Jacob Effron
00:40:01 · Diogo
[00:40:01] S1: Five years ago, it would have been harder but doable. Back when I was more naive... Oh, man. I sound like an old dude. When I was more naive and I thought that this would take a week of work, what convinced me... I I love writing documents. I love... Like, I think it's a great way to think. I wrote a document at OpenAI that was titled, we could have had AGI a year ago, at least by OpenAI's definition of AGI, which I don't think that people really listen to the letter of the definition, but I think it's actually a really great letter. I think that models from the... Starting from GPT 3.5 era, possibly even GPT three era, could have met the definition of AGI at the time, which is still the same, like, doing most of the world's economically valuable work by volume. It doesn't say by volume, but I assume the spirit of that is by volume. Because the models are so unbelievably smart. Like, GPT 3.5 level models surpassed median level intelligence at all sorts of things. And, like, shouldn't that be used for tons of automation, at least for the easy work? And so, yes, I think that something like this would have been harder, but it could have been done five years ago. Absolutely. And, you know, like, on the people who are, like, obsessed with, like, for example, like, the the benchmarks of the day and they see, like, what tasks AI is doing. These tests are, like, really hard. They're, like, really, really, really hard. GPQA was saturated twenty twenty three ish. Maybe not totally saturated, but it was, like, surpassing PhD student level at 2023, I believe. Give me Google proof question answering for, like, PhD level intelligence at these extremely hard multiple choice tasks. It was passing that level twenty twenty three. Like, how is it that that level of intelligence was not being put to work? That is super tragic. So I think we've had a lot of, like, raw material for intelligence for a very long time, and we're gonna see a whole lot of craziness. Yeah. I mean, speaking of
00:42:02 · Jacob Effron
[00:42:02] S0: that craziness, I think part of... I imagine there's a lot a lot of noise. You know, I feel like there's been a lot of noise in the ecosystem, and I bet many of our listeners will be familiar with, you know, after you guys came with these these models, you had OpenAI and Cloudflare and Microsoft, and everyone kind of released these these models. I I know you'll you'll say that the benchmarks aren't aren't a good way to test them. They obviously... You know, I've I've tried to show that that they're near equivalent. I guess, how do you think about... Like, you obviously have now invited a bunch of competition into the space. How do you think about, like, what, you know, keeps Jev... You know, I guess everyone likes to ask about moats, so I guess I'll ask it on on behalf of our listeners. Like, how do you think about that going forward for the business?
00:42:39 · Diogo
[00:42:39] S1: Oh, boy. It feels like other people are digging our moat for us, which is kind of not how I thought it would go, and I think that that's kind of bad for the field. I think that releasing models that are, like, kind of more benchmark optimized than real world optimized actually burns that goodwill for people to try things out, which is the only thing that matters. I I truly think that this is not great for the field, and my hope is that just giving unbelievable facts about reliability is going to be the north star. I think that will be immensely, immensely valuable for the world. There's just so much more to do there that I think that it's a little bit early to worry about competition. It's it's kind of like OpenAI worrying about a moat at at G P T 3.5. Like, who cares? Like, G P T 3 Point Five's market is nothing, or it was nothing, right, in comparison. It's still, I think, going to be nothing in the grand scheme of things. Like Yeah. We truly believe in Jevons paradox, obviously, hence the name Jev. Like, Jev and Paradox... Like like, are we seeing rebound, you know, more demand because it's so cheap? Absolutely. I believe every good sees rebound. Are we seeing backfire? Like, the total market size increases. This is this is when Jev and Paradox holds. Absolutely not. Like, it it... Like like, a key part of it has to be like, wow. This is so good and so cheap. I'm gonna be throwing it all over the place because it makes value in all of these places, and my budget for AI will actually increase.
00:44:12 · Jacob Effron, Diogo
[00:44:12] S0: I mean, speaking of investors, I guess you you obviously recently raised a a a pretty big round. How did you even think about the amount of capital that made sense to raise for for this business? I'm sure people were trying to throw money at you. So Yes. You know, how how do you how do you even, you know, think through that? [00:44:26] S1: I will say that people probably don't believe it, but, like, we can run our service profitably. And I hope I hope that we never have to do the thing of, like, temporarily subsidizing to get lock in. I I don't want to, like, axe it out entirely because, like, I'm a pragmatic person, but, like, I really hope that doesn't have to be the case. I hope that... I I find that to be, like, a very shadowy, sneaky way to get people to use your service. So I... And I hope to be the kind of person who's, like, you know, always collaborating in, like, these prisoners' dilemmas until, you know, until forced otherwise. So we actually did not need the money. We... And our goal instead was, hey. We want actually, like, the most valuable people on our side to help us build this business because we have no fucking clue how to do this. So we really opted for, like, who are the best people in AI who will help us. You can see who we got on our board.
00:44:26 · Diogo, Jacob Effron
[00:44:26] S1: I will say that people probably don't believe it, but, like, we can run our service profitably. And I hope I hope that we never have to do the thing of, like, temporarily subsidizing to get lock in. I I don't want to, like, axe it out entirely because, like, I'm a pragmatic person, but, like, I really hope that doesn't have to be the case. I hope that... I I find that to be, like, a very shadowy, sneaky way to get people to use your service. So I... And I hope to be the kind of person who's, like, you know, always collaborating in, like, these prisoners' dilemmas until, you know, until forced otherwise. So we actually did not need the money. We... And our goal instead was, hey. We want actually, like, the most valuable people on our side to help us build this business because we have no fucking clue how to do this. So we really opted for, like, who are the best people in AI who will help us. You can see who we got on our board. [00:45:27] S0: Well, I always like to end with a quick fire round where I just stuffed some overly broad questions into the end that I haven't had a a chance to ask elsewhere. I'm curious, you know, over the last month, you know, or last six weeks of having this product out there, like, what's one thing you've changed your mind on in AI in the last, like, six weeks, having just seen how Woah.
00:45:44 · Diogo
[00:45:44] S1: That's such a cool question. Woah. Woah. That's a quick fire question? Holy smokes. I will say that we were really worried if our philosophy about caring about, like, the actual reliability of intelligence was a mistake. Like, we had people in the team say, like, hey. Should we just release a Benchmax version and just hype it up and, like, do one of those early releases that, like, obviously is not great, but just to, like, test out if the market exists? I'm so glad we didn't do that. And I I say seven weeks because, like, I I don't remember when launch was, but, like, seven weeks, I think, was before launch. We really did not know this. We... It was, a big leap of faith for us, and, like, we were just really... Man. Like, I don't think we could be, like, proud of ourselves if we did the same overpromise, underdeliver. And, like, I think that that paid off super duper well. I actually don't know if it paid off. Maybe the product sold itself, and maybe it could have done even better if we, like, really optimized for for attention. But that was, like, awesome. And I actually feel like now that my brand has been solidified as our brand, I hopefully never have to do that, even close to that again, and that debate no longer exists for us. I hope. I hope. I don't know. I truly, truly hope because I have, like, a hard time, like, even giving, like, unnuanced, slightly incorrect answers because I think that that's just a disservice to anyone listening to me. I would rather just, like, not answer the question than do the standard, like, CEO compression of the truth that airs on the side of generous. So I hope that I will never really have to do that. But, like, something I've learned as well is that this reliability saga was sped run so fast for our class of models. I feel like... I I could be wrong, but my temperature check of the environment now is people have already gotten, like, extreme fatigue of, like, the benchmarks and not actually useful stuff way faster than I would have thought. Like, I I would have, like, thought that this would be a concern based on what I see because people keep falling for benchmarks every time a new model's released. Right? People are like, well, cool. Let's talk about it. And then it it it disappoints, except when it's like an OpenAI anthropic model, apparently. And that just happened. Kai...