The Frontier with Brendan Foody
00:12:18 · Jonathan Frankle, Brendan Foody
[00:12:18] S0: Very simply put, it's evals. It's measurement. You know, there's a lot more to say about that, but the bottom line is I think we have plenty of intelligence. I think that, you know, I the line I draw in the sand is like, I think Claude Sonnet 3.5, which also feels like a very quaint statement to make that makes me sound ancient. Some people don't remember anything before Claude 4 at this point. But back in my day, we had Sonnet 3.5, and that was an awesome upgrade. And from then on, it's kind of felt like the models are good enough at a lot of really complex enterprise tasks and knowledge tasks much of the time. The problem isn't the model being smarter. You can take the world's smartest person in the world and sit them down at a desk and say, do work. That's an underspecified task. So I think the real challenge is describing what you want AI to do and knowing whether AI did it. That's an eval task by and large. If we get better at that, I think we will unlock huge amounts of AI value for enterprises. If we don't get better at that, you could give me a model that is 10x smarter than Mythos and the world's smartest person without knowing what they're supposed to do and what it means to do a good job is still going to fail. [00:13:19] S1: And of course, there's sort of two processes that the enterprise would go through first creating the eval, and then second, hill climbing it. How much do you see the hill climbing process involving automated prompt optimization with JEPA or manual prompt optimization versus custom model training to hill climb those evals?
00:13:37 · Jonathan Frankle
[00:13:37] S0: It's a good question. The way I think about it is kind of metaphorically when it comes to software. Usually, if you're gonna create a prototype, you just throw it together in Python. It's not gonna be fast, it's not gonna be great, it's not gonna be robust, but you need to demonstrate that it's the prototype is good enough that it's worth bothering to actually build a high quality version of it. Or you make a wireframe in Figma. That's where you start. You don't go and build out a whole user interface and make a perfect product, write it in Rust, and then say, like, is anybody gonna buy this? Make a quick prototype and see if anybody likes it. And I think this is true even more so with agents, where we're at the early stages of knowing what a good agent is and when an agent is worth building. First, just pull up your favorite foundation model, do a little prompt engineering, maybe build one or two quick eval examples or kind of vibe checking examples, iterate on it a bit and be like, is this even possible? If I throw the world's best model at this and spend a lot of time coaching it, can I get it through this task successfully? Like prove that it's worth it. And then from there, show that people are like people actually like this, show that you can make it good. That's where evals come in, obviously. And that's where you know actually testing it with users comes in. And it's okay if it's expensive initially. This is still a proof of concept. Time is much more valuable. If you can fail fast, if you can determine that this is working, sure, spend the money. I think Cursor is kind of in some sense a giant example of this. They built an incredible product that was very popular on that. And once you have commitment to a use case, you know that something's going to be around for a while. That's a great time to train a custom model. Where, you know, I think of this as like switching from Python to Rust. You're really committed. Nobody should write their first prototype in Rust, I think. Um I don't know, maybe some people really, really like Rust. But when you're really committed to something, yeah, you need to make it efficient and make it scalable. Write it in Rust. And it's the same thing. Do some custom RL on a model. But there are lots of stages in between where, you know, there are a lot of steps in the ladder. Custom proptimization is awesome.
00:15:17 · Jonathan Frankle, Brendan Foody
Run Jeppa. Use Jeppa against an eval set that you've built. That'll get you something better than you could probably tune by hand or at least faster, and you'll get some more value out of it. Figure out whether you can use a smaller model for it to make it more efficient and more scalable. But at the end of the day, once you've committed to something at large scale, you know, custom models can shave off 5x or more of your cost. And, you know, we're seeing Cursor talk about this. I think Harvey just mentioned something about this recently. And I'm sure there are lots of other companies that haven't talked about it yet. I know at our summit this week, there are a bunch of slides that had custom models showing up all over the place. And, you know, I won't say any more than is in those slides, but you can imagine, you know, that's because these use cases are so popular that we know they're going to be around for a while and it's worth committing. [00:15:58] S1: Yeah. You and I are probably two of the most eval-pilled people in the world, but I imagine that a lot of enterprises aren't there yet. Uh, how are you seeing that journey develop? Like, are enterprises actually building robust evals, or do you think that's more so something that they need to do over the next 12 months?
00:16:15 · Jonathan Frankle
[00:16:15] S0: It's kind of, I think it's a bit of a bell curve, but I wouldn't underestimate how fast enterprises are getting savvy about this. I met with like the C-suite of a major healthcare company a few months ago. People were gonna ask, like, what is AI? What is an agent? They were grilling me on the finer points of evals and judges. Like the CFO and like the CEO were grilling me on the finer points of judges. Okay, this is like it's getting there. It's getting there. I think we've been through enough of these big tech transitions in the past decade or two. You know, cloud, mobile, like, I don't know, crypto, say what you will about that, but you know, felt like a tech transition for a minute. You know, even just the web, that I think any company leader who's survived this long and has grown this far knows that when you see a tech trend coming, learn as much as you can about it and prepare yourself and your company for it. So I see people understanding the concept. The commitment to really do it and do it well enough that you can build a flywheel is harder. I met with a Databricks customer this week that has actually annotated 12,000 examples. Um, they're in a medically sensitive application where they have to be really careful. They have 70-something judges that they've built and calibrated. It's insanely impressive. Like I was blown away when I heard this. And I've been chatting with them for a couple of years. Initially, it was talking to them about the importance of judges two years ago. Yeah. And now they are the most judge-pilled, eval-pilled people in the world. And it's extraordinary. They're like the level of rigor is something beyond basically anything I've ever seen anywhere, even if you paid somebody to do it. But they need to because they're in an area where using AI related to medicine would be very risky otherwise. So I'm seeing them as a leading indicator of where things are going. And once you have that, the amount of value you start to get out of it is huge. I'm also seeing kind of this debate between online and offline evals. Like how much do you do eval that is, you know, a static set of examples that you can constantly run against, your kind of unit tests. And how much are you just inspecting traces every day?
00:18:00 · Jonathan Frankle, Brendan Foody
And how do you get a pipeline from inspecting traces back into your offline eval, kind of so that, you know, when you see something weird, you add a unit test to it? I don't think any of these processes have really been figured out yet. We're kind of, I know we've talked about this, like metaphorically, we're in the 1960s of software for AI and you know, we're still using punch cards. I can't wait for us to discover the object-oriented programming of what AI looks like. [00:18:22] S1: Well, and we've worked with so many customers that have the thumbs up, thumbs down flywheels that they try to initially use for eval, but obviously the challenge with that is even if you have thumbs down signal, that doesn't give you the full understanding of what was the problem with that, what could have been different that would have made that a correct solution, and how do you effectively hill climb that in a iterative way? And so that ties so much into how do we effectively understand the full problem space of what's possible, what's good, what's bad, and measure that in a reliable way, which ties into a lot of the importance of offline evals. But maybe going off of that, I imagine that in the enterprise, you've seen so many examples of enterprises that are sophisticated, like you're describing, but also a lot of examples of enterprises that aren't quite there yet. What have been your largest takeaways on what makes a good eval? When you use the word rigor, what does that mean in the enterprise context of these evals actually being useful?
00:19:16 · Jonathan Frankle
[00:19:16] S0: So I'll give you the like grand sweeping cosmic version of this and the highly concrete version of this. The cosmic version is like imagine Waymo cars. We let 16-year-olds on the road if they can drive in a circle in parallel park and not, you know, crash the car, basically. Would we let a Waymo on the road if it could pass that test? And like, you know, we let 16-year-olds literally go and drive on any highway in the country if they can like drive in a circle and turn the car around properly without crashing. It's kind of crazy. We would never let a Waymo loose on public highways if it could pass that test. Um, and you know, I some people say, well, we're unfair to the AI. And I look at it the other way as like, with AI, you can scale something to an arbitrarily large scale. You know, Waymo is limited only by the number of cars they can manufacture, but otherwise, one software change scales everywhere at once. And with AI, you know, we've kind of built our entire world around human strengths and weaknesses and the fact that humans are limited. You know, we live in meat space, not cyberspace, as they would say in the 90s, but AI is not. So the way that we measure AI has to have a degree of rigor like a Waymo car, you know, where you have four different LIDAR sensors and, you know, annotation going on over the course of a decade to measure every ounce of driving in a way that would have been unimaginable a decade ago. And, you know, that's not about measuring AI driving. That's about measuring driving. And, you know, if I had a 16-year-old child someday and they were learning how to drive, if driving is still a thing at that point, I would want them to drive a Waymo type car that has all those sensors and is giving me and them constant feedback and measuring their driving. The driving test should be how do you do over the course of several days driving in a car with that many sensors? So I think with AI, the risk increases if you're going to deploy AI widely. But it also gives us this opportunity to measure work with a degree of rigor that goes beyond a six-month performance review where somebody says, Yeah, I'm feeling this way about your work. Like I don't think anyone believes a performance review is particularly rigorous at any job in the same way the driving test is not rigorous. So, what does that look like for any kind of work?