The Frontier with Brendan Foody
00:00:00 · Jonathan Frankle, Brendan Foody
[00:00:00] S0: We were in a three-month period of token maxing and leaderboards and now we're god I can't believe I used this word in front of 1500 people. But I said the era of token maxing is over and the era of value mogging is here. I'm gonna cringe, but I said it anyway. Again. Recorded again. [00:00:19] S1: I'm Brendan Fuoty and welcome to the Frontier. Today I'm really excited to welcome Jonathan Frankel, who previously created the lottery ticket hypothesis research paper, which is one of the most famous papers in AI, and then subsequently co-founded Mosaic ML, which he sold to Databricks, where he became their chief scientist. Jonathan is one of the most unique perspectives I've seen in the industry because he simultaneously is an extraordinary researcher, but is in the weeds with enterprises, understanding all the data required to make their models work. So looking forward to the conversation. Jonathan, I'm so excited to have you. Thank you so much for joining today. It's great to be here. I'm really glad the timing worked out. It should be a lot of fun. I figured we can just get started uh talking about your background. Uh I mean, obviously the lottery ticket hypothesis has made waves. I'm never going to escape that, am I? For for a long time since you uh created it during your PhD. Maybe you can share a little bit about the lottery ticket hypothesis, and then we can uh go from there to uh what brought you to Databricks and where you are today.
00:01:22 · Jonathan Frankle
[00:01:22] S0: Yeah, I will say to my team, if you're out there and you happen to see this, someone mentioned lottery ticket hypothesis, take a drink. That is the the team drinking game at this point, is you know, anybody mentions lottery tickets, got to take a drink. It was interesting work for 2018. I think, you know, if you roll back the clock to 2018 when you were like 12, you were not how old were you in 2018? I'm kind of terrified. I was 15 in 2018. Damn it. Not far off. The gray is starting to show. No, in 2018, I think you know, deep learning was early, um, or at least earlier, at least this phase of deep learning was earlier. People were excited about it. Results were starting to show up. We had Image Net results, there's like some cool, you know, AI and music stuff happening at Google, like kind of little glimpses of what we have today. But it was still very much a debate about whether we were gonna like what our understanding of deep learning was going to be. Were we going to understand this from a nice, clean theoretical model? Was there some order to the world that we were going to be able to impose on this? Or was the world just going to be messy? And as computer scientists, at least my training was you never want to just have an empirical statement that like this algorithm seems to be this fast. You want a proof. You want a worst case bound. We want to know for certain. We're a field that comes from math. And I think that kind of mathematical strain is still kind of like somewhere deep in the DNA of anyone who's trained in computer science. I mean, the lasting part of the lottery ticket work is really to be completely empirical, to just ask, what do we see in real models? At that point, real models meant CFAR 10. I think it was 200,000 parameters, maybe a million parameters on a good day. Um, you know, the data sets were 32 by 32 pixel images of like trains and buses, and we were classifying them, and that was like a hardcore expensive task on my two K80s. If you really want to go back in time and like you've probably never even encountered a K80 in your life, um, I now have one on my wall. But the idea back then was this debate like, are we gonna have a grand unified theory of everything of deep learning and like have a true understanding of it?
00:03:08 · Jonathan Frankle, Brendan Foody
Or are we going to just have to measure and empirically find our way through by looking at the properties of models? And we're here today to talk about evals, like it's very clear which one won. [00:03:17] S1: Yeah.
00:03:18 · Jonathan Frankle
[00:03:18] S0: We're not proving that the model behaves well. We're measuring its behavior. In 2018, that was a really controversial thing to say. If you go and look at the lottery ticket hypothesis papers, like iClear reviews, because they're all public, the scores were 9, 9, 5, and the five was there's no proof of this. And I remember when I was on the faculty market, this is just this is stuck with me forever. Someone during an interview asked me, You've done all this empirical work. Do you plan to do anything principled? What this person meant was principled is synonymous in computer science with theoretical and provable. I was really insulted because like empiricism's principled. We can do empirical science in a principled way. That was the state of the field in 2018. So the work itself was about understanding the training dynamics of models. And I ended up, you know, having some kind of interesting, curious findings that were popular at the time. I don't think they really stood the test of time in terms of being that useful, quite frankly. But it was cool at the time. And I think really the lasting impact was empiricism is a lens into understanding how to use neural networks that allows us to go further than theory will allow us to go. We can ask bigger questions that we just can't prove or describe in a mathematical way. And today, I think we all just take that for granted. Like, is anyone doing math about deep learning? Like any of Mercore's customers doing like proofs to try to understand if their model is good? Mercore wouldn't exist if you could prove this.
00:04:37 · Brendan Foody
[00:04:37] S1: Yeah, it all has come down to empirics. And that was obviously an incredibly impactful contribution to the field. And talk a little bit more about what brought you from your PhD to Mosaic and the early days of that team and why you joined, how you got started, what you worked on.
00:04:53 · Jonathan Frankle
[00:04:53] S0: Yeah. I mean, what brought me to Mosaic was a cold email from my, you know, the person who became my CEO saying, nice paper, want to do a startup on it? It's like, who the hell are you? And why are you cold emailing me? I don't know. If you have an MIT email address as a PhD student, you get a lot of weird emails from weird people asking weird things. And I was just like, okay, another crazy person. And actually, there was this really wonderful friend and lab mate, Jesse, who I happened to be running into that day. And I was like, I got this weird email. And he's like, You don't know who that is? You've never heard of a startup before? Like, you're an idiot. Um, you should really talk to this guy. I was gonna just ignore it. So I'm really glad I opened the email. And, you know, that turned into mosaic. Um, you know, Naveen Rao reached out and said, Hey, we should do a startup. And I was like, that's dumb. The lottery ticket paper is not that useful and it's never gonna be valuable in the world. And I still think that was right. But we evolved that into something about training efficiency and trying to look for empirical ways to this is gonna sound really quaint, but changing the training algorithm to get better quality in the model more efficiently and being okay with that, even though it was a different algorithm, which today we call doing deep learning. But back then that was a very controversial thing because you're gonna get something different than you would have gotten otherwise. And people were really nervous about that. So it was a controversial thing at the time to say, yeah, we'll train your model, but it'll be trained slightly differently than you planned. We'll get you to the same quality on the eval, and that's all that matters. That was a scary thing for people, which today sounds so hilarious.