Venture with Grace
00:33:21 · Grace Gong, Jerry Tworek
[00:33:21] S1: to do it now. [00:33:22] S2: So I was thinking that the window is actually very short for starting a type of company I want So to I didn't have much time to hesitate. The first and foremost reason is you need agents to be good enough to kickstart you. Without agents, starting an AI lab is a very slow process and requires a lot of time. If you see the neo labs that started a year ago, two years ago, they were moving in a glacial pace initially. The amount of systems you need to do, the amount of work that you need to build out to get anywhere is just a lot. With modern AI solutions, we're able to probably compress two years of work in three months. It's amazing how much quicker you can move right now than ever before. And the second thing is you don't want to be too late from a perspective of already someone being pretty far advanced into some types of research that are doing. I think we are leading the guard into finding something that is very different to transformer better than transformer in agentic workloads. But eventually, other companies will be trying to ask the same questions. They are, I think right now, today, still busy with training transformers. But that will definitely change. [00:34:36] S1: How do you meet Rohan? Plug from our show. So, like, actually, Rohan? Let's look at our subject.
00:34:36 · Grace Gong, Jerry Tworek
[00:34:36] S1: How do you meet Rohan? Plug from our show. So, like, actually, Rohan? Let's look at our subject. [00:34:45] S2: Rohan, my co founder, great scientist and great passionate of deep learning. When I was trying to assemble a crew for that company, you can imagine you give people phone calls that you roughly know and roughly know their research. Hey, I'm assembling a crew. This is what we are trying to do. Are you in or not? And Rohan was one of the people who answered that call and said, like, you know, you, Jerry. I mean... [00:35:10] S1: Oh, wow. Like, no one... Oh, I mean, like, I'm sure, like, is there anyone said no? No? [00:35:14] S2: Like, there are definitely some people that said no. And, like, our our our kind of, like, ratio of people who decide to join is fluctuating over time. But yeah, especially early on, we had a pretty good close rate. [00:35:26] S1: What would you think... I guess, what makes people join you guys? Is that just because you guys are solving the word class? The word first... Only the first word problem right there.
00:35:26 · Grace Gong, Jerry Tworek
[00:35:26] S1: What would you think... I guess, what makes people join you guys? Is that just because you guys are solving the word class? The word first... Only the first word problem right there. [00:35:36] S2: Yeah. At least one thing, we have a very good time of hiring excellent researchers, because I think we are one of the companies with both an ambitious research plan, solving interesting problems. And actually, we are able to pretty well articulate it for those who come long enough in journey. And definitely, for other roles, we are slightly more eye to eye with other companies. If you work on systems or on data at Core Automation, the main part of the story is you will try to develop those things in as agentic way as possible using as much automation. You will be trying to become a person that does the equivalent of a work of a team in places. And that's our advantage. [00:36:29] S1: According to Twitter, I don't think you like eval these days. [00:36:32] S2: Yes. I think I think evals mostly lead you to good hearting and to thinking about things too narrowly. There were... [00:36:39] S1: Eval's kinda like advertising. Right? So it's like you just pick whatever you're good at and then pick the right evaluation things too.
00:36:39 · Grace Gong, Jerry Tworek
[00:36:39] S1: Eval's kinda like advertising. Right? So it's like you just pick whatever you're good at and then pick the right evaluation things too. [00:36:45] S2: There there there is very little signal in those things. Everyone who ever releases a model, they say, oh, we are good on those evals, and they cherry pick, and they kind of present their models as good. And the actual market value of those models is very different to what those benchmark says. And because those evals are public, everyone just tries to look good. And there are much deeper differences between models. And unfortunately, because there's just not really a good way to take, people are trying all those different models and trying to figure out at least those that want to spend energy on it and people who don't want to spend energy just take the most popular model and run away with it. But I don't think it's even a very good way of thinking about both models, about evals. It's a bit of an acronus of times when the models were frozen and were not really meaningfully learning after they are deployed. So if you take a GPT off the shelf, it has some evals because that's what it was been trained on. But if the model can learn, then those eval scores are different tomorrow. Even can be different from minute to minute. [00:37:50] S1: How do you use... I mean, how do you know if your stuff works well?
00:37:50 · Grace Gong, Jerry Tworek
[00:37:50] S1: How do you use... I mean, how do you know if your stuff works well? [00:37:57] S2: Well, I know. I test it. I try it. You try it? Ideally, ideally, on the on the work that we are that we are doing. If the model does for me what I wanted to do, then then I am a happy person. If it doesn't, that's [00:38:11] S1: Oh, that's not very comfortable.... Just kidding. What is... I guess, like, when you're thinking about building a successful lab and building a successful company... Let... Let's talk about the researchers. Right? Who are the good researchers? I mean, obviously, I've done some researcher events myself. And I think in the market or in the startup land, people are trying to hire people from OpenAI or Anthropic. Even... Everyone's title is a member of technical staff, and now nowadays there's a member of nontechnical staff. So how do you... Who is the right person? I mean, obviously, people publish research papers. And how do you identify who is the right fit? And then you mentioned about you ensembling your own squad. People have different functions. And what are the functions that are typically the most difficult to hire versus the most easier to fulfill?
00:39:07 · Jerry Tworek
[00:39:07] S2: I don't think there are that much. It's mostly an intersection of a specific timing and who is in your network of the role that is easier to hire or who is more and more enthusiastic. But there are, in general, there are multiple roles you need to fill in the lab. And there are, obviously, the world, people doing that. The question is, is there a right person with the right skill set particularly right now interested in what you are doing in your company? I think what makes good researchers is one thing. They understand what they are doing, what they are working on. This is probably the biggest difference between, let's say, junior researchers and senior researchers. The time you spend thinking deeply about the topics you work with, you have somewhat a better and better grasp over time and mental understanding of neural networks and their dynamics of training. You generally want to hire people who avoid group think and who avoid just I'm waiting following things. The So difficult. It is difficult, but that's what you want specifically for researchers the most. The researcher that just thinks what everyone else is thinking is not really a very useful researcher, because research is about solving things that are unsolved and trying to go beyond the current methods, which is kind of like tricky in the current world where the stakes are very high and the dynamics between companies is what it is, but it's necessary to succeed, in my opinion. And last but not least, the most successful researcher generally understands systems and understands GPUs pretty well, because that's, in the end, is the subset we are working with, and that's the layer from which all of deep learning comes from. And we need to be able to combine the insights we have about the neurons that we are training with the math models that we are executing and how do we find the good middle ground between.