Venture with Grace
00:00:00 · S0, Grace Gong, Jerry Tworek
[00:00:00] S0: I actually don't know what it would look like, but... [00:00:02] S1: Okay. Hi, Jerry. Welcome to Venture with Grace. [00:00:06] S2: Thank you very much. Very happy to be here. [00:00:08] S1: So maybe we can talk about our Halloween special costume. [00:00:11] S2: Today is Halloween. [00:00:14] S1: Today is Halloween. Okay. You wanna give the rundown of, like, why you pick the Witcher? [00:00:18] S2: I'm the Witcher, Geralt of Rivia, Slayer of Monsters, and Transformers. [00:00:23] S1: Yeah. I'm the... Yeah. [00:00:24] S2: That's that's other person. I am on right now. There is specific architecture of neural network that I think got its place in everyone's homes and AI assistants, and we need to slay it and replace it with something I [00:00:39] S1: really like that you can tie anything into AI these days. I wanna start with... Okay, let's talk about Core Automation. Obviously, the big thing is, in the past, everybody's talking about transformers, and then you guys are kind of the transformer slayer. Yes. Tell us more.
00:00:39 · Grace Gong, Jerry Tworek
[00:00:39] S1: really like that you can tie anything into AI these days. I wanna start with... Okay, let's talk about Core Automation. Obviously, the big thing is, in the past, everybody's talking about transformers, and then you guys are kind of the transformer slayer. Yes. Tell us more. [00:00:57] S2: Transformers have been so successful that the AI researchers kind of, I think, became complacent and started believing transformer is the end of road. We've been able to scale those multiple orders of magnitude, and we've seen improvements and we've seen new skills. And I would like to be a little bit counterculture to that and show there are actually different ways how you can arrange your neurons, different ways how you can arrange your memory on a GPU that will yield itself to smarter models and better, more robust learning systems of the future. And this is what we are to do. I'm very happy in semi what transformers have already delivered. I'm very happy with the AI assistants that are there today. But there are always a better technology to be built. And this is what I, together with my company, have set out to do. [00:01:44] S1: I have so many questions. Number one is I feel like Transformers is kind of like... The specialization is kind of for identifying which part of the context or content is more important. So that's kind of the specialization. Then it also takes up a lot of training data and stuff like that. So for you guys, it sounds like you guys don't require a lot
00:01:44 · Grace Gong, Jerry Tworek
[00:01:44] S1: I have so many questions. Number one is I feel like Transformers is kind of like... The specialization is kind of for identifying which part of the context or content is more important. So that's kind of the specialization. Then it also takes up a lot of training data and stuff like that. So for you guys, it sounds like you guys don't require a lot [00:02:07] S2: of require data. Data is necessary. We think we can be more data efficient. There is some multiplier, but data is needed. And I think the biggest thing what we want to get rid of is, in some way, product specific, but even more importantly, customer specific fine Currently, the big labs are purchasing huge amounts of data for how to make models good at health, how to make them good for lawyers, how to make them good for programmers, how to make them good for making valuation of companies. And I think it's unrealistic for the models to have knowledge in one place of every profession and everything, what we want them to do. What we at Core Automation want to do, we want to build a model that learns from the user data and on the fly working with the user. So if you are a law firm, you don't need to necessarily buy off the shelf model that knows everything about your company's cases, but can learn together with you as you bring it along and show it to the data that you actually care about and you wanted to do, the model improves from that. So move more of the learning from the lab into actual customer sense.
00:02:07 · Jerry Tworek, Grace Gong
[00:02:07] S2: of require data. Data is necessary. We think we can be more data efficient. There is some multiplier, but data is needed. And I think the biggest thing what we want to get rid of is, in some way, product specific, but even more importantly, customer specific fine Currently, the big labs are purchasing huge amounts of data for how to make models good at health, how to make them good for lawyers, how to make them good for programmers, how to make them good for making valuation of companies. And I think it's unrealistic for the models to have knowledge in one place of every profession and everything, what we want them to do. What we at Core Automation want to do, we want to build a model that learns from the user data and on the fly working with the user. So if you are a law firm, you don't need to necessarily buy off the shelf model that knows everything about your company's cases, but can learn together with you as you bring it along and show it to the data that you actually care about and you wanted to do, the model improves from that. So move more of the learning from the lab into actual customer sense. [00:03:19] S1: Wait. Say more. So basically, I feel like nowadays, most amount of people come out of our podcast talking about open weight models. So essentially, you download stuff and then running stuff within your own database. But how do you guys kind of different from that?
00:03:19 · Grace Gong, Jerry Tworek
[00:03:19] S1: Wait. Say more. So basically, I feel like nowadays, most amount of people come out of our podcast talking about open weight models. So essentially, you download stuff and then running stuff within your own database. But how do you guys kind of different from that? [00:03:36] S2: First and foremost, we use different algorithms. The problem with open weight models is that fine tuning is hard and efficient. It requires a lot of people, and the results are not really guaranteed and not really very good. [00:03:49] S1: So you guarantee results? [00:03:50] S2: Yes. And we want to, through a different architecture, we can provide a lot of the things that are just hard to do with fine tuning transformers. On one hand, you have a neural network. On the other, you have data. And there's a question, how do you combine those things together? Fine tuning is one way to do it, which arguably is just not very good. And we are building alternative to that that we think will be much, much better. [00:04:18] S1: Okay. So I wanna talk about... Which part do you wanna talk about more? Like, the people or the business logic? Because I feel like, obviously... Let's talk about the people because, like, you know, you guys obviously hire like the chief people officer, like the former chief people officer of OpenAI, who like, obviously have been at Hudson Trading for over fifteen years, who have like, I'm sure like she have like, recruited like multiple, like, a ton of, like, amazing talents. And then last time I tracked on your team is, you know, like, 16, 20 people, something like...
00:04:18 · Grace Gong, Jerry Tworek
[00:04:18] S1: Okay. So I wanna talk about... Which part do you wanna talk about more? Like, the people or the business logic? Because I feel like, obviously... Let's talk about the people because, like, you know, you guys obviously hire like the chief people officer, like the former chief people officer of OpenAI, who like, obviously have been at Hudson Trading for over fifteen years, who have like, I'm sure like she have like, recruited like multiple, like, a ton of, like, amazing talents. And then last time I tracked on your team is, you know, like, 16, 20 people, something like... [00:04:53] S2: I'm only seven at this time. [00:04:54] S1: Okay. Wow. I [00:04:54] S2: remember our girl. [00:04:55] S1: Grew seven since, like, the last time Internet has updated. So I have so many questions in the AI talent space. So obviously, one of the things is like, talent nowadays, like, worth a lot more than the past. But nowadays, like, so many decisions are actually made by AI. Like, why is some people more valuable than others? And when you guys are thinking about talent planning, what is considered a good talent? So you come from the trading era or like trading... Not era, but like the trading squad. So why is that a better, significantly better AI people than people who was studying computer science?