Venture with Grace
00:22:18 · Grace Gong
[00:22:18] S0: Can we talk about like, I guess, like when you're working with the current companies, maybe you want to talk about like, would the enterprise be the target customer compared to like, more of like fast growing AI natives? Or like, how does like the customer aspect kind of like shape up? And when you guys are like working with the enterprise clients, like, what is like a common mistake that most people are have kind of like being just making the mistakes? And the other part is like when you're working with these companies, is there any kind of like checking framework? So for example, you just track like the core three things typically that's like the most amount that's expended by the company or how does that kind of work? So basically when you're actually tracking the token, where do you start?
00:23:10 · Vitaly Gordon
[00:23:10] S1: Yeah. So we are... Historically, we've been, you know, working with a lot of, I would say, yes, like, traditional enterprises. I would say now it's of the... There's more and more kind of AI native companies because, you know, they can have far fewer engineers, but still, like, their engineering output and kind of the bollocks might be actually, you know, larger than kind of some enterprises. So that's the... That's gonna other type of customers. So now what is the kind of, you know, the typical mistakes? And the typical mistakes is just like... I would probably say is like kind of looking for shortcuts. And the shortcuts are it's kind of like, let's look at, like, you know, before, in the... You know, when AWS just came out. Like, if you would go to, like, a large enterprise and just say, you know, give every developer a credit card, say, hey, you can now can spend, like, $20 on... A month or whatever, like, on AWS. That does not make you a cloud native company. Right? Just like the fact that... But somehow, we are doing similar things where we just, like, saying to every single engineer, oh, you know, here, you know, have get a pop out or cloud code or cursor or whatever, you know, tool you want, and now you're an AI native company. Like, again, unfortunately, that's just not how it works. Like, you cannot just basically buy your way into becoming AI native. Right? And so... And of the the way... And I shared at least kind of, you know, one metric that we go go through. So we start with, like, okay, measuring the outcomes, measuring the price per per outcome, but then start seeing, you know, distribution of word spend. So, like, I'll give you an example from, you know, one of our customers is what we found is their most kind of efficient engineers... And by the way, there could be, like, literally 10 times more efficient on a price per outcome. Could be also, like, their most expensive engineers, but could be also the... Their kinda cheapest engineers in terms of their salary and their AI spend because they're just so productive that, they actually, you know, aren't using AI in great manner. So one of the things that we also look at is on what can we learn from the best engineers who use AI and, like, what are they doing? We now...
00:25:22 · Vitaly Gordon
She added another kind of pillar to our platform that we're very excited about that will be coming soon, but it's more about education and kind of, like, how can we actually disseminate those best practices because... And I will say that's probably, like, the biggest problem with AI is, like, everyone has access to it. But, like, we see, like, a very, very wide range between, like, just how proficient people are using kind of those best practices compared to to other. So like, you know, we then go and kind of find the best engineers and what what they're doing. And also, like, where there are things that are, you know, still kind of like basic good practices of like, like, you know, companies are just giving, you know, like having these silly things where, like every single engineer has the same like budget for AI. Like you will never like think about allocating headcount this way. Like, okay, every single engineer can hire whoever they want. Right? And like, whoever hires first gets it, but somehow, like, we we do it with, you know, AI versus saying, well, you know, there's, like, projects that are strategic or projects that are important. There's like, and there's, like, side projects that people do. Like, we we should not just, go in, like, you know, get... Like, let, you know, the the people themselves decide where the budget should be spent. Like, we should kind of think about it. We should allocate it accordingly. And so those are a lot of the exercise we do where we work with customers and saying, okay. You know, what is important? Like, what are you trying to achieve? And let's make sure that kind of, you know, all the kind of, you know, arrows are pointing in that direction as opposed to, like, just kind of, you know, like I said, giving everyone some, you know, basic budget for AI and like, hopefully, like, oh, and, you know, let's wait now for two months and I will, you know, my company will start flying.
00:27:18 · Grace Gong
[00:27:18] S0: There's like, so like some specific token bill kind of like data that in your report, which is like, you compare like open model versus like frontier model on two eleven real tasks. Like the open route is like about 0.92 tasks versus amount, almost $2 for Claude Opus and GPT with similar quality. Are our enterprise overpaying for the Frontier Labs for coding work or no?
00:27:52 · Vitaly Gordon
[00:27:52] S1: Yeah, so you know what? And this is the crazy thing that since we published this report, there is like, I don't know, like 50 new models. It's just on a daily basis, this thing needs to be updated. This, by the way, what we're doing, this is what's just so hard about it, is when we talk to customers, they're all now just, you know, frustrated and just overwhelmed with just the kind of, you know, the barrage of innovation that is coming to it. And by the way, this is obviously not a bad thing. It's a great thing, but it's just like... It's just overwhelming. And so every time, like, what was it? We're recording on a Wednesday. I think, you know, on Monday, there was, you know, two major releases, a new Kimi model and a new, you know, Terra 6.1 from OpenAI. And both of these models are great. And like... But the question is like, you know, okay, great. What what does great mean? It's like, okay, it's great on public benchmarks. But like, in most organizations, like, the work that they do is fairly specific. Right? If you're, you know, a large bank and you're working on a, like, banking mobile app, like, you know, maybe the latest frontier model is just, like, not, you know, like, a lot of things that you don't need. But on the other hand, those models also becoming much more token efficient. Right? So maybe, actually, maybe you can use it and actually save money. And and and this is not even talking about, like, you know, like, Anthropic obviously has their, you know, Fable, Opus, Sonnet Haiku, you know, kind of... Codex has their, you know, Astra, you know, Sol, Terra Luna kind of a family of models, then you add to it to that each one of them has, like, a thinking kind of, like, scale. It is a... There's just, like, way, way, way too many options. Right? And, like, you just, like, in... You know, knowing which option is good for which task is just, like, impossible to know. And so because of that, like, what... Sometimes we see there's, like, organizations that have, like, full teams that are going in, like, you know, running evals on these models on kind of tab. But there was, like, you know, ways to automate it, which is also another thing that we build companies with, like, private benchmarks that just every time there's a new model come...
00:30:05 · Vitaly Gordon
Comes in, we quickly evaluate it and then we kind of, like, recommend, different kind of routing, for different tasks and, like, in... It's just completely automated enough, for the users. So that's kind of, like, happening now. If you add to it... So your question was about open weights model. You know, this just adds another kind of thing. Like, there's like amazing open weights model, but like, I think I heard, you know, something this week. I, you know, I didn't look that much into it. Cohere found that there was, like, something fishy in one of these, like, new models that it sends data to. And so there's, like, a lot of people that are, you know, concerned. Like, I I don't think we're really understand the, you know, risks with running kind of, like, open weights model and who runs them and who is responsible for them. So because of that, I think still the majority of companies will go with, like, the Frontier Labs, because they trust them. And it... Because it's honestly in make... In many cases easier to just... Like, you don't have to worry about compute. I'll give you just an example again from a customer. You know, when JEV came out, there was, like, you know, immediately 50 clone... Open source clones of JEV that it came out. And, you know, they're great because... Well, on one end, JEV is kind of, you know, fairly cheap, so, like, you should not worry about using it for the price. But there is, like, a lot of companies that it... The model was still still not approved for them to use, so they were looking for an alternative they can run themselves. And so they found open source and say, okay, well, this is free, but no, like, try to get the GPU on AWS right now. It's impossible. Right? So so actually, like, in the field, or you will have to pay a lot more. And so, you know, so... Again, while you kind of add the complexity of, like, you know, deploying these models and the GPU and the time it will take and making sure that when chasing all that, it's not clear that the open weights math might work for it. So it's just an ever evolving case which just requires a lot of attention. So there is no one definitive answer. You should use OpenWays or should you use Frontier. Unfortunately, it really depends.