Venture with Grace
00:25:22 · Vitaly Gordon
She added another kind of pillar to our platform that we're very excited about that will be coming soon, but it's more about education and kind of, like, how can we actually disseminate those best practices because... And I will say that's probably, like, the biggest problem with AI is, like, everyone has access to it. But, like, we see, like, a very, very wide range between, like, just how proficient people are using kind of those best practices compared to to other. So like, you know, we then go and kind of find the best engineers and what what they're doing. And also, like, where there are things that are, you know, still kind of like basic good practices of like, like, you know, companies are just giving, you know, like having these silly things where, like every single engineer has the same like budget for AI. Like you will never like think about allocating headcount this way. Like, okay, every single engineer can hire whoever they want. Right? And like, whoever hires first gets it, but somehow, like, we we do it with, you know, AI versus saying, well, you know, there's, like, projects that are strategic or projects that are important. There's like, and there's, like, side projects that people do. Like, we we should not just, go in, like, you know, get... Like, let, you know, the the people themselves decide where the budget should be spent. Like, we should kind of think about it. We should allocate it accordingly. And so those are a lot of the exercise we do where we work with customers and saying, okay. You know, what is important? Like, what are you trying to achieve? And let's make sure that kind of, you know, all the kind of, you know, arrows are pointing in that direction as opposed to, like, just kind of, you know, like I said, giving everyone some, you know, basic budget for AI and like, hopefully, like, oh, and, you know, let's wait now for two months and I will, you know, my company will start flying.
00:27:18 · Grace Gong
[00:27:18] S0: There's like, so like some specific token bill kind of like data that in your report, which is like, you compare like open model versus like frontier model on two eleven real tasks. Like the open route is like about 0.92 tasks versus amount, almost $2 for Claude Opus and GPT with similar quality. Are our enterprise overpaying for the Frontier Labs for coding work or no?
00:27:52 · Vitaly Gordon
[00:27:52] S1: Yeah, so you know what? And this is the crazy thing that since we published this report, there is like, I don't know, like 50 new models. It's just on a daily basis, this thing needs to be updated. This, by the way, what we're doing, this is what's just so hard about it, is when we talk to customers, they're all now just, you know, frustrated and just overwhelmed with just the kind of, you know, the barrage of innovation that is coming to it. And by the way, this is obviously not a bad thing. It's a great thing, but it's just like... It's just overwhelming. And so every time, like, what was it? We're recording on a Wednesday. I think, you know, on Monday, there was, you know, two major releases, a new Kimi model and a new, you know, Terra 6.1 from OpenAI. And both of these models are great. And like... But the question is like, you know, okay, great. What what does great mean? It's like, okay, it's great on public benchmarks. But like, in most organizations, like, the work that they do is fairly specific. Right? If you're, you know, a large bank and you're working on a, like, banking mobile app, like, you know, maybe the latest frontier model is just, like, not, you know, like, a lot of things that you don't need. But on the other hand, those models also becoming much more token efficient. Right? So maybe, actually, maybe you can use it and actually save money. And and and this is not even talking about, like, you know, like, Anthropic obviously has their, you know, Fable, Opus, Sonnet Haiku, you know, kind of... Codex has their, you know, Astra, you know, Sol, Terra Luna kind of a family of models, then you add to it to that each one of them has, like, a thinking kind of, like, scale. It is a... There's just, like, way, way, way too many options. Right? And, like, you just, like, in... You know, knowing which option is good for which task is just, like, impossible to know. And so because of that, like, what... Sometimes we see there's, like, organizations that have, like, full teams that are going in, like, you know, running evals on these models on kind of tab. But there was, like, you know, ways to automate it, which is also another thing that we build companies with, like, private benchmarks that just every time there's a new model come...
00:30:05 · Vitaly Gordon
Comes in, we quickly evaluate it and then we kind of, like, recommend, different kind of routing, for different tasks and, like, in... It's just completely automated enough, for the users. So that's kind of, like, happening now. If you add to it... So your question was about open weights model. You know, this just adds another kind of thing. Like, there's like amazing open weights model, but like, I think I heard, you know, something this week. I, you know, I didn't look that much into it. Cohere found that there was, like, something fishy in one of these, like, new models that it sends data to. And so there's, like, a lot of people that are, you know, concerned. Like, I I don't think we're really understand the, you know, risks with running kind of, like, open weights model and who runs them and who is responsible for them. So because of that, I think still the majority of companies will go with, like, the Frontier Labs, because they trust them. And it... Because it's honestly in make... In many cases easier to just... Like, you don't have to worry about compute. I'll give you just an example again from a customer. You know, when JEV came out, there was, like, you know, immediately 50 clone... Open source clones of JEV that it came out. And, you know, they're great because... Well, on one end, JEV is kind of, you know, fairly cheap, so, like, you should not worry about using it for the price. But there is, like, a lot of companies that it... The model was still still not approved for them to use, so they were looking for an alternative they can run themselves. And so they found open source and say, okay, well, this is free, but no, like, try to get the GPU on AWS right now. It's impossible. Right? So so actually, like, in the field, or you will have to pay a lot more. And so, you know, so... Again, while you kind of add the complexity of, like, you know, deploying these models and the GPU and the time it will take and making sure that when chasing all that, it's not clear that the open weights math might work for it. So it's just an ever evolving case which just requires a lot of attention. So there is no one definitive answer. You should use OpenWays or should you use Frontier. Unfortunately, it really depends.
00:32:16 · Grace Gong
[00:32:16] S0: You said that coding agents fail mostly for one dumb reason, which is obeying prompt instruction to literally. What would you say is like the most like ridiculous like agent failure you've seen and...
00:32:31 · Vitaly Gordon
[00:32:31] S1: Oh my god. Well, look, there is like, there's just, like, you know, so many so many problems from... Like, look. At the worst case scenario is, like, you know, kind of sometimes in order for, you know, like, some test, like, you know, these agents would go and delete a database. Right? Because when you... Especially, like, go in YOLO mode, that's probably, like, the... Like, you know, we have them. Obviously, those models became smarter over time, but it still it still happens. And, like, they maybe, you know, not delete a database, but would, you know, delete, you know, bunch of files that you need in your system. The... A little bit less kind of things, like, they're, you know, kind of like sometimes will fail to, you know, the test that will delete the test. Right? And so like, okay, awesome. That's not the, you know, what I, you know, I intended. So there is kinda like, you know, still, I would say, even alignment issues, you know, with coding. And and I would say probably something that just happens far more often, maybe, you know, not as annoying, is, like, in many cases, they just add a bunch of code that you didn't ask for. Right? Like, okay. You know, you asked for, like, hey. I just want to fix this, you know, one small bug and, you know, just add it. And then they, you know, added the, you know, like, a bunch of tests on documentation and some UI for you to, you know, verify it and, like, you know, they just, like, go and do a lot of things because they're trying to help. And it's just, you know, it's just hard to control these, you know, billion dollar boxes that, you know, have a mind of their own.