This Week in Startups
00:39:18 · Hon Weng Chong, Alex Wilhelm
[00:39:18] S3: Yeah, we could do that. But you know, I I I think for us, we we wanna make sure that, you know, we we... I don't wanna do a Kickstarter kind of thing. Right? At the start of [00:39:28] S1: the I don't I don't think that's I don't think that's Kickstarter ish, but I I respect where you're at. I guess then the the correct question to close with is this. Sometimes people say that venture capital is a little bit conservative. I think the VCs that bet on cortical were clearly being relatively adventurous, which is good. So as you guys do need more, capital, is the venture capital ecosystem in Australia, Europe, and the West at large? Is it interested in putting more money into the business, or are you guys still a little bit outside of what you might call it venture norms? [00:39:58] S3: Yeah. I think we're outside the venture norms. I don't know what it is, but the venture industry likes to gravitate to thematics. So in this case it's been, what is it? We went from transformer LLMs to agents and something whatever next year brings. It's very hard, I think, unless you fall into specific thematics to track that. So, you know, I think we have some of the best investors in the world backing us because they're just the ones who are willing to take bets, which is essentially what the industry initially started out with. Right? [00:40:34] S1: Blackbird Ventures and Horizons Ventures.
00:40:37 · Hon Weng Chong
[00:40:37] S3: Correct. Yeah. And we've had a new round that was just put together to get us a little bit more fuel to deliver our products by kind of people like 3C AGI, they've just come into the game. And we've had players like in Q Tel as well who had backed us. It's really about, I think trying to figure out who are the Mavericks in the space, who are the ones who have the imagination, right? And the sci fi background. Because at the end of the day, if you have 200, like maybe not 200, maybe like say even 10 foundational model companies, but they're all pretty much... I couldn't really tell you the difference between Claude versus Gemini versus ChachiBT versus, I don't know, what about Quen or Deep Sea? They are mostly the same. And so if they're all mostly the same and I can jump between one another, I don't really see any, you know, significant stickiness or moat to it. So anyway, that's a problem that the VC's do to resolve. But, rather than all jumping in into one space, I think we should try to spread it out. So for us, we're very focused on delivering the product and we welcome anyone who's interested, who's listening to your podcast, who understands the fundamental nature of building hardware first before you can get the software, you know, to to to look into the space. Because the the thing as as we already talked about, the AI space already had that groundwork put in by the gamers. Right? You didn't... We didn't need to invest in the hardware because the gamers just bought it already. So I think that's something that we have to bear in mind with any new compute space. There is always gonna be a hardware component, and we we we shouldn't shy away from having to, invest in... And build into that space.
00:42:18 · Alex Wilhelm, Hon Weng Chong
[00:42:18] S1: Well, I do I do look forward to the eventual future when we have Canva in Sydney, and we have Cortical in Melbourne, and we'll have a good old fashioned competition about who's gonna be the next future leader of Australian technology. Han, thank you so much for coming on. Thank you for answering all of my questions. And, when you do have so much extra production capacity, I'll give you my address to ship my CL one, and you just tell me where to send the check. Okay? [00:42:40] S3: Alright. Thanks, Alex.
00:42:40 · Hon Weng Chong, Alex Wilhelm, Jonathan Siddarth
[00:42:40] S3: Alright. Thanks, Alex. [00:42:41] S1: Thanks, Han. When Meta invested in Scale AI, absorbing its CEO and cofounder in the process, startups that competed with Scale saw an absolutely huge opportunity. Now mostly people focused on the data labeling side of what Scale had done historically as the place that startups might actually capture the most market share. But Scale also offered AI evaluation tools to folks who build LLMs. So startups that offer AI evaluation tools may also be in line to benefit from Scale aligning with Meta, a company that competes with other firms to build the next great AI model. Why work with Scale if Meta owns about half of it? One startup CEO, Turin's Jonathan Siddharth, told Reuters after Scale's partial exit to the social giant that leading AI labs now realize that neutrality is no longer optional amongst service providers. It's, quote, essential. So to help us understand the LLM evaluation market, just how big it is, and how data comes into play in 2025 to build those next models, please welcome to the show. It's Turing CEO, Jonathan Sarath. Jonathan, hey, welcome to the show. [00:43:44] S0: Thank you, Alex, for having me. It's great to be here. [00:43:46] S1: I'm very impressed. Also, we're both in our home offices today. And I think this just goes to show that remote work, not entirely dead, even though everyone seems to claim that it is. I'm glad that you're here. So sorry for folks who are less aware of what LLM evaluation is. Donathan, can you just give us the working definition from your side of the fence?
00:43:46 · Alex Wilhelm, Jonathan Siddarth
[00:43:46] S1: I'm very impressed. Also, we're both in our home offices today. And I think this just goes to show that remote work, not entirely dead, even though everyone seems to claim that it is. I'm glad that you're here. So sorry for folks who are less aware of what LLM evaluation is. Donathan, can you just give us the working definition from your side of the fence? [00:44:04] S0: Yeah. So, it's really important to benchmark LLMs to figure out where they are good at, where are gaps in the models so that you can generate data that could help the LLMs get better at those specific tasks. One big challenge, Alex, that I see today in the overall evaluation and benchmarking market is that a lot of the evaluations are somewhat academic and somewhat synthetic and don't connect to real world applications or real world use. Ideally, you want AGI to progress in a way where the models get better at useful tasks. So when we evaluate models, the three dimensions to look at are complexity. Are you evaluating them on really hard tasks? Real world use. Are you evaluating them on something that a human would actually care about? And third is diversity. You want a wide breadth of test cases that you're evaluating the models on. It's such an exciting space. And I think of evaluation as step zero of data generation. That's why we do both evaluation and data generation.
00:45:14 · Alex Wilhelm
[00:45:14] S1: Okay. So let's talk about the benchmarks because there's been some commentary. I think Apple had a paper that came out, and they said that, you know, one of the problems we have with a lot of the benchmarks that everyone likes to shot out their new LLM and put them up against and have the charts is that there's data contamination issues. There's overfitting. So based on what you just said about trying to solve real world problems and also the fact that we know that these benchmarks are getting a little bit, I don't know, dicey to use, is the standard way that companies announce how their LLMs perform an effective form of evaluation, or is that mostly window dressing to get more social media hits because you're one point higher on one test or another?