This Week in Startups
00:38:21 · Alex Wilhelm, Hon Weng Chong
[00:38:21] S1: So you have 100x more demand [00:38:23] S3: We have than 100x, yeah. And so they're on the waiting list and so forth. And then, you know, for the hardware, we have had a lot of labs and research groups reach out to us. We are, you know, very excited about this, but now we're looking at the challenge of, oh God, now we actually have to ship this thing and we'd have to ship lots of them. And if they break in the field, we're kind of screwed. So we better make sure they don't break out there. So there's a lot of like, we're doing our best at the moment to finalize the software, finalize the hardware, getting our partners ready. It's also very, very expensive CapEx wise, right? Because we have to, we don't charge our customers until we ship them, but we still have to build the units first. And so trying to figure out where the next source of funding is gonna come from is going to be a bit of a challenge. [00:39:11] S1: Han, why don't I just charge them half upfront and resolve some of your cash flow issues? [00:39:15] S3: Well, we could, but you know, it's... [00:39:18] S1: Do it. [00:39:18] S3: Yeah, we could do that. But you know, I I I think for us, we we wanna make sure that, you know, we we... I don't wanna do a Kickstarter kind of thing. Right? At the start of
00:39:18 · Hon Weng Chong, Alex Wilhelm
[00:39:18] S3: Yeah, we could do that. But you know, I I I think for us, we we wanna make sure that, you know, we we... I don't wanna do a Kickstarter kind of thing. Right? At the start of [00:39:28] S1: the I don't I don't think that's I don't think that's Kickstarter ish, but I I respect where you're at. I guess then the the correct question to close with is this. Sometimes people say that venture capital is a little bit conservative. I think the VCs that bet on cortical were clearly being relatively adventurous, which is good. So as you guys do need more, capital, is the venture capital ecosystem in Australia, Europe, and the West at large? Is it interested in putting more money into the business, or are you guys still a little bit outside of what you might call it venture norms? [00:39:58] S3: Yeah. I think we're outside the venture norms. I don't know what it is, but the venture industry likes to gravitate to thematics. So in this case it's been, what is it? We went from transformer LLMs to agents and something whatever next year brings. It's very hard, I think, unless you fall into specific thematics to track that. So, you know, I think we have some of the best investors in the world backing us because they're just the ones who are willing to take bets, which is essentially what the industry initially started out with. Right? [00:40:34] S1: Blackbird Ventures and Horizons Ventures.
00:40:37 · Hon Weng Chong
[00:40:37] S3: Correct. Yeah. And we've had a new round that was just put together to get us a little bit more fuel to deliver our products by kind of people like 3C AGI, they've just come into the game. And we've had players like in Q Tel as well who had backed us. It's really about, I think trying to figure out who are the Mavericks in the space, who are the ones who have the imagination, right? And the sci fi background. Because at the end of the day, if you have 200, like maybe not 200, maybe like say even 10 foundational model companies, but they're all pretty much... I couldn't really tell you the difference between Claude versus Gemini versus ChachiBT versus, I don't know, what about Quen or Deep Sea? They are mostly the same. And so if they're all mostly the same and I can jump between one another, I don't really see any, you know, significant stickiness or moat to it. So anyway, that's a problem that the VC's do to resolve. But, rather than all jumping in into one space, I think we should try to spread it out. So for us, we're very focused on delivering the product and we welcome anyone who's interested, who's listening to your podcast, who understands the fundamental nature of building hardware first before you can get the software, you know, to to to look into the space. Because the the thing as as we already talked about, the AI space already had that groundwork put in by the gamers. Right? You didn't... We didn't need to invest in the hardware because the gamers just bought it already. So I think that's something that we have to bear in mind with any new compute space. There is always gonna be a hardware component, and we we we shouldn't shy away from having to, invest in... And build into that space.
00:42:18 · Alex Wilhelm, Hon Weng Chong
[00:42:18] S1: Well, I do I do look forward to the eventual future when we have Canva in Sydney, and we have Cortical in Melbourne, and we'll have a good old fashioned competition about who's gonna be the next future leader of Australian technology. Han, thank you so much for coming on. Thank you for answering all of my questions. And, when you do have so much extra production capacity, I'll give you my address to ship my CL one, and you just tell me where to send the check. Okay? [00:42:40] S3: Alright. Thanks, Alex.
00:42:40 · Hon Weng Chong, Alex Wilhelm, Jonathan Siddarth
[00:42:40] S3: Alright. Thanks, Alex. [00:42:41] S1: Thanks, Han. When Meta invested in Scale AI, absorbing its CEO and cofounder in the process, startups that competed with Scale saw an absolutely huge opportunity. Now mostly people focused on the data labeling side of what Scale had done historically as the place that startups might actually capture the most market share. But Scale also offered AI evaluation tools to folks who build LLMs. So startups that offer AI evaluation tools may also be in line to benefit from Scale aligning with Meta, a company that competes with other firms to build the next great AI model. Why work with Scale if Meta owns about half of it? One startup CEO, Turin's Jonathan Siddharth, told Reuters after Scale's partial exit to the social giant that leading AI labs now realize that neutrality is no longer optional amongst service providers. It's, quote, essential. So to help us understand the LLM evaluation market, just how big it is, and how data comes into play in 2025 to build those next models, please welcome to the show. It's Turing CEO, Jonathan Sarath. Jonathan, hey, welcome to the show. [00:43:44] S0: Thank you, Alex, for having me. It's great to be here. [00:43:46] S1: I'm very impressed. Also, we're both in our home offices today. And I think this just goes to show that remote work, not entirely dead, even though everyone seems to claim that it is. I'm glad that you're here. So sorry for folks who are less aware of what LLM evaluation is. Donathan, can you just give us the working definition from your side of the fence?
00:43:46 · Alex Wilhelm, Jonathan Siddarth
[00:43:46] S1: I'm very impressed. Also, we're both in our home offices today. And I think this just goes to show that remote work, not entirely dead, even though everyone seems to claim that it is. I'm glad that you're here. So sorry for folks who are less aware of what LLM evaluation is. Donathan, can you just give us the working definition from your side of the fence? [00:44:04] S0: Yeah. So, it's really important to benchmark LLMs to figure out where they are good at, where are gaps in the models so that you can generate data that could help the LLMs get better at those specific tasks. One big challenge, Alex, that I see today in the overall evaluation and benchmarking market is that a lot of the evaluations are somewhat academic and somewhat synthetic and don't connect to real world applications or real world use. Ideally, you want AGI to progress in a way where the models get better at useful tasks. So when we evaluate models, the three dimensions to look at are complexity. Are you evaluating them on really hard tasks? Real world use. Are you evaluating them on something that a human would actually care about? And third is diversity. You want a wide breadth of test cases that you're evaluating the models on. It's such an exciting space. And I think of evaluation as step zero of data generation. That's why we do both evaluation and data generation.