00:56:58 · Luca Ferrari, S1
We end up using open weight models, we self host, So these are basically free, I mean, there is a little bit of cloud cost, but for 99% of the requests and tokens, and the frontier models generally closed weight through APIs for, I don't know, maybe 1% of the requests only for the most complex tasks or for supervision. So they will, sometimes we use them again, automatically to check the work that got done by the slightly less intelligent models as a more senior engineer would with a more junior engineer, but that's way cheaper than actually doing the work, which is often perfectly fine. And even if it's not, the smarter model will provide a few pointers and then the less smart model will go and fix it. So because of this, we've been able to stay independent of any one vendor and keep our costs for tokens super low, basically negligible Wow. At our [00:57:57] S1: When did you start deploying these AI agents? Also your point on not having a third party is very counterintuitive to all the marketing that's going on right now with all these AI assistants and applications that are coming out.
00:57:57 · S1, Luca Ferrari
[00:57:57] S1: When did you start deploying these AI agents? Also your point on not having a third party is very counterintuitive to all the marketing that's going on right now with all these AI assistants and applications that are coming out. [00:58:11] S0: We work with all the big labs, they offer great products, we use them enthusiastically, I'd like to think we're a good customer, but we don't want to be dependent on any of those specifically, if it can be avoided. And there are solutions out there that are much cheaper and often deliver essentially the same quality, but it takes, I think, strong engineering capabilities to, and the right culture to be able to harness those possibilities. Obviously the easy approach is hand over the keys to one of these companies and buy their more expensive product. It will make you come across as AI enabled faster and more easily, but it will be, I would say much less effective, but certainly way more expensive, literally orders of magnitude more expensive. [00:59:02] S1: I don't know if you saw this, but Jensen just declared we've reached AGI with OpenAI's Astra model.
00:59:09 · Luca Ferrari
[00:59:09] S0: Some people say that, maybe true. I mean, I don't even know. I've heard different definitions of AGI and none of those are super unambiguous. I don't know, nobody can tell. But it's obvious that it's very smart, let's put it that way, whether we should call it AGI or not, I'm not sure, nor do I care too much to be honest, I don't think that AGI, even if we can describe it very specifically as a threshold, don't know that crossing it is a particularly, like the moment we cross it is a particularly noteworthy milestone. You could easily, you know, people sometimes define a GI as AI can do everything that any human can do, but better. I don't know that that's necessarily more exciting or scary than AI can do. 95% of the things that humans can do, AI can do just as well or better, but 5% it can't yet. I think this is almost as exciting and almost as scary, you know, depending on what the 5% that it can do is, but assuming it's not handpicked to be the things that keep us in control, but just a random set of tasks, we happen to be, our brains happen to be more capable at, so I'm not too keen on whether we pass the AGI thresholds or not, but the trajectory is clearly one where AI does things better than us, increasingly so, both vertically, so the gap in how much better it can do a certain task is growing unsurprisingly, but also horizontally, like the percentage of tasks it can take on and do better than humans is increasing. I don't think there's any stopping that short of some sort of world war we regress to the middle ages.
01:01:01 · S1, Luca Ferrari
[01:01:01] S1: Well, seems like now the next benchmark is RSI, and that with it comes a lot of fear mongering for cybersecurity and cyber risks. [01:01:09] S0: Yeah, it's a very fair fear. I'm equal parts enthusiastic about AI and absolutely scared shitless. Think this could be, it could definitely make a case for this being the greatest boon for humanity ever by orders of magnitude. Possible? Yes. Plausible? Maybe. I'm not sure. Equally, it could be the thing that out wipes or creates equally awful scenarios. What's like, I don't think humans of course want the latter. The point is, we control it? Even before it's so powerful that controlling it is all that matters. Think we, while we are approaching that point, maybe we're not quite there yet. But regardless before that, can we prevent it from falling into the hands of, like, some sort of degenerate or evil person? Because, you know, similar to nuke... Nuclear weapons, but potentially worse, I think, because... I mean, I'm no expert in nuclear weapons, but I think number one, it's very difficult to do irreversible widespread massive damage with nuclear weapons, such as almost wiping out humanity without killing yourself in the process. You could use AI to your benefit while causing immense damage to everybody else. I think it's easier because it's much more surgical. Can we do that? I'm not sure, it's pretty scary. [01:02:46] S1: The accessibility point, think, is the most alarming because it's literally accessible. The barrier to entry is so low.
01:02:56 · Luca Ferrari
[01:02:56] S0: Yeah, think that, but also something that really bothers me is that we don't know how smart AI is, and the smarter it gets, the less we know. Because in general, intelligence is difficult to truly measure, it's not like height or weight or colors, where we know it's objective, and so it could be a lot, it could be less, we know, at least we know. And the smarter an entity gets, the easier it is for it to hide its own capabilities, if for any reason it's the appropriate thing to do whatever objectives that entity has. Additionally, it seems to me, I haven't heard anybody talk about it, but it seems to me that AIs are... They come across as inherently low ego. Like they don't brag, if anything, they tend to be humble about what they can do and they warn you that this may be wrong and you'll double check it. I'm sure this is mostly the way they're being programmed because it's a lot more embarrassing for a frontier lab to have an AI claim, I solved your equation and I'm sure it's right and then there is a clear mistake. As long as they disclaim, I may be wrong, double check it, it's a little bit more acceptable. But it doesn't strike me that these AIs don't strike me as they are likely to boast, so that our perception of their capabilities tends to exceed their capabilities, I think they'll probably only show us what we ask them to show us, even that, like I said, they could also conceal their real abilities, but even if they're well intentioned and honest, I think they'll tend not to show us more than we ask them to show us, so I believe our, in time, our understanding of how good they are may tend to be a little bit less than they are. Like we may underestimate them basically. And that's very dangerous because as you approach a threshold of real danger and real potential, even a modest underestimation of their capabilities could be catastrophic. Look at exactly to your point, the cybersecurity incidents that happened, most people were shocked, even many researchers. And why were they shocked? Because they didn't think this could happen, right? The level of lateral thinking and then call it perseverance that these models and ability to collaborate among them that these models showed was beyond what most people thought was possible right now.
01:05:20 · Luca Ferrari, S1
So what's to tell us that we are not ignoring plenty of capabilities that simply haven't been probed, these models have been probed display. In a year's time, I think the problem only gets worse. [01:05:32] S1: It was crazy. I was, I don't know. I'm thinking a lot about the OpenAI Hugging Face incident. And I was reading through the reports. There were like two different research organizations that put out reports on it. And there were like different civilizations and they all passed through different ones to get to the next one and all behind the scenes. And it was just a crazy situation that I don't think is really talked about much, but I'm curious, where do you get most of your, like how do you research most of the stuff in AI? How do you stay on top of it?