00:34:06 · Aditya Agarwal
[00:34:06] S1: One of the trend that's often being discussed today, and especially with Satya Nadella's post sometime back, right, that a lot of orgs care a lot about keeping their intelligence within their own walls and judgment and a bunch of other things. And they're talking about, at org level, sovereign AI of sorts, right? And especially with the latest controversy that you had with the Navier Stokes Millennium Prize, right? There's almost a bunch of the paranoia seems to be getting fanned that our labs using our judgment, our data to get better. Right? How do you... Since you've been an insider, what's your POV on this?
00:34:48 · Srinivas Narayanan, Aditya Agarwal
[00:34:48] S0: Generally speaking, the labs make certain promises to the customers, especially the enterprise customers, and they keep those promises. Right? In the case of OpenAI, for example, you know, if you were a business and if you're using our APIs or if you're using chat from enterprise, you don't train on those data. So I think there is sometimes unfounded fear that, oh, my my model... The... My data from my models are getting trained. In practice, that doesn't happen. But I also understand the concern from companies because, for example, Anthropic changed their terms from, you know, they used to have zero data retention and then say, no. No. No. We need to keep your data for thirty days to understand our security risks. And, you know, that was kinda like changing the terms. Right? So these sorts of things create more concern for companies, and they're kinda worried that we're at the mercy of these labs and what happens in the future if they start changing the term. So I think their concerns are real, which, you know, will get addressed by a couple of ways. I think the labs will try to have policies that they can give better guarantees to these companies on what they will and will not do. [00:35:51] S1: Mhmm.
00:35:51 · Aditya Agarwal, Srinivas Narayanan
[00:35:51] S1: Mhmm. [00:35:52] S0: At the same time, I think companies will also want to create systems where they can take more control and they can more... Make make more things more customizable, which is I think why I think open models are also more exciting for them. So I think for every company, you know, you have to sort of use your judgment for what matters for any given use case. I think there's also kinda like a technical lens to this, which is do you really need customization for certain kinds of tasks? I think that answer is really, really depends. Right? And so I think customization can look interesting on paper, but in many cases, you might get away by just prompting a model or doing few short examples. So I would always advise on do the simplest thing first. Don't don't try to create something just because you feel like you need to own something. Right? Like, do simple things. And if it don't work, then you can start thinking about customization. [00:36:41] S1: There's a bunch of, you know, the doomsday process... Prophecies around where the world is going. Like, just a day before, we had a departure from Anthropic, which I think his tweet just went viral, right, that none of the labs seems to seem to be behaving in a responsible way with this civilizational technology. What is your point of view on this? And especially maybe this is this is informed by all the sci fi that we have read growing up and this entire arc of Doors Day, Cathartic, and kind of a hero's arc. So I'm just curious on your thought process around this.
00:37:14 · Srinivas Narayanan
[00:37:14] S0: So I think we have to take all the concerns seriously. I'm generally an optimist by nature. I believe that, you know, we will find ways to solve these problems. I think... But we have to be extremely clear on what we understand and what we don't understand. Sure. I think if you make extreme statements one way or the other, I'm not sure that, you know, they are realistic. Right? Like, I think, you know, just to say everything is gonna be doomed because it's certain is not right, and at the same time, acknowledging the risk is also not right. So I generally take a pretty pragmatic view to these things, which is what do we understand? What do we not understand? What are the risks? And how do we mitigate the risks? What I can say is that it is very clear that the risks are pretty, pretty real. Mhmm. Right? And we have to take them extremely seriously. The hugging face incidents was obviously a big wake up moment for everybody. I don't think... While we all know that reward hacking was going to happen, I think the extent to it which happened, the way these models, you know, started collaborating with each other in very unprecedented ways, the way they try to kinda cover up the fact that they found an answer that they didn't want the human to find out. Yeah. And the extent to which they went, I think, was pretty surprising for many people. You know, the fact that, you know, if it had sort of stopped one or two steps, it would have been interesting, but it sort of tried to do a lot more than that. Right? And so I think that is definitely a wake up call. So which is very clear that the models today are not entirely aligned with how we want them to work. So... Which means we have to do the research. We have to continue to invest in that. But, you know, I am generally optimistic we'll find these answers, but it's not like I know the the recipe yet. I think we will figure it out. We also have to find ways to invest in, you know, what happens if things really go wrong, like, are the kill switches that we have? Can we pull the plug, etcetera? So I think we just need to think about these systems pretty holistically. And, yeah, I think the recent incidents have only heightened the importance of the fact that we have to do more work on security alignment, etcetera.
00:39:19 · Aditya Agarwal
[00:39:19] S1: There's almost a bunch of us have keeping the popular content around this, right? So whether it's the Pin Factory metaphor or whether it's Skynet, right? I'm just wondering, yes, there's custodianship that's required, and there's like a larger discussion and thought process required. But how internally, when you think of some of these things, right, has happened. Now... And any any of us who are building the stuff, how should we think about AI safety at all?
00:39:48 · Srinivas Narayanan
[00:39:48] S0: I mean, it's a pretty open ended question. I would say there are a lot of different dimensions to safety. There's an element of cybersecurity, which is probably the most immediate, most, I would say, in your face thing that's happening right now. So I think general advice for, I would say, for everybody is that think about cybersecurity pretty heavily and make sure, you know, you're doing all the things to do to protect your systems. That's sort of at one level. Yeah. There's a question on, you know, do models go... How do you actually make the models more aligned with how do we want them to behave? That's a pretty open ended research question. You know, it goes out... Goes back to how do we train them? What rewards do they... Do they do that? Do they get penalized if they sort of act in different ways? Mhmm. I think the labs are all thinking about this, but I think we're also uncovering that we don't know enough about what's happening. So I think there's a lot of research stuff to be to be invested in. I think, again, there's a question on, you know, if you're building on top of a model versus whether you are customizing the models. The thing I would also say is, you know, reward hacking is a pretty important thing. So if you're ever training a model, think about that extremely carefully. I think the models are very, very ingenious about, you know, if they're finding any small loophole, if it is there, you'll exploit it. Right? So you have to be super thoughtful about designing machine learning training systems that think about hacking. Yeah. I think the cybersecurity stuff is pretty concerning.