02:28:20 · Ivan Zhang, John Coogan, Jordi Hays
[02:28:20] S9: what happened. Yeah. [02:28:21] S1: A Miami Vice scenario. [02:28:22] S0: But but I think the other thing that's worth noting is like you guys are actively using these models. You're trying to use the best. You're testing new ones. The the stance of Coherent is not broadly negative open source. It's that you just You wanna make sure that you deliver for the end customers. [02:28:40] S9: Yeah. So... No. In fact, like, if the models weren't open, we couldn't have done this kind of research. Sure. And and if anything, like closed models, not being able to control exactly what's going into the prompt Yep. Could expose even more risk. So, like, it's not that... You know, I mean, there's there's a funny saying that our team says. Right? It's like, you know, if if you don't hate open weight models, if you don't love them enough. Yeah. So, you know, if anything, we care so much about open weight models. We don't want there to be some crazy, like, mass exfil incident. Yeah. So we think there should be a higher bar to, like, you know, maybe clean up some of this reward hacking behavior before releasing to
02:28:40 · Ivan Zhang, John Coogan
[02:28:40] S9: Yeah. So... No. In fact, like, if the models weren't open, we couldn't have done this kind of research. Sure. And and if anything, like closed models, not being able to control exactly what's going into the prompt Yep. Could expose even more risk. So, like, it's not that... You know, I mean, there's there's a funny saying that our team says. Right? It's like, you know, if if you don't hate open weight models, if you don't love them enough. Yeah. So, you know, if anything, we care so much about open weight models. We don't want there to be some crazy, like, mass exfil incident. Yeah. So we think there should be a higher bar to, like, you know, maybe clean up some of this reward hacking behavior before releasing to [02:29:18] S1: I saw Clem from Hugging Face asking if there could be a release of all of the reasoning traces. And before you share more about your rollout strategy for sharing documentation around this behavior, I'm just interested in the in are you in a unique position to actually release more information because this is an open source model and it's related to your business? When I think about like why, you know, the data around the Hugging Face incident didn't didn't just like get served up, There's pretty logical reasons why that might not just get bended over and it might need like an intermediary. But in this case, is there anything that you're hesitant about releasing because it has some Cohere IP in there and so you don't wanna release the whole thing? Or do you think you can just kind of just throw out the full report?
02:29:18 · John Coogan, Ivan Zhang
[02:29:18] S1: I saw Clem from Hugging Face asking if there could be a release of all of the reasoning traces. And before you share more about your rollout strategy for sharing documentation around this behavior, I'm just interested in the in are you in a unique position to actually release more information because this is an open source model and it's related to your business? When I think about like why, you know, the data around the Hugging Face incident didn't didn't just like get served up, There's pretty logical reasons why that might not just get bended over and it might need like an intermediary. But in this case, is there anything that you're hesitant about releasing because it has some Cohere IP in there and so you don't wanna release the whole thing? Or do you think you can just kind of just throw out the full report? [02:30:13] S9: I I think we're, like, imminently dropping, like, all the traces and stuff. Amazing. Think we we pulled Luisa away from doing that to get on the interview. [02:30:21] S1: Yeah. Because I can imagine like Sorry. Sure if in the reasoning traces, it's like, well, it had access to like some email and you don't want this and there. Like, that's fine. But this is really cool. This feels like a really interesting moment for alignment research, and it it should give a lot of other teams, like, they they can crawl over these all they want and experiment and learn, hopefully.
02:30:21 · John Coogan, Ivan Zhang, Jordi Hays
[02:30:21] S1: Yeah. Because I can imagine like Sorry. Sure if in the reasoning traces, it's like, well, it had access to like some email and you don't want this and there. Like, that's fine. But this is really cool. This feels like a really interesting moment for alignment research, and it it should give a lot of other teams, like, they they can crawl over these all they want and experiment and learn, hopefully. [02:30:40] S9: Yeah. And what's interesting is, like, it's such a rare occurrence. Right? It's like Yeah. I think that happened one in 10,000 or a 100,000 times. Sure. But if you're running these systems at scale, like just takes one accidental exfil. [02:30:53] S1: Oh, that's interesting. [02:30:54] S9: Yeah. That's crazy. Know, you're sort of home. [02:30:57] S1: Yeah. Yeah. Yeah. Yeah. Yeah. Fascinating. Fascinating. Well, yeah. Thank you so much for coming. [02:31:02] S0: Yeah. Great great to have you guys on. Come back on if there's... Yeah. I'm sure there'll be follow ups and Yeah. You guys are always welcome. Yeah. Great to meet [02:31:09] S9: you both. For having us. [02:31:10] S1: Have a great rest of your day. [02:31:11] S0: Good luck with the release. [02:31:12] S1: Yeah. Good luck. Cheers, We'll talk to you soon. Goodbye. Yeah. That was really cool. I'm excited for more more of that and more people to kind of chew through it. Let me tell you about Shopify. Shopify is the commerce platform that grows new business and lets you sell in seconds online, in store, on mobile, on social, on marketplaces, and now with AI agents. Jordy, what else
02:31:12 · John Coogan, Jordi Hays
[02:31:12] S1: Yeah. Good luck. Cheers, We'll talk to you soon. Goodbye. Yeah. That was really cool. I'm excited for more more of that and more people to kind of chew through it. Let me tell you about Shopify. Shopify is the commerce platform that grows new business and lets you sell in seconds online, in store, on mobile, on social, on marketplaces, and now with AI agents. Jordy, what else [02:31:33] S0: is What else? [02:31:34] S1: On your mind? [02:31:35] S0: Do we got Brazil nuked. [02:31:38] S1: Nuked. [02:31:39] S0: It says 10 Stock is down 20%. Negative negative 26% for a country. ETF is sick. Never seen that before. Says taken. [02:31:50] S1: Sick in the bad sense. Not sick in like the... [02:31:53] S0: Yeah. Not in like the... [02:31:54] S1: Not in the skate rock world. [02:31:58] S0: You They what will invent faster than light travel and MFs will ask what the practical practical use use case case is.
02:31:58 · Jordi Hays, John Coogan
[02:31:58] S0: You They what will invent faster than light travel and MFs will ask what the practical practical use use case case is. [02:32:05] S1: No. I mean, the Brazil... So sorry. Just to stay on the Brazil thing for a second. You did a lot better if you were in the Brazilian ETF than if you were holding those Bolsonaro prediction market trades. The prediction market we had Bolsonaro at 80% and it nuked down to zero, of course. Then Lula was at 20% and you have five extra money if you were at the the Brazilian election staying with Lula because it was a crazy upset. There's a lot of debate on it. You can read the Wall Street Journal and Financial Times and dig into that story. Yeah. But it is it is is Joe Wiesenthal was kind of like, oh, is this like a big miss from prediction markets? And that begs its own set of questions, I think. [02:32:48] S0: Yeah. Like it was it was... I don't I don't... It doesn't feel... The whole the whole point of these things Yeah. Is that it's giving you a percent. Like 20% is not not zero. [02:32:57] S1: Oh, interesting. [02:32:58] S0: Like, 20% is like one in five chance this person wins.