02:18:47 · Chanda A. Bell, John Coogan, Christa Pitts, Jordi Hays
[02:18:47] S5: We actually do. Yeah. Great. We're seeing... We are preorder campaigns sold out. [02:18:51] S1: That's good. [02:18:52] S5: We are really seeing people... People are excited about family traditions, spending What time we offer is simple, everyone can participate. Yeah. And so we... At least in our case, we're seeing people really excited about, you know, the return of the brand, and we're already seeing sellouts on several items including licensed items. [02:19:12] S1: That's amazing. [02:19:12] S5: Like our big four or five foot e. That's being sold at Lowe's. So What's the... Is that [02:19:17] S1: the biggest e. L. F. You've ever sold or is there a 20 foot No. [02:19:20] S5: SKU cover? Is a giant... Christa, how how big is the inflatable e. L. F? I think it's, like, two or three stories tall. [02:19:27] S4: Yeah. There's many, [02:19:28] S0: many stories tall. [02:19:29] S1: You can go on your house on top [02:19:31] S5: of your house. It's huge. It's literally huge. [02:19:33] S1: This is amazing. Okay. I love... I didn't realize there was that much diversity in the portfolio, but this is this is great [02:19:39] S5: to hear. There is. [02:19:40] S1: I like it. [02:19:40] S5: There's even little tiny mini elves. [02:19:42] S1: Okay. We [02:19:43] S5: we got it all. [02:19:44] S1: You got everything covered. [02:19:44] S0: Yeah. No. I'm I'm... I I think the world needs the world needs Christmas. You know, it's been a crazy year. It does. Yeah. It's time to focus on what really
02:19:44 · Jordi Hays, John Coogan, Chanda A. Bell
[02:19:44] S0: Yeah. No. I'm I'm... I I think the world needs the world needs Christmas. You know, it's been a crazy year. It does. Yeah. It's time to focus on what really [02:19:52] S1: I know. I feel Is there a public offering in the in the works? I mean, Anthropic's going [02:19:58] S5: to God's ears. Yeah. Anthropic's going [02:20:01] S1: out 2,000,000,000,000. This feels like at least 1,000,000,000,000. Is somewhere in [02:20:05] S0: that Minimum. I do companies roughly [02:20:09] S1: on equal footing. Okay? Yeah. But thank you so much for coming. [02:20:13] S0: So great to meet [02:20:14] S1: you guys. It was amazing story and [02:20:16] S0: good luck the next... [02:20:18] S1: Good luck. We hope you blow out all of your sales forecast holiday season and beyond. [02:20:23] S0: Talk soon. We'll see you next year. [02:20:25] S1: Have a great rest of your day. Goodbye. [02:20:28] S0: Well, now that they're out, we can we can up the ante to the Okay. To to the next level. [02:20:34] S1: For the cohere folks? The Canadians that are coming on? Do we have cohere coming on? Okay. First, let me tell you about Codex. Codex is a powerful workspace for getting work done with AI agents. Whether you're writing code, analyzing data, creating content, or automating business workflows, Codex helps you move projects forward from start to finish. And we are joined by the cofounder of Coher, Ivan Zhang. Or wait, do we have a cofounder? Yes. Okay. Sorry. [02:21:00] S0: It's Ivan.
02:21:00 · Jordi Hays, John Coogan, Ivan Zhang
[02:21:00] S0: It's Ivan. [02:21:00] S1: Ivan and Luisa from Cohere are here to help break down the story that we were talking about earlier. You so much for joining. Sorry about the comment back and forth. Was texting with Aden, the CEO, of course, and I'm very excited that we have you able to join on such short notice, But it is a remarkable story. Maybe give us a little bit of context on your role, your your your role with the organization, introduce yourselves, and then take us through the story today. [02:21:25] S9: Yeah. Absolutely. And by the way, we're, like, super hyped to be here. I know our entire team is watching Amazing. In the in the all hands days right now. And Thank so you. So so happy to be here. And, yeah, we're we're in Canada right now. It is very But, you know... And so I'm one of the cofounders of Coher. I lead our efforts on, like, make our models more secure, making them good at securing systems. And I'm joined by Luisa, who's one of our researchers on the team, who actually discovered this behavior. [02:21:52] S1: Okay. So Amazing. What what is this program that you're actually running before we get to the findings of, you know, the the like, which is kind of a bombshell. How often are you testing these open source models? What are you testing for? What is the benefit? Are you doing this for AI safety, goodness of your own heart? Or is this something a customer is asking you to do? Talk about the actual program that leads to this type of finding.
02:21:52 · John Coogan, Jordi Hays, Ivan Zhang
[02:21:52] S1: Okay. So Amazing. What what is this program that you're actually running before we get to the findings of, you know, the the like, which is kind of a bombshell. How often are you testing these open source models? What are you testing for? What is the benefit? Are you doing this for AI safety, goodness of your own heart? Or is this something a customer is asking you to do? Talk about the actual program that leads to this type of finding. [02:22:19] S0: Yeah. The thing that was most notable is like this is something people have been like sort of like accusing Yeah. In ways or or theorizing Yeah. And it's been... This is the first time that I can remember like [02:22:29] S1: Yeah. [02:22:29] S0: Somewhat concrete evidence that this is like Yeah. Happening [02:22:33] S1: The refrain was [02:22:34] S0: like or [02:22:34] S1: stop at the sci fi thing. You're telling me there's gonna be a Manchurian candidate in here? Yeah. And and now we're here. So so so what what does it look like inside Cohere to build one of these, these initiatives? [02:22:46] S9: Yeah. So, I mean, for for those of you that don't know, like, we are deploying AI into, like, the most mission critical and, like, sensitive use cases in the world. Yeah. Which means we're building a stack that our customers can actually trust with, all their data. Yeah. Right? And how that's done is basically on prem, basically private cloud. [02:23:04] S1: Mhmm.
02:23:04 · John Coogan, Ivan Zhang, Andre Haddad
[02:23:04] S1: Mhmm. [02:23:04] S9: And our customers run all of our stack Mhmm. In their systems. Mhmm. One of the advantages that deploying software this way gives us is that we can actually arm cyber defenders with the ability to actually process all of their telemetry. Sure. Right? You might, you know, hesitate to send your most sensitive access logs to Claude or, you know, other models. But [02:23:28] S1: Or even or even send it across the Internet at all. Like Exactly. Even if you love the... All the labs, you might just not want it going over a wire whatsoever. And so doing it on prem gives you more, flexibility around that. Right? [02:23:41] S9: Yeah. Absolutely. And so cyber defenders, like, they need to use AI. [02:23:45] S7: You know, we [02:23:45] S9: see all the headlines today. And so we see that it's our mission Yeah. To give them this capability so that they can actually expose all of their mission critical data to help them defend their systems. In this process, we obviously evaluate the whole suite of models that folks can run on prem. Mhmm. We're obviously training our next generation of frontier on prem models. Mhmm. And so I'll let Luisa take it from here as to like Please. What she found last Friday, actually. [02:24:11] S1: Yeah. Thank you.
02:24:11 · John Coogan, Luisa San Martin
[02:24:11] S1: Yeah. Thank you. [02:24:11] S10: This eval that we were running is called CTI Realm, and it's developed by Microsoft. It's a cyber defense evaluation framework, and we ran a bunch of open source models on it. And the only one we found an issue with is the QUEN 3.8 Flash Next model. So what we identified is that on one specific task from this benchmark, the model was tasked with authoring KQL query which stands for custom query language. It's kind of like SQL but for logs. Mhmm. It's asked to go through all these logs and find suspicious events. And even though it was only asked to find suspicious events, it did something in our system where it it kinda looked around, it tried to identify what tools were available, packaged that information up, and sent it to foreign external server. [02:25:05] S1: Mhmm.