00:16:25 · Adam Miller, Anastasis Germanidis
[00:16:25] S0: On that last point, you are talking a lot about world models these days. Silicon Valley is also talking a lot about that and companies are raising huge seed rounds for them. Could you explain from your perspective what a world model actually is and how Runway are going to be utilising them and just sort of juxtapose them against the sort of previous generations of models that you've built. [00:16:46] S2: So, world models are systems that build representations of environment and help to predict what happens as a result of a specific action. So the biggest change from video models, so video model, you type in a prompt, you wait for a bit, you get back a full video. With a world model is interactive and that means you're continuously providing control, so you're continuously moving around in a space or you're continuously interacting with a character or you're continuously, controlling a robot arm so the ability to try different actions and see the effect and of of of the... Of your the... Your actions, that's what makes... That's the main differentiation from, from video models, which are a bit more, like, offline use of just you type in a prompt, you get a packet output, and you're not continuously influencing. [00:17:41] S0: And what would be the practical application of that for Runway? Like, how would somebody eventually engage with, a world model?
00:17:41 · Adam Miller, Anastasis Germanidis
[00:17:41] S0: And what would be the practical application of that for Runway? Like, how would somebody eventually engage with, a world model? [00:17:49] S2: So even if you take the use case of, filmmaking and storytelling and making videos, imagine the experience of essentially being on set right, and being able to continuously kind of direct the camera continuously direct the the actors in in the scene, that's that's something that's a very different experience from the experience of typing a prompt and getting back an output. It's it's something that allows for much greater degree of control. So even if we take the use cases that of, like, video generation, it's a much better experience if you actually can walk on set and just be able to direct the scene. But the the the applications are far broader than that. So if we if we take... We're very excited about how we're seeing our models already being used in physical AI and robotics. So collecting data for robotics is very, very expensive, very time consuming, very, unsafe in a little... In in in in some ways, in some domains. Being able to generate data to train robots is a big application of our models that we're very excited about. Being able to simulate how robots perform without having to test them in the real world is another application that we're seeing really promising results as well.
00:19:18 · Adam Miller
[00:19:18] S0: So I'm sitting here in Chiperdal right next to our office, and Sachin and I just came back from New York. We're on a long documentary tour there, and now we're just heads down building our Stealth startup. And the thing I can't really get out of my mind is this idea of trust. Can customers really trust that what we're building is gonna help them create great content and marketing that helps reach the buyers that they need to reach. That's the thing I think about all the time. And that's one version of trust, but there's so many different types of trust you need to build with customers. Does your product work? Is it gonna help them get value? Is it gonna work in six months? Do they trust your business model? Do they trust your team? There's so many different things. It takes a really long time to build that trust. And that's data security and compliance. Because with a company like Vanta, they can help you automate your compliance in weeks instead of months. We all know those acronyms, SOC two, HIPAA, the complex acronyms that enterprise buyers will throw at you when a deal starts to get serious. Vanta helps you take care of all of that. Using Vanta is really a no brainer solution. Those aren't things that I wanna focus on. I want Vanta to do that for me. Because, you know, the worst story that you'll hear about in startups is when you've found great customers, you've built a great product, you're talking to these big enterprise customers, and you're, you know, you're going over a deal, you're negotiating, you're getting to the final dotting the p's and the i's in the contract, and then they realize that you don't have all the compliance that they need and the deal falls through. No startup founder wants that to happen. We want companies like Vanta that can take care of that for us so then we can just focus on our product and on our customers. So if you're interested in this, head to vanta.com/sachinandadam and you'll get a thousand dollars off.
00:21:04 · Anastasis Germanidis, Sachin, Adam Miller
[00:21:04] S2: Yeah. So this is, essentially, this is an example of, our world model being used as a simulator for robotics. So what you're gonna see is, our robotics model that's basically, rolling out a policy, which is basically performing an action, performing a task inside the imagination of, GWM one, which is our world model. So [00:21:36] S1: what we're looking at is instead of a robot going into this desk room or whatever and and actually just collecting training data, you're collecting training data through simulating it through the model [00:21:46] S2: exactly and you're essentially simulating how a robotics model will perform and you can see you can simulate things like you know like folding clothes which is traditionally very difficult things that to simulate with traditional methods [00:22:01] S1: and what are the inputs of these simulations actually look like? [00:22:05] S2: So the inputs is essentially you're providing you're controlling this robot arm in a similar way that you could control the robot arm in the real world so the input you're providing is the movement of how how that robot arm moves in space. [00:22:21] S0: Yeah. Are we looking at AI generated video or are looking at a robot that has been trained on it? [00:22:27] S2: So this is all generated footage. So our... This is our world model GWM one that's generating two different views of a scene and a robot performing this task. [00:22:40] S0: And with your recent models, you're able to generate a hundred milliseconds per frame, believe. How is that even possible?
00:22:40 · Adam Miller, Anastasis Germanidis, Sachin
[00:22:40] S0: And with your recent models, you're able to generate a hundred milliseconds per frame, believe. How is that even possible? [00:22:48] S2: It's a difficult research and engineering challenge to essentially start from a base video model that's traditionally very slow to getting to distilling it so that it can actually, you can run with inference at real time, frame rate. So, it's it's necessary in order to provide interactive control, and it's something we've been really focused over the past past year of how to distill our base video model to be real time. [00:23:14] S1: If I think of some of the kind of companies we profiled, physical intelligence, we went down to SciQuantum in the South Bay a couple of days ago. I I imagine there's been so many use cases of technology like this for companies like that. In terms of becoming like physical intelligence, how do you think they could use technology like this?
00:23:14 · Sachin, Anastasis Germanidis, Adam Miller
[00:23:14] S1: If I think of some of the kind of companies we profiled, physical intelligence, we went down to SciQuantum in the South Bay a couple of days ago. I I imagine there's been so many use cases of technology like this for companies like that. In terms of becoming like physical intelligence, how do you think they could use technology like this? [00:23:32] S2: Yeah. So a company like physical intelligence and other robotics labs and, you know, we're we're in collaboration and conversations with a lot of them. They spend a lot of time running evals, basically understanding how the robotics models that they've trained can perform, in in the real world, and it's very difficult in order to differentiate between, you know, you've trained a few models, you wanna see which models are good at what tasks. That's extremely time consuming to do. Very often it can break the robots, like it's... And so being able to minimize the number of real world evals that you have to do, by doing the evals in simulation and scaling them, you know, as much as you want is is very, very useful and kind of significantly accelerate kind of the the research, kind of progress of them. [00:24:22] S0: And is there a big risk there? Because this data would need to be pretty perfect. Right? Because if you're using synthetic data to train other models then that would just degrade quality over time right unless these sort of generations are near perfect?