No Priors: Artificial Intelligence | Technology | Startups
00:12:52 · Mark Zuckerberg, Elad Gil
[00:12:52] S0: love my aggressive reaction. [00:12:53] S4: I'm like Danny Glover, you know, and Lisa Waffen. I'm almost at retirement. But I think one of the things that was always lacking was this integrative nature across the different layers of biology, and the developmental biologists would work on their own, the molecular biologists would be doing different experiments. And so that's what I was curious about. Typically, there's a reductionist view of biology and there's a systems view, and those people didn't really work together deeply. And so one of the exciting things about what you're doing actually is how you're bridging that. And so that was kind of the basis for the question as well.
00:13:20 · Alex Rives
[00:13:20] S2: Yeah. And if I could add something there. You know, it's I think that, you know, we're in the age of this kind of information theory in biology. And so, you know, there there are levels of of complexity and hierarchy in biology, and kind of each level is is made up of and, you know, constituted by the lower levels. And so as you want to have that kind of more complete description and you want to have systems that can really generalize and begin to actually answer, you know, experimental questions digitally that you could ask in the lab, you need to have kind of the right basis for modeling at every level. And so I think what's really unique about what we can do is to, as Priscilla and Mark were saying, really build information at each of these different layers, collect them, collect kind of those connection points, but then also really kind of do it at the scale that will reveal that underlying information architecture. And that's gonna be really critical to actually be able to build digital representations that can answer new experimental questions.
00:14:22 · Sarah Guo
[00:14:22] S3: One of the things that inspires me most about this effort is really what Priscilla said, which is like, well, there's so much we actually understand about biology and what if we could? Which I think is actually very different from lots of other incredibly interesting and useful AI problems we attack where we're like trying to replicate human behavior. I'm like, a lot of that data's, you know, on the internet or captured. And without pretending to understand all human behavior, can predict a lot of it. I thought one of the most interesting things in your release was actually, you know, the mechanistic interpretability stuff you alluded to, which is, can we actually extract new knowledge from, you know, what the model believes is happening? Right? Can you talk a little bit about that?
00:15:03 · Alex Rives
[00:15:03] S2: Yeah, I'm really excited about that. So I think, you know, in mechanistic interpretability, kind of traditionally it's been applied to large language models with the goal of understanding, you know, kind of what is the representation space of a large language model? How does it compute things, and does that really connect to, you know, what we understand about our intuitive understanding of the world? And so there's, I think, this really rich toolkit that has been developed to start to be able to ask those questions. So kind of what does that mean for biology? One of the classes of models that we train are these protein language models. So they're really, you know, trained on the codes of proteins. And so anything they learn about biology is is kind of emergent. And we've seen that they can learn things like biological structure and biological function, and that's just kind of emergent from this, you know, token prediction training task. So, you know, as we think about, like, mechanistic interpretability in those models, you know, we're really seeing the unknown because the models have been trained on billions of protein sequences. They've been trained on, you know, both known and unknown biology. And yet they're developing these representations that start to kind of capture things that we can really see correspond to that reductive picture of biology that's been built up over the centuries. So kind of you can start to connect the dots between proteins where we kind of really don't know anything about them with proteins where we do know something because there's that kind of underlying structure grammar that's linking them in the representation space of the model.
00:16:37 · Sarah Guo, Alex Rives, Mark Zuckerberg
[00:16:37] S3: And at the extreme, it could be, you know, we're going to understand systems in the body that we didn't before or the mechanism of action for a new treatment because we can ask the model, right, interrogate that representation. [00:16:47] S2: That's right. The hope is that you kind of really learn the underlying basis for how it's making the predictions. And so you open up the black box and you can actually understand kind of the biology that the model is representing. [00:16:58] S3: So asking for a friend, you know, you guys all believe in venture backed companies as a way to have impact on the world. Was it like collecting data on zebrafish or the span of the data, or the wet lab work, or just the scale? Like what makes this a better fit for this big nonprofit, you know, ecosystem effort versus a venture backed company? [00:17:24] S0: Well, I think we just want to give tools to the whole scientific community. [00:17:27] S3: And I mean, like, so I I think
00:17:29 · Mark Zuckerberg
[00:17:29] S0: in order to have the biggest impact, I mean, part of it is just we're I mean, it's not actually clear that we couldn't run it as a business if we wanted to. I just think that we'll have a bigger impact by getting this in more scientists' hands quicker by doing it as open source projects instead. So, yeah, I mean, I think that that's kind of the approach. But I don't know. It's an interesting question. I'm not sure that I mean, obviously, you were doing it as a for profit company, a bunch of the modeling before, then you run into certain issues. You have to raise a large amount of money in order to build the compute clusters. I think in a lot of ways, the data is actually even more of a constraint. Because if you look at the scale of these models compared to language models, they're smaller, but they're smaller because the amount of data is less. In order to get the data, it's not just like there's some factory somewhere that you can pay to produce the data. You actually need to invent new, novel, scientific approaches to be able to do the, know, for example, the type of cellular engineering we're doing in New York or the types of devices in Chicago, which is why, you know, when we're talking about this concept of frontier biology and frontier AI, the frontier biology is you need to do real science to advance different biological methods in order to be able to observe the things that create the data that go into the model. Mhmm. So it's not just like an off the shelf thing that you can create. Now that's a pretty big effort. I don't know that there are, like, that many things like that that are done as as biotechs. I think it's just the scale of the ambition of what we're doing, the time horizon over which we're committed to doing it. I think part of the theory is, like, if you're building tools that are this complicated, you kinda wanna have a ten to fifteen year time horizon on on building out these efforts. And then the scale of capital required I mean, I guess there's no rule that said that you couldn't do it as an incredibly well funded startup, but I think that this just made more sense. And then it also is is simplifying strategically to not have to think about you're gonna make money with the different things. I mean, we just we wanna get the models in people's hands. We release them as open source.