Invest with AI
00:14:50 · Robert Yang, Brett Caughran
So like when I wanted to when we first started the company, I just wanted to be able to just build the human without without really answering why this is useful, right? So just wanted to like put in like emotions, long-term memory, right? Just motivation, all of this good stuff into an agent and have them just live alongside human um without really answering the business model. Um, so and that's kind of what we did. [00:15:18] S1: And what what was there a final moment or something you saw when you were at MIT that really was the the catalyst to lead? Because that's a pretty cushy life lifestyle. Yeah. Um what uh what was sort of the final deciding insight?
00:15:35 · Robert Yang
[00:15:35] S0: Yeah, yeah, absolutely. So I was having a lot of um, so I was having a lot of existential crisis, but at the same time, I felt like I missed the last wave, right? So the major wave of NAI between about 2017 and 2023 was essentially scaling of language models, right? So uh 2017 was a transformer paper, and then uh and then in 20, I think 2019, um 2020 was like around that time was the scaling papers, right? So that's that was a main paradigm, and I felt like I just missed out on that entire paradigm, and I just didn't want to like um be a follower. But in 2023, early 2023, so the language models started to be available to people to experiment with, and there were a couple papers from academia that were really influential and just essentially wrapped these language models and built agents, right? Now we're very familiar with agents, but in 2023, early 2023, they were still very new. One paper is called the Voyager paper. They essentially build an agent that um that goes in Minecraft and would like like just like play the game. And um, but it it's it's not really just like playing the game for the sake of like achieving a specific goal. It was actually just exploring, exploring the game, which I thought is really, really uh cool. And another thing that's similar is um there was a paper from Stanford, the Stanford AI Town paper by June Park, who is actually our first, our first investor, an NGO investor. So so June Park had this paper that had like a 20-something agents, also kind of like this language model wrapped with memory, right? Um, that would run around in a virtual tongue. That's like a Pokemon tongue. And they would run around and they would like host birthday parties and they would like try to cook and like buy stuff, right? None of them are real, right? None of these things are real. There is no real food and they cannot consume food. But it was uh it was really uh like a shocking demonstration of the future, right? Where you you don't just have chatbots, you have agents that live and just live their life, right? Um so I read that paper and I I was pretty shocked. But what was even more, I think, shocking for me is we had an intern, uh she she was great. Um her name's uh Ivy Zhao, and uh she was essentially asked to present this at a journal club.
00:18:20 · Robert Yang
Uh and instead of presenting the paper, she built this entire demo with this agent, like in a in a 3D game, right? Uh in a game built in this Unity, which is a game engine. So so that's when I realized wow, this is not only super cool, this is actually very approachable, right? Uh Ivy is extremely good and extremely smart, but the fact that you have a student in a like in a week just reproduce something like that means that this is gonna be like very rapidly iterated on. Um, so at the time, my um my prediction um was okay, 2023, we have this agent research, and by 2025, these agents are gonna create massive in economic value, right? Um, like they're gonna become like very powerful products. Uh and 2025 is I think when we got cloud code, right? So when we got cloud code, when we got cursor agent, uh so it roughly checked out. Uh, and I felt this urgency, right? Because I already missed the last wave of the language model scaling. I cannot mive miss uh the agent wave again. Um, so I directed my lab, half of my lab, immediately to work on this agent direction. And so we were building agents that uh that are living in this uh 3D world. They're like long-living agents that would just uh again, it's like Grockbot today, right? They're just persistent, they were always living. Uh so we're building them in back in 2023. Uh it's it's pretty crazy if you're looking back. Back then there were no Asian benchmark, right? So today we have so many benchmarks, but back in 2023, there were actually no benchmark, and we had to come up with some crazy benchmark. So we we built a little benchmark that's called a mystery, murder mystery, which is there essentially is a murder, uh like a hype hypothetical murder with like uh different uh different um um different people who are suspicious. Uh and then uh the the benchmark is whether or not a detective can actually solve the murder. So uh we did that, yeah. We didn't have better, better benchmarks, yeah.
00:20:42 · Brett Caughran
[00:20:42] S1: Yeah, and tell me about tell me about Project Sid.
00:20:46 · Robert Yang
[00:20:46] S0: Yeah, so so essentially we um uh now we started the company in uh late 2023, early 2024, and um we were we were still doing these like long-living agent in games, and the idea is you would build a co-player for people, like in Minecraft. And at the time, um there were someone uh someone we we got some competition, right? Some other people were also building uh these agents in Minecraft, and I just felt like okay, we need to do something to show people that uh we can do agents at a completely different scale, right? So I was thinking uh it's it's not just it's not impressive enough if you just have one agent that can kind of like play along a uh a player. What if what if you can have a society of agents, right? That's that really just ongoing, just live on, right? So we had um and the Stanford AI town paper already did 20 something agents, so so I didn't want to just do that again, right? That felt like very small. So we wanted to do a thousand agents. Agents, uh, and we did that. Uh I think I think it's uh it was it it was actually very, very challenging um just technically. Um there's a there's obviously the intelligence part that we can get into, but just having a Minecraft server that kind of support a thousand agents like uh together is hard. Like our agents were like dropping out of the server all the time. And and it's it got very expensive to run, right? Because we had no business model, so we're just running this and we got some VC money. Um, thank goodness. Uh, and then we were running these iterations, and it would cost like uh a couple thousand dollars a day to just run these a thousand agents for a few hours, right? Uh, not even a thousand agents to run like like 50 agents maybe for a few hours, which is uh what our benchmark was. At that point, we had a slightly better benchmark. We essentially would run like 50 agents and have them live on um live on uh like for three hours and see like what they have done, right? And our criteria is like how many unique items in Minecraft have they collected. So that's um and we would benchmark this. But so there were lots of fun things we had to do for this. And the the biggest regret I had is uh is not like really fully open sourcing our code. Uh, I think we we put some some demo online.
00:23:28 · Robert Yang, Brett Caughran
It got it got hyped uh for uh for a little bit, um, got pretty well received, but I think it could have had even bigger impact if we had just really like open sourced everything. Um at the time we were still thinking we would make some money off this, like got a product around it. So we were kind of protective of that. [00:23:51] S1: What was the what was the most entertaining part of that? Or entertaining or memorable part of that?