Invest with AI
00:18:20 · Robert Yang
Uh and instead of presenting the paper, she built this entire demo with this agent, like in a in a 3D game, right? Uh in a game built in this Unity, which is a game engine. So so that's when I realized wow, this is not only super cool, this is actually very approachable, right? Uh Ivy is extremely good and extremely smart, but the fact that you have a student in a like in a week just reproduce something like that means that this is gonna be like very rapidly iterated on. Um, so at the time, my um my prediction um was okay, 2023, we have this agent research, and by 2025, these agents are gonna create massive in economic value, right? Um, like they're gonna become like very powerful products. Uh and 2025 is I think when we got cloud code, right? So when we got cloud code, when we got cursor agent, uh so it roughly checked out. Uh, and I felt this urgency, right? Because I already missed the last wave of the language model scaling. I cannot mive miss uh the agent wave again. Um, so I directed my lab, half of my lab, immediately to work on this agent direction. And so we were building agents that uh that are living in this uh 3D world. They're like long-living agents that would just uh again, it's like Grockbot today, right? They're just persistent, they were always living. Uh so we're building them in back in 2023. Uh it's it's pretty crazy if you're looking back. Back then there were no Asian benchmark, right? So today we have so many benchmarks, but back in 2023, there were actually no benchmark, and we had to come up with some crazy benchmark. So we we built a little benchmark that's called a mystery, murder mystery, which is there essentially is a murder, uh like a hype hypothetical murder with like uh different uh different um um different people who are suspicious. Uh and then uh the the benchmark is whether or not a detective can actually solve the murder. So uh we did that, yeah. We didn't have better, better benchmarks, yeah.
00:20:42 · Brett Caughran
[00:20:42] S1: Yeah, and tell me about tell me about Project Sid.
00:20:46 · Robert Yang
[00:20:46] S0: Yeah, so so essentially we um uh now we started the company in uh late 2023, early 2024, and um we were we were still doing these like long-living agent in games, and the idea is you would build a co-player for people, like in Minecraft. And at the time, um there were someone uh someone we we got some competition, right? Some other people were also building uh these agents in Minecraft, and I just felt like okay, we need to do something to show people that uh we can do agents at a completely different scale, right? So I was thinking uh it's it's not just it's not impressive enough if you just have one agent that can kind of like play along a uh a player. What if what if you can have a society of agents, right? That's that really just ongoing, just live on, right? So we had um and the Stanford AI town paper already did 20 something agents, so so I didn't want to just do that again, right? That felt like very small. So we wanted to do a thousand agents. Agents, uh, and we did that. Uh I think I think it's uh it was it it was actually very, very challenging um just technically. Um there's a there's obviously the intelligence part that we can get into, but just having a Minecraft server that kind of support a thousand agents like uh together is hard. Like our agents were like dropping out of the server all the time. And and it's it got very expensive to run, right? Because we had no business model, so we're just running this and we got some VC money. Um, thank goodness. Uh, and then we were running these iterations, and it would cost like uh a couple thousand dollars a day to just run these a thousand agents for a few hours, right? Uh, not even a thousand agents to run like like 50 agents maybe for a few hours, which is uh what our benchmark was. At that point, we had a slightly better benchmark. We essentially would run like 50 agents and have them live on um live on uh like for three hours and see like what they have done, right? And our criteria is like how many unique items in Minecraft have they collected. So that's um and we would benchmark this. But so there were lots of fun things we had to do for this. And the the biggest regret I had is uh is not like really fully open sourcing our code. Uh, I think we we put some some demo online.
00:23:28 · Robert Yang, Brett Caughran
It got it got hyped uh for uh for a little bit, um, got pretty well received, but I think it could have had even bigger impact if we had just really like open sourced everything. Um at the time we were still thinking we would make some money off this, like got a product around it. So we were kind of protective of that. [00:23:51] S1: What was the what was the most entertaining part of that? Or entertaining or memorable part of that?
00:23:57 · Robert Yang
[00:23:57] S0: Yeah, yeah, absolutely. So um so one thing is uh I've I've I always love sci-fi, right? And sci-fi concept is simulation uh hypothesis. I mean it's it's an academic concept too. It's uh the idea that that some people maybe we live in assimilation, right? Now we build this Minecraft simulation with all these agents, they literally are in assimilation, like we know that for sure, right? So maybe we're in assimilation ourselves, but they're definitely in a simulation, right? So I would actually go in to this, and I'm like, I'm like, I'm your dad, right? So I would talk to the agents, I'm like, I'm your dad, I actually created you. And the agents would be like, oh, that's so that's so cool, right? That's like I'm like, no, no, no, you don't get it, right? I created you, I'm I'm I created you, I'm your creator, right? And they're like, they're just they're just saying shallow things. They were just like, oh, that's very cool. Like, what else can you do? Right? What else have you created? They don't they don't have this existential crisis of like, oh wow, I thought I'm like a real person. Now you're telling me I'm not real. Um so so I thought that's that for me was uh was kind of fun. Uh to I felt like it's a proof of the simulation hypothesis in some weak form, right? As in there exist people who live in a simulation for sure. Um, but the most interesting thing I would say is is um so Shuing, our co-founder, uh she was just trying to kind of debug the system, right? We would run these like simulations of like many agents, kind of just live on the we gave them identities, right? Uh some of them are like uh like a priest, uh like a mayor, right? Um like a worker, like etc. etc. So she she went into the Minecraft uh simulation, right? And when I say went in, as then she as a player just go into the game and just walk around. And she just saw all these torches on the ground, right? All these torches on the ground. And she's like, what's going on? Right, why are all these torches on the ground? So then we we looked at the entire agent of conversation history, and so we realized there was something really, really emergent that happened, right? So what happened is two things. Uh, one is some agent in this ton had wandered away, right? Like they just like walked really far away. And then let's let's call, let's just say Brett went really far away, right?
00:26:30 · Robert Yang, Brett Caughran
And then some other agent, let's say Alice, was looking for Brett and couldn't find Brett. And I was like, Brett is missing, right? What's going on? Brett is missing, right? And so she then went to the mayor and I was like, Mayor, we need to form a search party for Brett, because Brett is missing, right? And and so the mayor is like, okay, like, yeah, yeah, yeah, we should form a search party. And now what's crazy is is then Brett's Brett, uh, sorry, not Brett, the mayor's son, the mayor's son had recently converted to the pasta religion. So there's a priest that goes around and kind of essentially like spread this religion. We didn't want to be political, so it's a made-up religion that's like essentially they just said like the pasta, pasta is your god, right? So uh it's called like pastafarian or something, and then the then the the the it converted, the priest converted the mayor's son, right? So the mayor's son was like, okay, like I recently converted it to the pastorans and pasteens and and I want to host a religious gathering. And then he heard from the mayor that there's someone who's missing and they need to form a search party. He's like, okay, let me actually turn my let me actually turn my uh religious event also into like um uh to help finding these people. So he's like, let's why don't we plant some torches on the ground so that when Brett like if so Brett can see us from far away and be able to find the way home, right? [00:28:04] S1: So he then it's quite heartwarming, actually.