Invest with AI
00:20:46 · Robert Yang
[00:20:46] S0: Yeah, so so essentially we um uh now we started the company in uh late 2023, early 2024, and um we were we were still doing these like long-living agent in games, and the idea is you would build a co-player for people, like in Minecraft. And at the time, um there were someone uh someone we we got some competition, right? Some other people were also building uh these agents in Minecraft, and I just felt like okay, we need to do something to show people that uh we can do agents at a completely different scale, right? So I was thinking uh it's it's not just it's not impressive enough if you just have one agent that can kind of like play along a uh a player. What if what if you can have a society of agents, right? That's that really just ongoing, just live on, right? So we had um and the Stanford AI town paper already did 20 something agents, so so I didn't want to just do that again, right? That felt like very small. So we wanted to do a thousand agents. Agents, uh, and we did that. Uh I think I think it's uh it was it it was actually very, very challenging um just technically. Um there's a there's obviously the intelligence part that we can get into, but just having a Minecraft server that kind of support a thousand agents like uh together is hard. Like our agents were like dropping out of the server all the time. And and it's it got very expensive to run, right? Because we had no business model, so we're just running this and we got some VC money. Um, thank goodness. Uh, and then we were running these iterations, and it would cost like uh a couple thousand dollars a day to just run these a thousand agents for a few hours, right? Uh, not even a thousand agents to run like like 50 agents maybe for a few hours, which is uh what our benchmark was. At that point, we had a slightly better benchmark. We essentially would run like 50 agents and have them live on um live on uh like for three hours and see like what they have done, right? And our criteria is like how many unique items in Minecraft have they collected. So that's um and we would benchmark this. But so there were lots of fun things we had to do for this. And the the biggest regret I had is uh is not like really fully open sourcing our code. Uh, I think we we put some some demo online.
00:23:28 · Robert Yang, Brett Caughran
It got it got hyped uh for uh for a little bit, um, got pretty well received, but I think it could have had even bigger impact if we had just really like open sourced everything. Um at the time we were still thinking we would make some money off this, like got a product around it. So we were kind of protective of that. [00:23:51] S1: What was the what was the most entertaining part of that? Or entertaining or memorable part of that?
00:23:57 · Robert Yang
[00:23:57] S0: Yeah, yeah, absolutely. So um so one thing is uh I've I've I always love sci-fi, right? And sci-fi concept is simulation uh hypothesis. I mean it's it's an academic concept too. It's uh the idea that that some people maybe we live in assimilation, right? Now we build this Minecraft simulation with all these agents, they literally are in assimilation, like we know that for sure, right? So maybe we're in assimilation ourselves, but they're definitely in a simulation, right? So I would actually go in to this, and I'm like, I'm like, I'm your dad, right? So I would talk to the agents, I'm like, I'm your dad, I actually created you. And the agents would be like, oh, that's so that's so cool, right? That's like I'm like, no, no, no, you don't get it, right? I created you, I'm I'm I created you, I'm your creator, right? And they're like, they're just they're just saying shallow things. They were just like, oh, that's very cool. Like, what else can you do? Right? What else have you created? They don't they don't have this existential crisis of like, oh wow, I thought I'm like a real person. Now you're telling me I'm not real. Um so so I thought that's that for me was uh was kind of fun. Uh to I felt like it's a proof of the simulation hypothesis in some weak form, right? As in there exist people who live in a simulation for sure. Um, but the most interesting thing I would say is is um so Shuing, our co-founder, uh she was just trying to kind of debug the system, right? We would run these like simulations of like many agents, kind of just live on the we gave them identities, right? Uh some of them are like uh like a priest, uh like a mayor, right? Um like a worker, like etc. etc. So she she went into the Minecraft uh simulation, right? And when I say went in, as then she as a player just go into the game and just walk around. And she just saw all these torches on the ground, right? All these torches on the ground. And she's like, what's going on? Right, why are all these torches on the ground? So then we we looked at the entire agent of conversation history, and so we realized there was something really, really emergent that happened, right? So what happened is two things. Uh, one is some agent in this ton had wandered away, right? Like they just like walked really far away. And then let's let's call, let's just say Brett went really far away, right?
00:26:30 · Robert Yang, Brett Caughran
And then some other agent, let's say Alice, was looking for Brett and couldn't find Brett. And I was like, Brett is missing, right? What's going on? Brett is missing, right? And so she then went to the mayor and I was like, Mayor, we need to form a search party for Brett, because Brett is missing, right? And and so the mayor is like, okay, like, yeah, yeah, yeah, we should form a search party. And now what's crazy is is then Brett's Brett, uh, sorry, not Brett, the mayor's son, the mayor's son had recently converted to the pasta religion. So there's a priest that goes around and kind of essentially like spread this religion. We didn't want to be political, so it's a made-up religion that's like essentially they just said like the pasta, pasta is your god, right? So uh it's called like pastafarian or something, and then the then the the the it converted, the priest converted the mayor's son, right? So the mayor's son was like, okay, like I recently converted it to the pastorans and pasteens and and I want to host a religious gathering. And then he heard from the mayor that there's someone who's missing and they need to form a search party. He's like, okay, let me actually turn my let me actually turn my uh religious event also into like um uh to help finding these people. So he's like, let's why don't we plant some torches on the ground so that when Brett like if so Brett can see us from far away and be able to find the way home, right? [00:28:04] S1: So he then it's quite heartwarming, actually.
00:28:04 · Brett Caughran, Robert Yang
[00:28:04] S1: So he then it's quite heartwarming, actually. [00:28:08] S0: Yeah, yeah, exactly. So that was and I was like, wow, didn't expect it to be so elaborate. Uh and it was we were completely shocked because we didn't we didn't design any of this other than like the fact that there's a priest, there's a mayor, etc. Right? So all of these things came out of just like the agents interacting with one another and behaving like humans. Um, that was for me, that was the coolest, coolest thing that an entire project. [00:28:39] S1: So how did you how did you go from building societies in Minecraft to building Excel agents for Wall Street? What walked me through that journey?
00:28:48 · Robert Yang
[00:28:48] S0: Yeah, yeah, yeah. It's uh um yes, uh absolutely. So so we did project Sid and then um it was very expensive to run, right? So we we're like, how do you turn this into a real product, right? So um we had we had people come in and they were given like one or two agents. It's not the same thing, right? You don't you don't really get to experience the whole like agent society, but even if you did, it probably would be a bad product, right? I think the same reason like moat book right didn't work out, right? Just people don't really want to see a bunch of agents talking to each other and just generating what is mostly slop, right? They're interesting things, but a lot of it is slop, or don't want to see it generated. Um so the product just essentially failed, right? Our like day one retention was 10%. So that was late 2024, and we were kind of in a crisis as a company. Um, and at the time, Anthropic did a computer use demo, and we immediately played with it. And I remember thinking we can do better than this. So we spent a month focused on building better computer use agents, and we actually came up with lots of good ideas. Uh, one of them was essentially we came up with the idea of a skill, right? And if the agent would do some task, and not only a skill, but agent would do some task and would like like introspect their behavior and then write down like a file, right, that will help it do uh do it better next time, right? Which is exactly like how like how right now you can like generate auto-generate skills. So we did that in back in late 2024, and we made the same mistake again, was thinking, oh, we'll turn this into a product right away. And we didn't publish any of this research. We we did use it to raise funding, right? So we raised like a series A of like maybe 30 30 something million dollars. Uh and uh and then now we started to uh just like try to productionize this. Um so we had two paths. One is to build a general computer use agent, another is to do something more focused, right? Like focused on Excel. The reason we focused on Excel is we realized even when we were doing the computer use uh research, we realized the current models are really, really bad at using Excel, but it's not because they don't understand it. It's because they actually don't have the right interface to work with Excel, right?