Invest with AI
00:35:26 · Robert Yang
[00:35:26] S0: Yeah, yes. So that was a pretty difficult time for us um because it went hyper-viral, right? So we had lots of people come in and test it, um, but then a lot of them churned because uh I think the model is like um not quite there at the point, right? The models was not quite there. Our harness can obviously could have been improved uh more, but I think fundamentally it's just like we were maybe like one generation early, right? Um, so a lot of people come in and try to do very sophisticated things and just realize it's it's not it's not really doing that. Um and uh it was a difficult time because for like two, three months we had on paper, it's very viral. Um, but when we were looking at the metrics, everything was going down, right? Like people were churning and the retention was too low, right? So so the revenue just kept going down. Uh in fact, we attempted a fundraise that didn't work during that time because like people were like, oh, this is very viral, let's talk. And but then when they see the actual numbers, like things were going down. So uh in around October last year, um, we just sat together and we're like, okay, let's let's just focus on making this really, really good. Right. Um, so the team we we um we we essentially cut every other project, just really focused on Excel. And one of the biggest things I think we did is to build a really, really strong internal benchmark, right, for real work. Uh so this is uh Richard, uh who's our chief product officer now, he led that work where he essentially collected uh he essentially built these tasks that are in distribution, which means it's like similar to what our users would do, right? For example, it's like uh leverage buyout, right? This is for private equity, uh discounted cash flow, this is for like like public company investment. Um and and the important thing is is it needs to be hard enough for the agent, but also the you need to be able to trust the score, right? Uh so building a benchmark is very hard because sometimes you you have a task and you have some prompt. Uh, there are many ways to do the task, right? So there is no one correct way, and and then you cannot really like evaluate the output. So the task needs to be designed in a way that's both like natural, as in this is something a user might really do, but also precise enough that there is one correct answer, right?
00:38:02 · Robert Yang
So that was pretty hard. Now, once we have built these tasks, then it become much easier to kind of iterate on the agent harness, right, which is like everything that goes around the language model. And we did a lot of like um like detailed optimization for the Excel use case, right? Making sure that like essentially um it can really understand the the Excel API, right? Uh, and uh how does the agent manage its context very efficiently so it can it can do the task because like if you think about just naively, how do you take a big big Excel like uh table, right? Um just you have lots of call, you can have lots of columns and lots of rows. How do you put that all in the the context of the language model? And if you're not careful, you're gonna just pollute their context right away, right? So there had to be a lot of engineering to be done. Uh that's very detailed, very specific to spreadsheet. Um and once we started to do these things, then uh it's just started to work, right? We saw all the metrics were going up, like retention is up, like users started to grow, and then we started to get into the enterprise, had figured out how to do like enterprise grade security and things like that, right? We have never done that before. Um, and then it's then it's uh it just started to become better, right? So so um I think turning point was probably November last year, right?
00:39:33 · Brett Caughran
[00:39:33] S1: Um yeah and and uh you walk me through you know, there's there's really been no in sort of the core hedge fund community that that I sort of lived in, there hasn't really been a transcendent product yet. Um I think most hedge funds I I I talked to really mostly use Claude. Um I had a moment sort of independently in September, you know, September 22nd when Opus 5.5 came out, and you know, the the model started to pass my modeling benchmarks in a way that was pretty astounding. The day I tweeted that, David from from Shortcut reached out, he's like, Have you tried shortcut? I'm like, Yeah, man, I tried it last July and it was terrible. And he's like, try it again. Um and um and so from that sort of moment from September to uh September last year to September this year, um you sort of found this ICP in in the hedge fund in the hedge fund level. Can you tell us why that happened?
00:40:37 · Robert Yang
[00:40:37] S0: Yeah, yeah, absolutely. So I think um I I think a lot about this, right? Uh we I don't think we had like great foresight, right? That this is a good um um this is kind of good customer for us, right? Um we were we were still, I would say in the beginning, still didn't have a clear understanding of how to position ourselves, right? We're like we're the best Excel agents, right? But but then mass companies with massive distributions like Microsoft, cloud, how how do you differentiate against them, right? Well, our answer is always like, well, our product is just better, right? Like our agent is better, more accurate, blah, blah, blah. And it turns out for a lot of people, it doesn't matter as much, right? Or like they're not as sensitive to this, right? Because let's say I want to like build a personal budget tracker, right? Um, and I'm not very rich, right? So I'm not going out and buying yachts, right? So it's like it's there's not that much, right? It's like, okay, like try to put your like limit your grocery, right? Um things like that. Um, so so cloud uh for Excel or like um Microsoft Copilot can do these things just fine, right? What the people who are able to most like feel that difference between our product and um and our competitors' product, right? Like cloud for excel are people who are doing really cutting-edge demanding work, right? And it turns out a lot of these are like the hedge fund analysts, right? It's not exclusive to those, but as you know, the hedge fund analysts, the buy-side analysts, these are in some way the pinnacle, right? Of like the work they do is extremely like the sophisticated. It has to be because if it's not good, then you just don't wait, don't make money, right?
00:42:29 · Brett Caughran, Robert Yang
[00:42:29] S1: Um certainly the tolerance for air is uh incredibly low. Yeah, exactly. One error and the one lazy error in the wrong cell of a financial model can be really destructive. [00:42:40] S0: Yeah, exactly, right? You can get fired, right? If you just put give some slop to your uh PM. Uh so so these so these people they really felt the difference. And uh and uh um and I think in retrospect, uh this is a really great match between our company gene and and our customers, right? Our company gene has always been like very research and tech driven, right? That's why I think when a lot of people just are kind of just like a lot of our competitors, maybe not Anthropic, but you know, like there are other competitors, they don't really have benchmarks for a long time, right? It's just kind of vibed, right? Just feel the vibe of it. And we had benchmark which allow our agents to be better, and we can compare it against Claude for Excel and know for a fact it's better, right? Like we can compare it and say, oh, like the token efficiency is higher, the performance is higher, the success rate is higher, right? So that's I think that focus on accuracy, because we have been doing agents, we have been benchmarking agents since 2023, um, just resonated with uh like our investor customers who are also very much about accuracy. And precision and performance.
00:42:40 · Robert Yang, Brett Caughran
[00:42:40] S0: Yeah, exactly, right? You can get fired, right? If you just put give some slop to your uh PM. Uh so so these so these people they really felt the difference. And uh and uh um and I think in retrospect, uh this is a really great match between our company gene and and our customers, right? Our company gene has always been like very research and tech driven, right? That's why I think when a lot of people just are kind of just like a lot of our competitors, maybe not Anthropic, but you know, like there are other competitors, they don't really have benchmarks for a long time, right? It's just kind of vibed, right? Just feel the vibe of it. And we had benchmark which allow our agents to be better, and we can compare it against Claude for Excel and know for a fact it's better, right? Like we can compare it and say, oh, like the token efficiency is higher, the performance is higher, the success rate is higher, right? So that's I think that focus on accuracy, because we have been doing agents, we have been benchmarking agents since 2023, um, just resonated with uh like our investor customers who are also very much about accuracy. And precision and performance. [00:44:02] S1: Yeah, that makes sense. Um and uh you know, building product in um in the world of AI sort of feels like a constant building of sand castles and the better lesson sort of evolving, like as has clock directly has gotten better. How do you think about you know Excel agent as a as a platform?