Lenny's Podcast: Product | Career | Growth
00:16:41 · Roman Ugarte
[00:16:41] S0: Well, one thing to emphasize is the early access group is not necessarily just people that are highly influential taste makers.You're in this category, Lenny, and we certainly wanted to get a lot of your feedback just from being very close to many other products on the market and just being a power user of these things. But we also wanted to get early access to more unconventional profiles that we as a company had never really interacted with. So one example is there's a coffee shop owner that was a friend of a friend through the company who had heard about Grok Bot, and one day somebody on the core team had shown them a demo of the test flight and got very excited. And this coffee shop owner ended up being not only amazingly an amazing power user of Grok Bot, but also a rich source of feedback for us. We have a very lively thread with many, many bugs that get identified or feature requests, and it's a completely different use case of running a small business. And so for example, if the Shopify integration was a bit flaky or if it wasn't writing copy for products in a particular way, we would get really rich feedback on that, which is pretty different from the type of feedback we'd get from dog feeding this internally. And so I think it was an important exercise for us to check our blind spots and say, A, this is going to be a very general product that is not just a thing that developers use. In fact, it is likely that this is most powerful for non-developers. We need to understand that group much better. And then B, is we absolutely live in this kind of Silicon Valley AI bubble, which I think is a useful place to be to kind of push the frontier and push the future of how these products are evolving. But we need to actively get out of that because I think a product like this has the chance of really being the way that the mainstream user and the mainstream business customer can interact with AI in a way that's useful.
00:18:41 · Lenny Rachitsky, Roman Ugarte
[00:18:41] S1: Let's go back to the timelines real quick just to kind of understand that. So it was a month from first line of code to internal beta, and then what happened after that? [00:18:50] S0: It was about three weeks from internal beta to public launch, and then I think we're about three weeks out from public launch as of recording this. [00:19:00] S1: Wow. Okay. So a month of building the first thing, three weeks only of iterating, and then it's been only three weeks since launch. It feels like it's changed the world from my vantage point. So wow, okay. What most changed in those, I don't know, in those three weeks of internal beta, let's say?
00:19:21 · Roman Ugarte
[00:19:21] S0: We unshipped a lot. I wish I could have shown you what things looked like maybe two weeks out from launch where I think we'd realized that the core team, we had a lot of experimental features that we wanted to get internal feedback on, which was useful. We also were kind of putting pseudo developery visibility tools into Grok Bot instead of having a separate observability pane for those things. So for example, we actually did expose sometimes a lot of the internal thinking of the models and the specific memories it would store and all of these things, which was useful to debug issues. And if you're building the product, you didn't want to go somewhere else to maybe pull that context, but we had to really aggressively trim what we think the user absolutely needs to see in the surface versus what they don't. I think there's even more room there to run, which is something the team's focused on right now is how can we just ruthlessly simplify this product and abstract away anything the user doesn't need to actively be thinking about? So that was one big push was unship, a lot of jank. And then I think the second big push those few weeks was making it just work. And I think a lot of what people want from AI is not this thing with lots of dropdown menus and bells and whistles, but just a thing that you describe a task, a task that's meaningful to you, and it goes and it does it and it comes back with complete work or it comes back with something for you to react to and then steer it in its kind of next cycle. And so in order to actually deliver on that promise, it's actually not a lot of feature product roadmap style stuff. It's like hill climbing five really important problems in the backend that many users don't directly experience, but you totally feel when your bot is going off and doing something and can't click the right button, or your bot is off and doing something and can't log into a website and it just completely stalls your ability to make progress on that task. And so those few weeks we collected a really rich set of what are tasks that people actually are giving bot, how can we quantify these things and how can we see week over week that across those categories of tasks that we're hill climbing on very important dimensions to making that just work behind the scenes?
00:21:48 · Lenny Rachitsky
[00:21:48] S1: Is there an example, one of those hills you were climbing that was kind of a technical breakthrough or technical challenge you overcame that really helped it just work?
00:21:56 · Roman Ugarte
[00:21:56] S0: Yeah. One example was from rolling out Grok Bot, one group inside of the company that was actually incredibly bot pilled, so to speak, was our go-to-market team was sales. And there are a bunch of tools that sales uses that do not have well-supported MCPs or APIs. And I think that's a lot of what made bot so before and after powerful for this group was these were things that they just could not give another AI tool reliably. We would get stuck at some part in the process. And then Bot kind of felt like they had an assistant or kind of felt like they onboarded someone to their personal team, they gave it a laptop and it could just run. And so there were a bunch of small things and probably a list of 10 or 20 of them of places where for whatever reason, the mouse would just not have fine enough control to click on exactly that part of the Salesforce dashboard or something like that, that we would have to take back to the core team working on really the infrastructure to say, "Here's a very concrete case of where the agent not having this visibility into the browser or this visibility into the pixels on the screen is making it impossible for this task to be done." And that was just a lot more tangible than seeing a number on a dashboard slowly creep up. It was kind of like new chunks of work getting unlocked and you would immediately feel the feedback where you would ship an improvement that was kind of behind the scenes, kind of infrastructure-y. And then the next day you would just get this outpour of love and appreciation from the sales team that now this workflow that was failing the last seven days finally works. And it's just a constant exercise of finding those next tasks to unlock and then solving them.
00:23:45 · Lenny Rachitsky
[00:23:45] S1: So computer use basically improvements seems like a big unlock. I heard also I had Adam Ward on the podcast who's head of recruiting, head of hiring, basically head of talent. I heard that his team was one of the top users of Grok Bot.