Lenny's Podcast: Product | Career | Growth
00:14:30 · Lenny Rachitsky
[00:14:30] S1: Is there anything else there? Any examples come to mind?
00:14:32 · Roman Ugarte
[00:14:32] S0: Yeah, I think another thing we really tried to pay attention to in the early onboardings was just how much users wanted to see. And I think it is something a bit shocking or just different about Grok Bot when you first start using it versus some of the other products that you mentioned, where a lot of the internal mechanics of how Grok Bot works are not shown to the user. And the reason for that is we think as these models get smarter, the same way that your teammate, you wouldn't ask for second by second updates of exactly all the buttons they're pressing and websites they're going to. I think it's too much to ask your bots to do that too. And I think it's honestly just overwhelming and can create more harm than good. And so we moved completely in the other direction of you send a message, you tell your bot to do something, it just starts doing it. It sends you progressive updates as it sees fit, and you just see that typing indicator in the little green active circle kind of Slack-like that it's active, it's doing work, it'll get back to you soon. But you don't see the internal mechanics, you don't see the tool calls, you don't see exactly every little click it's making on its own computer. And we really wanted to take a strong stance that users did not need to see all of those mechanics, and so that's where we started. And we did get some feedback that's like, I would love to see my bot's to-do list. I would love to see roughly how it's prioritizing tasks and what it's doing. And that's great feedback, but it was useful to hear that nobody wanted the long stream of just text streaming out and chain of though sequences. So that also gave us more confirmation that that was the right direction.
00:16:17 · Lenny Rachitsky
[00:16:17] S1: The fact that you did two to 300 onboarding calls with a small team, I know the team grew over time, but just that is a huge time commitment and you could argue a distraction from the building, clearly not a distraction, clearly a core part of the success. Do you feel like that's the volume people need to do to figure out what actually needs to happen?
00:16:41 · Roman Ugarte
[00:16:41] S0: Well, one thing to emphasize is the early access group is not necessarily just people that are highly influential taste makers.You're in this category, Lenny, and we certainly wanted to get a lot of your feedback just from being very close to many other products on the market and just being a power user of these things. But we also wanted to get early access to more unconventional profiles that we as a company had never really interacted with. So one example is there's a coffee shop owner that was a friend of a friend through the company who had heard about Grok Bot, and one day somebody on the core team had shown them a demo of the test flight and got very excited. And this coffee shop owner ended up being not only amazingly an amazing power user of Grok Bot, but also a rich source of feedback for us. We have a very lively thread with many, many bugs that get identified or feature requests, and it's a completely different use case of running a small business. And so for example, if the Shopify integration was a bit flaky or if it wasn't writing copy for products in a particular way, we would get really rich feedback on that, which is pretty different from the type of feedback we'd get from dog feeding this internally. And so I think it was an important exercise for us to check our blind spots and say, A, this is going to be a very general product that is not just a thing that developers use. In fact, it is likely that this is most powerful for non-developers. We need to understand that group much better. And then B, is we absolutely live in this kind of Silicon Valley AI bubble, which I think is a useful place to be to kind of push the frontier and push the future of how these products are evolving. But we need to actively get out of that because I think a product like this has the chance of really being the way that the mainstream user and the mainstream business customer can interact with AI in a way that's useful.
00:18:41 · Lenny Rachitsky, Roman Ugarte
[00:18:41] S1: Let's go back to the timelines real quick just to kind of understand that. So it was a month from first line of code to internal beta, and then what happened after that? [00:18:50] S0: It was about three weeks from internal beta to public launch, and then I think we're about three weeks out from public launch as of recording this. [00:19:00] S1: Wow. Okay. So a month of building the first thing, three weeks only of iterating, and then it's been only three weeks since launch. It feels like it's changed the world from my vantage point. So wow, okay. What most changed in those, I don't know, in those three weeks of internal beta, let's say?
00:19:21 · Roman Ugarte
[00:19:21] S0: We unshipped a lot. I wish I could have shown you what things looked like maybe two weeks out from launch where I think we'd realized that the core team, we had a lot of experimental features that we wanted to get internal feedback on, which was useful. We also were kind of putting pseudo developery visibility tools into Grok Bot instead of having a separate observability pane for those things. So for example, we actually did expose sometimes a lot of the internal thinking of the models and the specific memories it would store and all of these things, which was useful to debug issues. And if you're building the product, you didn't want to go somewhere else to maybe pull that context, but we had to really aggressively trim what we think the user absolutely needs to see in the surface versus what they don't. I think there's even more room there to run, which is something the team's focused on right now is how can we just ruthlessly simplify this product and abstract away anything the user doesn't need to actively be thinking about? So that was one big push was unship, a lot of jank. And then I think the second big push those few weeks was making it just work. And I think a lot of what people want from AI is not this thing with lots of dropdown menus and bells and whistles, but just a thing that you describe a task, a task that's meaningful to you, and it goes and it does it and it comes back with complete work or it comes back with something for you to react to and then steer it in its kind of next cycle. And so in order to actually deliver on that promise, it's actually not a lot of feature product roadmap style stuff. It's like hill climbing five really important problems in the backend that many users don't directly experience, but you totally feel when your bot is going off and doing something and can't click the right button, or your bot is off and doing something and can't log into a website and it just completely stalls your ability to make progress on that task. And so those few weeks we collected a really rich set of what are tasks that people actually are giving bot, how can we quantify these things and how can we see week over week that across those categories of tasks that we're hill climbing on very important dimensions to making that just work behind the scenes?