Lenny's Podcast: Product | Career | Growth
00:07:24 · Roman Ugarte
Are people actually going to switch from other internal tools, other external tools to use Grok Bot as their primary agent surface? And in that first week after that all hands, I mean, I can't even tell you just the outpour of love for Grok Bot from people that maybe you wouldn't expect or from groups of the company that maybe you wouldn't expect, people who were daily driving ChatGPT or a chat interface, switching all of their day-to-day agentic tasks over to Grok Bot as their primary surface for doing work. And so the internal reception was really extraordinary. Can tell some funny stories from that week or two period. And then once we saw, I think the internal reception, we immediately switched into let's get this ready for the world. There's a lot of work to do to scale this out to millions of users. And then that led to the GA launch that we had a few weeks ago. It was not obvious at all. I think you're completely right that that was one of those original decisions that at the time we had a lot of discussions about, and I'm very glad with where we landed. And I think to your point, it being a brand new product that you control every pixel of the experience and you have this consistent vision about where knowledge work is going and it's all contained in this new thing, I think has contributed a lot to the success. But there were a lot of discussions about Cursor, for example, and some of our coding products, people use it for non-coding tasks all the time. And these coding agents are really excellent at some of these things, but you run into small paper cuts. Sometimes the product itself is kind of intimidating to non-technical users. There's a brand association with these things. And so I think we evaluated that path and I think we saw what maybe some of our competitors had been doing of this is all just one surface, you add new tabs for each new form factor and it feels a little cluttered. And I think for users, they can feel that, that this was not a single consistent vision of the way that work should work. And instead it's three different visions that all kind of share a screen and you can hop between, but it is kind of a shipping your org chart style thing that I think users are reacting negatively to. And so we decided let's just start completely from scratch. Let's see where we can get from there.
00:10:36 · Roman Ugarte, Lenny Rachitsky
There might be some really amazing opportunities to bring people from other surfaces into this more bot native experience, but it's really important for people to just have an amazingly simple and amazingly powerful experience. [00:10:55] S1: That is a really valuable lesson for people to take away here, just that that might be the solution instead of adding in complicated and existing AI product. So interestingly, Codex went the other direction and it's like a different path than a different product, but it's interestingly, they're like, now we're going to make it one thing. So there's many ways to make it work and it also feels like the path you take there will kind of lead you, but maybe we'll look back and be like, that was not maybe the best idea. Something you mentioned that you did that is also really unique is this onboarding of early users. I heard you and your team onboarded two to 300 people manually, including me. Talk about why you thought that was necessary and what you learned from that experience and just how long that period was of this kind of manual onboarding.
00:11:41 · Roman Ugarte, Lenny Rachitsky
[00:11:41] S0: I mean, you just learned so much and the first few onboardings were pretty painful. I'm glad you got a good one, Lenny, but there were some that were kind of rough and we learned a lot. And I think it was important for the core team to be in the room for those and to just sit on a call for 20 minutes when the computer isn't spinning up or when someone's in onboarding and they're just incredibly confused so that immediately after you're like, that can never happen again. We need to solve this tomorrow because tomorrow I'm onboarding this person and it needs to go better. And so there was about a two-week period where we were in that mode and onboarded a couple hundred people. And not only did we learn a lot about the product, I think we didn't really know ... I think sometimes with these products there's some group think of ways to use them. And I think internally, because people inside of SpaceXAI were just constantly sharing tips and tricks for how to use Grok Bot, some patterns were starting to emerge that we thought would be useful to the world, but we weren't really sure and we definitely didn't want to bias the world. [00:12:46] S1: Is there an example of that?
00:12:48 · Roman Ugarte
[00:12:48] S0: So when we rolled out Grok Bot internally, there was about a week or two where the common pattern of the way people would interact with the product was you would have five to 10 bots and each bot you would give a different scope, a different domain, and it was kind of shorthand for different lanes of work. And then at around the end of week two, we started to see these messages internally in Slack of people promoting one of their bots who is a bit of a standout performer, and it was like their kind of primary personal assistant promoting that to their chief of staff. And then they would actually mostly talk to their chief of staff and the chief of staff would fan out all of these tasks to the other bots and would kind of manage the team. And there are some funny screenshots of people actually telling the bot they're promoting that they're promoted and the bot is asking if they get a raise and is their token budget higher, all of these things. And we kind of took note of that and I think more of the companies started to slightly shift in that direction, but it was not the majority of the company. People use this product in very different ways. And so in some of the onboarding sessions and just from the early access program in general, we really did not want to lead the witness and say, create a chief of staff bot, here's exactly the way that that chief of staff should manage all of the other bots and see if early access users would get there themselves. And we actually did see that many of them did. And so then we had a bit more of an opinionated take in the product of this feels like a pattern that's working, this feels like a pattern that we should slightly encourage, but it shouldn't be a one-way door. And there were a few other examples of internal thesis that we really wanted to make sure would bear out in actual external usage without us imposing that in the product.
00:14:30 · Lenny Rachitsky
[00:14:30] S1: Is there anything else there? Any examples come to mind?
00:14:32 · Roman Ugarte
[00:14:32] S0: Yeah, I think another thing we really tried to pay attention to in the early onboardings was just how much users wanted to see. And I think it is something a bit shocking or just different about Grok Bot when you first start using it versus some of the other products that you mentioned, where a lot of the internal mechanics of how Grok Bot works are not shown to the user. And the reason for that is we think as these models get smarter, the same way that your teammate, you wouldn't ask for second by second updates of exactly all the buttons they're pressing and websites they're going to. I think it's too much to ask your bots to do that too. And I think it's honestly just overwhelming and can create more harm than good. And so we moved completely in the other direction of you send a message, you tell your bot to do something, it just starts doing it. It sends you progressive updates as it sees fit, and you just see that typing indicator in the little green active circle kind of Slack-like that it's active, it's doing work, it'll get back to you soon. But you don't see the internal mechanics, you don't see the tool calls, you don't see exactly every little click it's making on its own computer. And we really wanted to take a strong stance that users did not need to see all of those mechanics, and so that's where we started. And we did get some feedback that's like, I would love to see my bot's to-do list. I would love to see roughly how it's prioritizing tasks and what it's doing. And that's great feedback, but it was useful to hear that nobody wanted the long stream of just text streaming out and chain of though sequences. So that also gave us more confirmation that that was the right direction.