The a16z Show
00:15:43 · Josh Elman, David Pawlan
[00:15:43] S0: And did they just get to work, or did any of them actually work with you to understand this tension that you were building? Because I think what's really human about something like that is actually understanding what you mean by that and presenting you trade offs. Yep. Because there are times where I'll totally take the red eye to save both some time and money, but then I want, like, the hotel room on the other side so I can get a little nap in the morning Yes. Which then undercuts the money but still maximizes my time spent. And so I really think it's just like like getting to know the individual person and break down those tensions Mhmm. And present the trade offs and different options Yep. Is really important. Did any of them kinda get to what you were about? [00:16:21] S1: So they all did ask questions, and I responded those questions honestly. I tried to be a little ambiguous and and vague in my response because I didn't wanna lead them too much. But the most interesting part of this little experiment was I I put them all in a chat room. And I made it three rounds of building the itinerary, And after each round, the agents would present their itinerary and their explanation as to why they believe that was the best proposal. Because I wanted to see, did the agents actually get biased by the other agents and didn't actually then influence their plan? [00:16:49] S0: Mhmm. [00:16:50] S1: And Grokbot was actually the only one that, you know, very methodically took value from the other ones and implemented it into its own plan. [00:16:58] S0: K. And and I have
00:16:58 · Josh Elman, David Pawlan
[00:16:58] S0: K. And and I have [00:16:59] S1: one more question on the flowers. [00:17:00] S0: Yeah. Because... So you you were [00:17:01] S1: just ordering flowers for your... [00:17:02] S0: Yourself, I guess, just to, like, test their flower. My girlfriend. [00:17:04] S1: She got a got a nice delivery of of flowers. [00:17:07] S0: Okay. So this is actually really important because if you're delivering to somebody else [00:17:11] S1: Mhmm. [00:17:11] S0: You were talking about saving money, saving money, saving money. But, like, you don't want your girlfriend to think you just got the cheapest flower possible. I hope... Well, maybe, you know, she likes your, you know, penny saving nature. But but otherwise, like, I would think you also wanna, like, impress her and also be able to get her the ones if she's really into orchids to get her orchids. So did it do anything to try to make sure also that it wasn't just this pure price saving, but also something that's gonna actually make you, you know, earn the excitement and stand out a little bit for [00:17:38] S1: So for this experiment, no. And and the purpose of this was I wanna test two different dimensions. Yeah. And if I'm going to run this test, I might as well have a beautiful consequence of my girlfriend getting flowers Fair enough. As the outcome. But it... It's a great point. Right? Like, in in the real world, if you're gonna ask that question, there's gonna be added dimensions that are, you know, second layer dimensions to cost efficiency or to delivery time. And make [00:18:01] S0: sure the price doesn't show
00:18:01 · Josh Elman, David Pawlan, Anish Acharya
[00:18:01] S0: sure the price doesn't show [00:18:02] S1: up when she actually gets the flowers. Or like a note as an example. Some of them did ask, do you want me to write... Leave a note with with the flowers when they arrive? [00:18:09] S2: Yeah. It's interesting because there's this sort of thing that has to happen when you talk to the agent about these things, which is your implicit preferences have to be made explicit. [00:18:17] S1: Mhmm. [00:18:17] S2: You know, and there are people who like to think like, oh, I'm not cheap. I just like to save money. Right. But you're not gonna have to kinda maybe like get over that and tell the agent, I'm actually pretty cheap. Right. And I just need you to actually run with that. Whereas when you're actually taking these actions on your own behalf, you can build those preference in without having to, like, sort of look in the preference mirror. Yep. So there's also something very fascinating about how does the agent interact with you in a way that sort of maintains your sense of identity and safety and doesn't cross any of these boundaries. Totally. [00:18:46] S1: And and I think something that is really interesting with that specifically is Dot because it has so much of this context that you've already built with ChatGPT. Interesting. When we filmed the first podcast, it hadn't come out yet. [00:18:57] S2: And then
00:18:57 · Anish Acharya, David Pawlan, Josh Elman
[00:18:57] S2: And then [00:18:58] S1: I think a few days later, it it came out. That's right. By and large, most people I talk to still are not using it as their preference of the 300... I did a survey of, you know, AI power users, 354 people. Only 3% were actually using it as as their primary agent. However, it does have this kind of added value of the the main thing that people pointed to as to why they really preferred it and why they liked it and why they wanted to try it was because it has all this building context. Yeah. A view just from your chat threads. People are using ChatGPT as a therapist, and so it has so much context on you. [00:19:29] S2: Interesting. [00:19:30] S1: Where it's gonna know that when you say, well, I'm not cheap. It knows it's like [00:19:33] S2: You're a little cheap. A [00:19:35] S1: little bit. A little bit. [00:19:35] S0: I'm curious why people think that ChatGPT has this great set of preferences from... On them. Did you actually get any sense if they said they really like dots, like, what they had been giving to ChatGPT over the years? And do you think they think that's replicable? Or do you think it's like, hey. Once it's already been a few years, we're now in this point where, like, my memory's there. Why would I switch?
00:19:35 · Josh Elman, David Pawlan
[00:19:35] S0: I'm curious why people think that ChatGPT has this great set of preferences from... On them. Did you actually get any sense if they said they really like dots, like, what they had been giving to ChatGPT over the years? And do you think they think that's replicable? Or do you think it's like, hey. Once it's already been a few years, we're now in this point where, like, my memory's there. Why would I switch? [00:19:56] S1: It's totally replicable. It... It's it's just the matter of fact that they've been using it the longest, and so it has the most context on them. But then that brings up another good point of the the friction to switch. Yeah. And I I ran the test across the top 15 agents on the benchmark of which ones can you actually pull context from and say, like, hey. I wanna switch to another one. Give me an empty of all of my memory that you have so I can switch easily. All but one we're at... We're actually willing to do it. And so you could just get a memory file, everything you need, submit it to the next one. [00:20:24] S0: And was it good, or was it uncanny? [00:20:26] S1: It's it's pretty solid. I mean, it's... Obviously, I'm sure they're they're caging some some of some of the fruits of of your labor. Yeah. But by and large, it it did really break everything down.
00:20:26 · David Pawlan, Anish Acharya, Josh Elman
[00:20:26] S1: It's it's pretty solid. I mean, it's... Obviously, I'm sure they're they're caging some some of some of the fruits of of your labor. Yeah. But by and large, it it did really break everything down. [00:20:35] S2: I'm not sure if you've ever looked at any of those markdown files, but it's a little... Terrifying is the wrong word, but it's like, wow, you really got your finger on me. Yeah. Kinda like chafe against me. Like, may... I don't want you to know me as well as you seem to think you know me. [00:20:49] S1: Yeah. After the first few, I just... I decided to stop reading them. [00:20:51] S2: Yes. Yes. Yes. Yes. It's actually a really interesting and now common onboarding pattern where it will say, hey. Go and paste this prompt into ChatGPT. Download the file. Upload it to me. Yep. We'll see how long that lasts. [00:21:03] S1: Totally. Another cool onboarding thing I've seen is you jump on... Like, it asks you to jump on a call. Yeah. And it's like, hey. Let's just just talk to me for a few minutes. Yeah. Yeah. Ask some questions. And it's it's really interesting to see what context it can it can pull from those questions. [00:21:18] S0: And they're very disarming. Like, they just get you Like, it's kinda crazy. [00:21:21] S1: Totally. But I think I think it brings up another variable of context, which is photos. Right? Like, this this new paradigm of using photos to pull an incredible amount of context on you in ways that you wouldn't even think to describe. Yeah. I mean, I