Zero-Shot Learning
00:09:36 · Dev Tagare, Adarsh Hiremath, Nancy Wang
[00:09:36] S1: Right. [00:09:36] S2: Right? It's just super ramble y, verbose, including information that... [00:09:40] S3: Or gives everyone a free flight ticket, you know? [00:09:42] S2: Or or it could give everyone a a free flight ticket, and there could be a glitch that uppays the customer out a thousand dollars for every interaction. Right? You you don't know. So you need quality control and evals over the whole chain of processes to understand whether it's actually doing a job well. [00:09:55] S1: Along the same trajectory, how do you account for, like, disaggregated learning? As an example, feedback loop in in the customer support example could be my deflection rate or resolution rate is 80%. [00:10:08] S2: Mhmm. [00:10:08] S1: But then six months later, I'm getting complaints on a specific arc or axis or trajectory, and then that is feeding back into the the overall process. Like, how does that work?
00:10:08 · Dev Tagare, Adarsh Hiremath, Nancy Wang
[00:10:08] S1: But then six months later, I'm getting complaints on a specific arc or axis or trajectory, and then that is feeding back into the the overall process. Like, how does that work? [00:10:19] S2: Yeah. So maybe the high level observation is use cases where the agent has to balance a bunch of different stakeholders' perspectives and also aggregate a bunch of different contexts are the big rock problems that we wanna solve, and they're super hard to solve because it's not as straightforward as me just prompting my GPT, claw, Gemini, whatever favorite model. Right? Because I'm always there to steer it. I have a certain set of preferences. It adds that to the memory. Right? So category a of problems with many stakeholders, a bunch of different sources of context are the really, really tricky problems to solve. Mhmm. And the way that we've we've gone about it is making sure that you can actually index the entire company's context by pulling in all the different sources. [00:11:02] S3: Mhmm. [00:11:02] S2: And that context is typically beyond a single person or a single data source. Mhmm. It's supposed to be a comprehensive and holistic view of what good of that company looks like. [00:11:09] S1: 100%.
00:11:09 · Dev Tagare, Adarsh Hiremath, Nancy Wang
[00:11:09] S1: 100%. [00:11:10] S2: And the second thing is what happens when there's actually a failure. Right? Mhmm. Typically, models can actually identify these failures if it's not confident in the answer, and then escalate it in triagic to a human. Typically, the first round of of learning updates we wanna do are in context space, not parameter space, because it's way more compute efficient. So then you can triage it to a human, update the skill. Mhmm. And if it warrants it, you could do fine tuning on model if you're using an open source model. But given how smart the models are, the former is actually usually a better approach. [00:11:38] S1: 100%. Yeah. [00:11:39] S3: So, maybe coming to, like, the technical deep dive, right, I think Dave started to unpack the layers a bit. Going back to your point around how work's actually being done, and not just sort of, you know, handcrafting prompts and sending it out there with with best hopes. Right? Is... What are some platform primitives that you're building to contextualize all of that information?
00:11:59 · Adarsh Hiremath
[00:11:59] S2: The platform that we use has three components to it. Discover, deploy, and improve. Right? Discover is you connect with all the data sources of a company. Right? SaaS applications, the actual subject matter experts who know what good looks like in that context, so on and so forth, documents. You're able to pull it out and put that in a digital space that is accessible for an agent that is retrievable. And and every time that agent runs, it can pull the right context contextualized to that company. Mhmm. The second thing is actually deploying the agent. Typically, you know, the the way that humans interact with these agents is in platforms like Slack. Right? So it's it's where work happens. You you wanna invoke the agent where you're already doing your workflows, and you actually wanna make the UI UX such that people are, excited to to use it, and it's a pleasure to use it as opposed to a pain. And then last is the improve step, which is where a lot of the magic is, where you want an eval set to figure out where the models are are failing. And when you do figure out a failure, you wanna one update that eval set to make sure that that failure never happens again, and then you can run it every single time you wanna redeploy the model. And then the second thing is you wanna take the piece of context that addresses that model failure, suggest a diff to the skills file, and then put that in the specification of the agent.
00:13:11 · Dev Tagare, Adarsh Hiremath, Nancy Wang
[00:13:11] S1: Makes a lot of sense. Exactly. And, like, so to to paraphrase or or summarize, the updates are happening via skills. Skills are learned by actually a combination of, like, human interactions across problems, across surfaces, and you triangulate the three and other evidence or grounding data you might collect over time to then continuously update skills and then push that back so that becomes your feedback [00:13:39] S2: loop Exactly. [00:13:40] S1: Into the system. [00:13:41] S3: So, an example, maybe, that crosses many different functional groups. For example, demand gen pipeline. Crosses marketing, sales, can also touch product teams because it might talk about certain feature requests. So, in that case, where do you draw the line to, you know, Dave's point around skills, around super structured specification versus, let's say, tacit knowledge that is very contextualized to a specific company?
00:14:07 · Adarsh Hiremath, Dev Tagare
[00:14:07] S2: Right. So, question is, where do you draw the line between company specific and then skills Well, that are thankfully, there's there's a lot of overlap between that. Right? For example, a model that has underlying, you know, reasoning capabilities that are really, really good will will likely do a better job of of demand gen than a model that has really bad reasoning abilities. But in that specific use case, right, the part that's really, really cool about a use case like that is that it involves so many different departments and context sources. Right? Just to maybe give one example of a deployment for that use case is you would want the agent to actually have access to all the different releases that are happening. So maybe something like Jira, a Slack channel with releases. Secondly, is it would want the typical guidelines of a marketing department. Right? What is the actual thing that you would Right. Put out as a as a piece that is designed to get demand or email outbound that is designed to get demand. So that could be Google documents. It could be emails. It could be, onboarding documents for the marketing team. And then the third thing is you would want to actually give it the ability to do the outbound motion, right, to generate more and more demand. So that is the tool and the capability to to do that, whether it be an email, socials, or or other things. Right? [00:15:11] S1: Yeah.