Zero-Shot Learning
00:11:59 · Adarsh Hiremath
[00:11:59] S2: The platform that we use has three components to it. Discover, deploy, and improve. Right? Discover is you connect with all the data sources of a company. Right? SaaS applications, the actual subject matter experts who know what good looks like in that context, so on and so forth, documents. You're able to pull it out and put that in a digital space that is accessible for an agent that is retrievable. And and every time that agent runs, it can pull the right context contextualized to that company. Mhmm. The second thing is actually deploying the agent. Typically, you know, the the way that humans interact with these agents is in platforms like Slack. Right? So it's it's where work happens. You you wanna invoke the agent where you're already doing your workflows, and you actually wanna make the UI UX such that people are, excited to to use it, and it's a pleasure to use it as opposed to a pain. And then last is the improve step, which is where a lot of the magic is, where you want an eval set to figure out where the models are are failing. And when you do figure out a failure, you wanna one update that eval set to make sure that that failure never happens again, and then you can run it every single time you wanna redeploy the model. And then the second thing is you wanna take the piece of context that addresses that model failure, suggest a diff to the skills file, and then put that in the specification of the agent.
00:13:11 · Dev Tagare, Adarsh Hiremath, Nancy Wang
[00:13:11] S1: Makes a lot of sense. Exactly. And, like, so to to paraphrase or or summarize, the updates are happening via skills. Skills are learned by actually a combination of, like, human interactions across problems, across surfaces, and you triangulate the three and other evidence or grounding data you might collect over time to then continuously update skills and then push that back so that becomes your feedback [00:13:39] S2: loop Exactly. [00:13:40] S1: Into the system. [00:13:41] S3: So, an example, maybe, that crosses many different functional groups. For example, demand gen pipeline. Crosses marketing, sales, can also touch product teams because it might talk about certain feature requests. So, in that case, where do you draw the line to, you know, Dave's point around skills, around super structured specification versus, let's say, tacit knowledge that is very contextualized to a specific company?
00:14:07 · Adarsh Hiremath, Dev Tagare
[00:14:07] S2: Right. So, question is, where do you draw the line between company specific and then skills Well, that are thankfully, there's there's a lot of overlap between that. Right? For example, a model that has underlying, you know, reasoning capabilities that are really, really good will will likely do a better job of of demand gen than a model that has really bad reasoning abilities. But in that specific use case, right, the part that's really, really cool about a use case like that is that it involves so many different departments and context sources. Right? Just to maybe give one example of a deployment for that use case is you would want the agent to actually have access to all the different releases that are happening. So maybe something like Jira, a Slack channel with releases. Secondly, is it would want the typical guidelines of a marketing department. Right? What is the actual thing that you would Right. Put out as a as a piece that is designed to get demand or email outbound that is designed to get demand. So that could be Google documents. It could be emails. It could be, onboarding documents for the marketing team. And then the third thing is you would want to actually give it the ability to do the outbound motion, right, to generate more and more demand. So that is the tool and the capability to to do that, whether it be an email, socials, or or other things. Right? [00:15:11] S1: Yeah.
00:15:11 · Dev Tagare, Adarsh Hiremath
[00:15:11] S1: Yeah. [00:15:11] S2: Yeah. And let's say you queue up a a piece of material that is not desirable or one of the stakeholders flags as, you know, being incorrect. Can you go into the platform and update and suggest the diff? And do that over and over again for a bunch of use cases. [00:15:23] S1: Makes a lot of sense. So, like, we're we're sort of entering into the the context discussion, if you will. One thing that keeps coming up, Adarsh, is the ontology for context. Right? Different companies call kinda similar things differently. I may call Jira a buganizer, you may call Jira just a ticket or in my queue or something like that. Right? And the fields could be different. So in this space, how do you deal with the problem of semantic meaning of data or ontology as a problem space? And then how do you define success, in terms of, like, an on ramp, assuming the company's ontology Mhmm. And your interpretation of it? [00:16:04] S2: The thing that we have found is understanding semantic meaning in a very, very big comp... You know, the company dataset is actually more or less a solved problem. Mhmm. Right? Of course, there are different optimizations that you can make to make recall, you know, better and better. Mhmm. But for the most part, taking that data, splicing it appropriately [00:16:25] S1: Mhmm.
00:16:25 · Dev Tagare, Adarsh Hiremath
[00:16:25] S1: Mhmm. [00:16:26] S2: Putting it in a high dimensional space using whatever embeddings model you want, then retrieving over it is something that will give you very, very good results, especially when paired with a really, really smart model. So that understanding piece is a problem that we support, but it's not as interesting to us as the second problem, which is how do you actually create the graph that represents all the relationships between the different data sources of a company. Right? Like, it's less interesting to us to know what what linear means in a in a company's context or or Jira or Buganizer or something like that. Mhmm. It's how does that actually fit into the whole, you know, process of shipping a feature? Yep. Right? What happens after Jira? [00:17:04] S1: Yep. [00:17:05] S2: Where does the collaboration for human work happen? Is it Slack or something else? What does good collaboration look like? When you're reviewing a, you know, feature that's about to be released, what does that review process look like? And when you're releasing the feature, what does the feedback loop look like if there's a a big error, and you wanna actually find out that error, maybe push a fix automatically. Right? Like that whole set of different steps and tying them together and making sure the knowledge graph for that company's context is is actually something that represents relationships accurately is the harder part.
00:17:05 · Adarsh Hiremath, Dev Tagare
[00:17:05] S2: Where does the collaboration for human work happen? Is it Slack or something else? What does good collaboration look like? When you're reviewing a, you know, feature that's about to be released, what does that review process look like? And when you're releasing the feature, what does the feedback loop look like if there's a a big error, and you wanna actually find out that error, maybe push a fix automatically. Right? Like that whole set of different steps and tying them together and making sure the knowledge graph for that company's context is is actually something that represents relationships accurately is the harder part. [00:17:33] S1: And and in this, again, we we come back to the question of, like, time is not continuous. As an example, I may pick up a task today, I may do it over the next five days, abandon it randomly, pick it up in two weeks again, so on and so forth. And in your process of discovery, maybe you find this as a high impact task to have actually done. So help us understand how do you think about that process of discovery and then tying it back to disaggregated steps, if you will. That models to your point earlier, basically an agent trajectory. We're moving from agent is solving for a goal to agent is sticking to a trajectory to get to the goal side of paradigm. How do you think about that?