Why We Killed Our Multi-Agent Pipeline — Subbiah Sethuraman and Abhilash Asokan, ZS Associates

频道: AI Engineer
视频: https://www.youtube.com/watch?v=u6jJcIFDLE4
原文语言: en
统计: 共 38 轮 · Suba 14 · Abhilash 21

[0:01]

[music] Uh, okay. So, I'm Suba. I head AI engineering at ZS. Uh, >> I'm Ablash. I'm director of AI engineering at CS. >> So, ZS, we are a tech firm. We work with many of the top companies in the world, including a lot of the top farmers actually. Um so today's talk I think we wanted to as I think already introduced right we wanted to talk from our experience right building multi-agent pipelines uh what are the mistakes we

[音乐] 呃，好的。我是 Suba，我在 ZS 负责 AI 工程。呃——我是 Abhilash，我是 ZS 的 AI 工程总监。ZS 是一家科技公司，我们和全球很多顶尖企业合作，其中包括很多顶尖的 pharma 公司。那么今天这场分享，就像刚才介绍的那样，我们想结合自己的经验来聊一聊——我们在构建 multi-agent pipeline 的过程中，都犯了哪些错误

[0:34] Suba

did right and what did we learn and how did we fix them so um so I'm going to orient it more on pharma commercial domain and for people in the room probably who are not aware quickly u pharma has two main functions one is R&D right the drug discovery right uh and the clinical trials part and then there's a commercial how do you take a drug to a patient basically and and Within commercial there are different functions like once you create a drug

犯了哪些错、学到了什么，以及是怎么把它们修复的。我会更多地围绕 pharma 的商业化领域来讲。在座可能有些人不太了解，我快速说一下：pharma 主要有两大职能，一个是 R&D，也就是药物发现和临床试验这部分；另一个是商业化，简单说就是怎么把一款药送到患者手里。而在商业化里面又有不同的职能，比如说你造出一款药之后

[0:59] Suba

right um what is the performance of a brand how is the drug performing in different markets right uh then there are things right your field force your reps how effectively are they engaging uh right with everyone there are things around patient journey how a patients are adopting a drug right I think if there is any therapy switch which is happening so as you can think about there is a lot of analytics which really happens in a commercial domain and how

一个品牌的表现怎么样、这款药在不同市场里卖得如何；还有你的一线队伍、你的销售代表（reps），他们跟各方的沟通触达到底有多有效；还有围绕患者旅程（patient journey）的东西，患者是怎么开始用这款药的、有没有发生治疗方案的切换。所以你可以想象，商业化领域其实有大量的分析工作在进行。那么

[1:22] Suba

do typically analysts So there are four steps right what analysts do right so first there is always something called a signal detection right so signal can be something like okay the prescription is what a doctor is writing maybe is there is there a drop in the prescription so that's a signal so once you got a signal the second thing what an analyst does is why is this signal really failing what is the reason for it so is it like there

分析师通常是怎么做的呢？他们的工作大概分四步。第一步永远是所谓的 signal detection（信号检测）。这个 signal 可以是这样的：处方是医生开出来的，那处方量有没有下降？这就是一个 signal。一旦你拿到了一个 signal，分析师做的第二件事就是：这个 signal 背后到底哪里出了问题、原因是什么？是不是因为

[1:46] Suba

is a competitor drug which has come in because of that is it is it reducing is it is it because maybe a payer coverage for the drug has reduced or maybe the the reps on the ground there's no proper uh they're actually not taking the benefits to the doctors and once you arrive at the reason the next step becomes okay what is the action do you take so if reps suppose if reps the coverage is not good in a particular region should do we have to

有一款竞品药进来了，所以导致处方在下降？还是因为某个 payer 对这款药的报销覆盖降低了？又或者是一线的 reps 没到位、他们其实没有把这款药的好处真正传达给医生？一旦你找到了原因，下一步就是：那你要采取什么行动（action）？比如说，如果某个区域里 reps 不够、覆盖不够好，那我们是不是得

[2:10] Suba

increase that and once you do that what is the what is the outlook right is my brand is my sales performance is is it going to improve right so these are the four things which happens now for some of the top farmers what we have done is how do we in an agentic way right I think how do we actually mimic this this analytics work so what we did we built agents for every step right signal detection we said okay we'll have

把它加上去？做完这件事之后，接下来的展望（outlook）又是什么——我的品牌、我的销售表现会不会因此改善？所以就是这四步。那么对于一些顶尖的 pharma 客户，我们做的事情就是：怎么用一种 agentic 的方式，把这套分析工作给模拟出来。于是我们为每一步都构建了 agent。signal detection 这一步，我们说，好，我们来搞一个

[2:34] Suba

an agent for signal detection it'll identify identify the signals for me second what are the root cause right for the signals right so in this case we have two agents one we call it a source localization so for example if my sales is dropping at a national level is it because it's dropping at say a particular region or is it dropping for a payer So we need to understand that we we are trying to identify the source of it and

专门做 signal detection 的 agent，让它帮我识别出这些 signal。第二步是找这些 signal 的根因（root cause）。这里我们用了两个 agent，一个我们叫它 source localization（来源定位）。举个例子，如果我的销售在全国层面下滑，那到底是因为某个特定区域在下滑，还是因为某个 payer 在下滑？我们得搞清楚这一点，也就是要定位问题的来源。然后

[2:57] Suba

then once we understand what is the real reason right I think the the sales performance has gone down that's another agent the driver attribution agent and then the last step is your synthesis right so depending on the cause now what is action you have to take and what is outlook like typically how an analyst used to do and all of this we used to have an orchestrator agent which connects all these agents together so now what happened once we had the

一旦我们搞清楚真正的原因是什么——也就是销售表现为什么下滑了——这就是另一个 agent，叫 driver attribution agent（驱动因素归因）。最后一步是综合（synthesis），根据原因得出你该采取什么 action、展望（outlook）会怎样，就跟分析师平时做的一样。而所有这些 agent，我们之前是用一个 orchestrator agent 把它们串在一起的。那么，当我们有了这套

[3:20] Suba

system what did it generate right It generates something an information packet like this, right? It clearly tells you the signal, right? So, first it says maybe my brand's prescriptions have dropped 18% in some territory, right? In some time frame, maybe four weeks. It then tells you the reason why did it why did it drop, right? The reason it says because a payer actually, right, the coverage for this drug has actually they moved it

系统之后，它会生成什么呢？它会生成一份类似这样的信息包（information packet）。它会清楚地告诉你 signal。首先它会说，我的品牌处方量在某个区域下滑了 18%，是在某个时间段内、比如说四周里。然后它会告诉你原因，为什么会下滑。它给出的原因是：因为某个 payer 实际上把这款药的报销

[3:44] Suba

to a lower tier. So, for patients, it's expensive actually to to buy this drug. the action it says okay because doctors are writing less prescriptions maybe send more sales reps to talk to doctors right and increase the number of prescriptions which you're writing and then if you take this action maybe your outlook your sales performance is going to increase so all of this looks good high level but then if you if you look

调到了更低的档位（tier），所以对患者来说，买这款药其实变贵了。而它给出的 action 是：既然医生开的处方变少了，那就多派点销售 reps 去跟医生沟通，把处方量提上去。然后如果你采取这个 action，也许你的 outlook、你的销售表现就会改善。所有这些从高层面看都挺不错，但你要是仔细看

[4:08] Suba

at at it closely it's not very coherent right the cause is right it identified the right cause right because patients can't afford the d drug right but the action it said it didn't really focus is on the payer part, the insurance part of it, right? It just said reps, right?

就会发现它其实不太连贯。原因是对的，它识别出了正确的 cause——因为患者负担不起这款药。但它给出的 action 却没有真正聚焦在 payer 这块、也就是保险这块，它只说了 reps，对吧？

[4:24] Suba

Focus more reps actually, right? And then the outlook because the action is wrong, the outlook is not going to match. So at each level, if you see it is actually derived the right fact, but then there is no single agent which is owning which understands the end to-end picture basically.

它说的是多投入 reps。而由于 action 错了，那 outlook 自然也就对不上了。所以你看，在每一个环节上它其实都推导出了正确的事实，但是没有任何一个 agent 是真正统筹全局、理解端到端整体图景的。

[4:40] Suba

So why did this happen? Like why did it fail, right? So obviously it's not the LLM which failed, right? It's the way how we split the work, right? because we tried mimicking the analyst behavior and we did it. The first first key issue is like a language model is actually determining your signals. So signals like things like your sales drop is a simple information which you can use statistical methods to actually go and

那为什么会这样？为什么会失败呢？显然不是 LLM 失败了，而是我们拆分工作的方式出了问题。因为我们试图去模拟分析师的行为，而且我们真的照做了。第一个关键问题是：我们让语言模型去判定你的 signal。可是像销售下滑这种 signal，其实是很简单的信息，你完全可以用统计方法去

[5:04] Suba

fetch this information. You don't need a language model actually right to fetch this information. Second is as your multi- aents there's a lot of context handoff which is happening and context is actually getting lost at each of these handoffs.

把这个信息取出来，根本不需要用语言模型。第二个问题是，在你的 multi-agent 系统里，会发生大量的上下文交接（context handoff），而在每一次交接的过程中，上下文其实都在丢失。

[5:17] Suba

So for example, the driver attribution agent is actually determining the right cost. But then the next agent, the synthesis agent is actually is not able to understand why is that right the the payers are finding the drug to be expensive. It's not understanding the weightage of an insurance coverage going going down. So that is a key information right which is getting lost. And the last big piece is there's no shared

举个例子，driver attribution agent 其实已经判断出了正确的原因，但接下来的那个 agent、也就是 synthesis agent，却没法理解为什么——为什么 payer 会觉得这款药很贵。它理解不了保险报销覆盖下降这件事的分量。所以这是一条关键信息，就这么丢掉了。而最后一大块问题是：这些 agent 之间没有共享的

[5:39]

understanding of the business domain knowledge for all these agents. All these agents don't understand metrics right so things like TRX the number of transa number of prescriptions which a doctor writes right what are the relationships between them why does it go up or down so those are the the reasons so then what did we do uh so I called Abilash to come and solve for this okay uh thank you SA so we had like three problems uh so we

对业务领域知识没有共同的理解。所有这些 agent 都不理解那些指标（metrics），比如 TRX（处方总量），也就是医生开出的处方数量，它们之间是什么关系、为什么会涨会跌。这些就是失败的原因。那我们后来做了什么呢？于是我把 Abhilash 叫过来解决这个问题。好的，谢谢 Suba。所以我们当时有三个问题，我们

[6:05] Abhilash

thinking how are we going to solve this uh our first instinct was we'll go back to the drawing board we'll start designing it again. So maybe the topology was wrong, the skills were wrong, the tools were wrong, or maybe the maybe we have to define a better handoff, design a better handoff or a better schema between agents.

在想该怎么解决这个问题。我们的第一反应是：回到起点，重新再设计一遍。也许是 topology 错了，也许是技能（skills）错了、工具（tools）错了，又或者我们得定义一个更好的交接（handoff）、设计更好的 handoff，或者 agent 之间更好的 schema。

[6:26] Abhilash

But we took a step back. We didn't do any of that. We like all of us, we went back to cloud code. So we opened a very plain empty directory. I rent cloud code then give it just bash and the database then give it an actual signal which we identified then started observing what it is doing. So while we look at what code does, we are able to figure out fix for all the three issues we discussed the first the first part.

但我们退了一步，这些我们一件都没做。我们所有人一起，回到了 Claude Code。我们打开一个非常朴素的空目录，我运行 Claude Code，只给它 bash 和数据库，再给它一个我们真实识别出来的 signal，然后开始观察它到底在做什么。在观察 Claude Code 行为的过程中，我们把前面讨论的那三个问题的解法全都找到了。先说第一部分。

[6:58] Abhilash

So what was happening is the agent was looking at data and deciding on a signal. Sometimes it's applied some statistical methods. Sometimes it's barely look at the data and say this is a signal. Sometimes it's actually a signal. Sometimes it's a noise. This is something we don't want an agent to do.

当时的情况是，agent 在看数据，然后自己判定一个 signal。有时候它会用一些统计方法，有时候它几乎没怎么看数据就说这是个 signal。有时候那确实是个 signal，有时候其实只是噪声。这正是我们不希望 agent 去做的事情。

[7:16] Abhilash

This is a completely deterministic workflow. So we separated it out from the agentic system. So we built a pure deterministic workflow with different statistical methods. We put guard rails, we put thresholds, we put prioritization. Everything happened before the agent even kickstarts. So we run an automated pipeline which scans through the data identify signals for each of the KPA. Is anything happening with that? any

这完全是一个确定性的（deterministic）工作流。所以我们把它从 agentic 系统里剥离出来，用各种统计方法搭了一个纯确定性的工作流。我们加了护栏（guard rails）、加了阈值（thresholds）、加了优先级排序。所有这些都在 agent 启动之前就完成了。于是我们跑一条自动化的 pipeline，扫描数据，为每一个 KPI 识别 signal。它有没有出现什么状况？有没有任何

[7:42] Abhilash

anomalies which is happening any trend which is very uh any trend which is baking based on that we identified a signal we put it on a queue the moment a signal comes to the queue the agent wakes up so the agent's dope is to investigate not to identify the second part so mainly the issue which we previously what sub was mentioning there is no coherence in the output which the agents produced so we started consolidating we Look at

异常，有没有什么正在形成的趋势。基于这些，我们识别出一个 signal，把它放进一个队列里。signal 一进队列，agent 就被唤醒。所以 agent 的职责是去调查（investigate），而不是去识别。第二部分，主要就是前面 Suba 提到的那个问题——agent 产出的结果之间缺乏连贯性。所以我们开始做整合，我们观察

[8:12] Abhilash

how cloud code operates. It's able to do a lot of operations. So we started designing around that. So it's repeatedly writing a function and querying database. So we give it a tool for that. So this consolidated the entire process into a single agent. That doesn't mean that we didn't do parallelism. We still do parallelism. What we removed is do we need distributed reasoning? We didn't by the judgment to be distributed between

Claude Code 是怎么运作的。它能做很多种操作，于是我们就围绕这一点来设计。它会反复地写一个函数、去查询数据库，那我们就给它一个专门做这件事的工具。这样就把整个流程整合进了单个 agent。这并不是说我们不做并行（parallelism），我们仍然会做并行。我们去掉的是——我们真的需要分布式推理（distributed reasoning）吗？我们不希望判断被分散到

[8:39] Abhilash

agents that we wanted to consolidate to a single agent. So that what exactly what we did. Then occasionally we observed cloud code it's launching sub aents dynamically for a very particular focused task. So we did the same thing because if suppose you want to understand if rep activity in a particular region that's an investigation which you need to run that you can still delegate to a sub agent you can get back the uh results back not

各个 agent 之间，我们想把它整合到单个 agent 里。所以我们就是这么做的。后来我们偶尔观察到，Claude Code 会针对某个非常具体、聚焦的任务，动态地启动 sub-agents。于是我们也照做，因为假设你想了解某个特定区域的 rep 活动情况，那是一次你需要去跑的调查，你仍然可以把它委派给一个 sub-agent，把结果拿回来，而不是

[9:04] Abhilash

the reasoning or the judgment that is still controlled by the by the main agent but the investigation part of it we did we're delegating it to a sub agent. So these are some of the things we kept based on observing what code was do cloud code was doing. So that give us a more lighter architecture than what was initially there. But still it doesn't solve the problem. That's why you see a knowledge graph in the diagram.

把推理或判断也交出去——推理和判断仍然由主 agent 来掌控，但调查这部分，我们确实会委派给 sub-agent。这些就是我们在观察 Claude Code 的行为之后保留下来的一些做法。这让我们得到了一个比最初轻量得多的架构。但这仍然没有解决问题，这也是为什么你会在图里看到一个 knowledge graph。

[9:28] Abhilash

So it still doesn't have the business context. It still doesn't understand all the entities, the domain, the KPIs, how do they relate to each other. So that's something we wanted to solve for because the agent was looking at data looking at tables then trying to infer the relationship that which was not scalable and it often produce relationship which is which is not actually exist in the data.

因为它仍然没有业务上下文（business context），仍然不理解所有的实体（entities）、领域、以及各个 KPI 之间是怎么关联的。这是我们想要解决的问题，因为 agent 当时是在看数据、看表，然后试图去推断它们之间的关系，而这是没法规模化的，而且它经常推断出数据里根本不存在的关系。

[9:57] Abhilash

So what we did so we've been in this working in this field for a lot whole lot of years. So we've been doing this for our clients. So we had a lot of domain experts who understand the farmer domain very well. We're doing commercial analytics for the clients. So we start sat with them start building a knowledge graph. We start trying to map out the domain.

那我们做了什么呢？我们在这个领域已经深耕很多很多年了，一直在为客户做这些事。所以我们有很多非常懂 pharma 领域的领域专家，我们一直在为客户做商业化分析。于是我们就跟他们坐下来，一起开始构建 knowledge graph，开始尝试把这个领域梳理、映射出来。

[10:19] Abhilash

So we are able to build a knowledge graph. We are able to identify different entities their relationship between that. So if you see there are like geographic entities, there are payers. How how geographic is connect to a payer or an account? How does this connect to a brand? Then from B brand or payer, how does it go to a KPI? How does a KPI relate to like a uh secondary KPI,

于是我们成功构建了一个 knowledge graph，能够识别出不同的实体以及它们之间的关系。你看，这里有地理实体（geographic entities），有 payer。地理是怎么跟一个 payer 或者一个 account 连接的？它又是怎么跟一个品牌连接的？然后从品牌或者 payer，怎么走到一个 KPI？一个 KPI 又是怎么跟二级 KPI 关联的，

[10:42] Abhilash

tertiary KPI? How does one KPI drives another KPI? So, we started mapping out all of this information and then creating our our knowledge graph. Once you have the knowledge graph, then we wanted to let the agent to navigate the knowledge graph. So, the knowledge graph is not just something the agent looks up for data. It is a control plane for the agent.

以及三级 KPI 的？一个 KPI 又是怎么驱动另一个 KPI 的？所以我们开始把所有这些信息梳理映射出来，然后创建我们的 knowledge graph。一旦有了 knowledge graph，我们就想让 agent 去在这个 knowledge graph 里导航。所以这个 knowledge graph 不只是 agent 用来查数据的东西，它是 agent 的控制平面（control plane）。

[11:04] Abhilash

So, what do I mean by that? So the knowledge graphs dictates what the agent can look into, what path it can take, what investigation hypothesis uh it can evaluate. So for example, if you when you do this analysis, the root of the problem say something like your TRX is declining at a national level. This could be when you do source localization. So that's our uh contract for how to find the bear. It might happen within a region. It could

这是什么意思呢？knowledge graph 决定了 agent 能查看什么、能走哪些路径、能评估哪些调查假设（investigation hypothesis）。举个例子，当你做这个分析的时候，问题的根源可能是这样的——你的 TRX 在全国层面正在下滑。这时你会去做 source localization，这就是我们关于「怎么找到问题所在」的约定。它可能发生在某个区域内部，也可能

[11:33] Abhilash

be concentrated in a territory. it would be concentrated in a combination of a territory or a payer or an account. So there are whole lot of dimensions which the agent needs to evaluate. Now it's a lot of permission permutation combination. So the knowledge graphs gauge the agent on how to find where this is concentrated.

集中在某个 territory 里，也可能集中在 territory、payer 或 account 的某种组合上。所以 agent 需要评估的维度非常多，这就是一大堆的排列组合。而 knowledge graph 会引导 agent 去找到问题究竟集中在哪里。

[11:50] Abhilash

Then the why why part of it why it is happening. When you figure out that okay something is concentrated at say the decline is concentrated at a particular region. Now you need to figure out the why. This is where where the uh KPS and the relations comes in. Once the agent is able to narrow down the where then it's able to go and figure out the u figure out the why part of it, which KPS is driving what graph acts as the control surface. So

接下来是「为什么」的部分——为什么会发生这种情况。当你搞清楚了，比如说下滑集中在某个特定的区域，接下来你就得弄明白背后的原因。这时候 KPI 和这些关联关系就派上用场了。一旦 agent 能把「在哪里」这个范围缩小，它就能进一步去搞清楚「为什么」的部分，也就是哪个 KPI 在驱动着什么，而 graph 在这里就扮演了 control surface（控制面）的角色。所以

[12:18] Abhilash

agent every edge is a hypothesis. So the agent can go and evaluate that hypothes hypothesis. Uh it doesn't go outside of this. So that gives a more bounded surface for the agent to investigate. So like everyone was mentioning about loop. So we also build a loop. So what the agent does? So the agent first start with an entity. It goes to the graph.

对 agent 来说，每一条 edge（边）都是一个 hypothesis（假设）。所以 agent 可以去验证这个假设，而且它不会跑到这个范围之外，这样就给 agent 划出了一个边界更清晰的探查空间。就像大家之前都提到 loop 一样，我们也搭了一个 loop。那 agent 具体是怎么做的呢？agent 首先从一个 entity（实体）出发，进入到这个 graph 里。

[12:46] Abhilash

It looks at the neighborhood of the graph. Then it it figures out the edges. So it's got some hypothesis. It will go it'll go back to the original data. evaluate that hypothesis. Look at the actual numbers. Then reason over it. Then it will it will either find it find it contradicting or it finding supporting the the evidence supporting the hypothesis. If it's supporting it,

它会去看这个 graph 里的邻居节点，然后找出那些 edge，于是就得到了一些假设。接着它会回到原始数据里去验证这些假设，看看实际的数字，然后基于这些做推理。之后它要么发现证据跟假设是矛盾的，要么发现证据是支持这个假设的。如果证据是支持的，

[13:10] Abhilash

it started traversing through the graph. So this repeats this until it ran out of all the hypothesis or it's able to figure out the root cause. So this this concludes this run. this runs and it'll be able to figure out the uh root cause of the problem.

它就会继续在这个 graph 上往下遍历。这个过程会一直重复，直到所有假设都验证完，或者它找到了 root cause（根因）为止。这样一轮运行就结束了。这么跑下来，它就能找出问题的根因。

[13:28] Abhilash

So once we build this so maybe like after like 50 plus turns a whole lot of tokens it's able to produce something an analyst uh was able to produce maybe in three or four weeks in like maybe 20 30 minutes. Thank you. So just to wrap it up um key takeaways first thing is I think we should not be introducing human constraints or design constraints into architecture. I think let the architecture be derived actually number one. Second I think any complex

所以我们把这套东西搭起来之后，大概经过 50 多轮、消耗一大堆 token，它能在差不多 20 到 30 分钟里，产出一个分析师可能要花三四周才能做出来的成果。谢谢。那我来简单总结一下，几个关键的 takeaway：第一点，我觉得我们不应该把人为的约束、或者说设计上的约束强加到 architecture 里，我认为应该让 architecture 自然地推导出来，这是第一点。第二，我觉得任何复杂的

[13:58] Abhilash

workflows will have deterministic parts and agentic parts. Don't let agents actually run the deterministic part. Right? So I think we need to break break that off. The third you need to have one agent right which owns the reasoning end to end right this agent can actually take the call to use sub agents tools skills to actually spawn off other other tasks basically right but you need one agent to own the the reasoning and the last

工作流都会有确定性的部分（deterministic）和 agentic 的部分。别让 agent 去跑那些确定性的部分，对吧？所以我觉得我们得把这两块拆开。第三点，你需要有一个 agent 来端到端地掌控整个 reasoning，这个 agent 可以自己决定要不要去调用 sub-agent、tool、skill，去派生出其他的任务，对吧，但你必须有一个 agent 来主导 reasoning。最后一点，

[14:23] Abhilash

probably the most important I think graph cannot be treated just as a lookup player I think graph has to be treated as a control plane which the agent uses to navigate and takes the next decisions basically okay with that I I think thank you.

可能也是最重要的一点：我认为 graph 不能只被当成一个 lookup layer（查询层）来用，graph 必须被当作一个 control plane（控制面），让 agent 用它来做导航、决定下一步的决策。好，那我的分享就到这里，谢谢大家。

[14:39]

Thanks. Thanks for telling me this. [applause] >> [music]

谢谢。谢谢你们的分享。[掌声] [音乐]