Tool, skill, or subagent? Decomposing an agent that outgrew its prompt

频道: Claude
视频: https://www.youtube.com/watch?v=mWvtOHlZM-I&t=48s
原文语言: en
统计: 共 125 轮

[0:00]

[music] >> All right, fantastic. Can everyone hear me? Thumbs up, all good? All right. Everyone, I hope that you have had a fantastic day at Code with Claude London and so far today. My name is Will. I'm on our engineering team at Anthropic. I sit on a team called Applied AI. What that means is I essentially split my time between internal engineering work and time spent building agents with

好，太棒了。大家都能听到我说话吗?能听到就竖个大拇指?好。各位,希望你们今天在 Code with Claude London 过得很愉快。我叫 Will,在 Anthropic 的工程团队工作,具体是 Applied AI 这个团队。这意味着我基本上一半时间做内部的工程工作,另一半时间和客户一起搭建 agent。

[0:44]

customers. So, folks, imagine that you built and shipped an agent to solve a problem. I'm sure that's something that a lot of the folks in this room have actually done. And imagine this that this agent worked fantastic, right? But it works so well that a few weeks after shipping, you were asked to add some additional capability to the agent. A few weeks after that, you received more business

好,各位,设想一下:你搭建了一个 agent 来解决某个问题,并把它上线了。我相信在座很多人其实都干过这种事。再设想一下,这个 agent 跑得特别好,对吧?但正因为它跑得太好了,上线几周之后,有人来找你,让你给这个 agent 再加点新能力。又过了几周,你又收到了新的业务

[1:10]

requirements and you added additional capability. This pattern continued and continued until before you know it, your system prompt had grown to become several hundred lines long. You have dozens of tools and sub-agents that exist for your agent. And because of the complexity, you've started to see regressions in the areas that your agent was previously accelerating in. So, if this is you,

需求,于是你又加了新能力。这个模式不断重复、重复,等你回过神来,你的 system prompt 已经长到好几百行了。你的 agent 挂着几十个 tool 和 subagent。而正是因为这种复杂度,你开始发现:那些原本被 agent 加速的环节,现在反而出现了倒退。所以,如果你正处在这种处境,

[1:36]

you're not alone. We see this type of scenario happen pretty commonly with customers and actually with ourselves included in that. So, within this workshop, we are going to simulate an agent that is essentially grown to a complexity where we start to see degradation in its performance. We're then going to walk through some of the decisions that we as engineers and

你并不孤单。这种情况在客户身上挺常见的,其实我们自己也一样会碰到。所以在这个 workshop 里,我们要模拟一个已经膨胀到一定复杂度、开始出现性能退化的 agent。然后我们会一步步过一遍,作为工程师和

[2:00]

architects make in order to improve the design of our agent to restore the performance that we expect with the additional capability. Specifically, we're going to make some decisions around tools and skills and sub agents.

架构师,我们要做哪些决策,来改进 agent 的设计,在保留新增能力的同时把我们期望的性能找回来。具体来说,我们会围绕 tool、skill 和 subagent 做一些决策。

[2:17]

As we modernize the stack of our agent, we want to make sure that we're using the right agentic primitives at the right time. So, when do you use a tool, when do you use a skill, and when do you use a sub agent? We're going to talk through all of that in this session. As I mentioned, folks, this session will be hands-on. So, let's go ahead and get started. I first want to walk you

在我们给 agent 的技术栈做现代化改造时,我们想确保自己是在对的时机用了对的 agentic 原语(primitive)。那么:什么时候该用 tool?什么时候该用 skill?什么时候该用 subagent?这场分享里我们会把这些都讲透。我前面提到过,各位,这场是动手实操的。那我们就开始吧。我先带你们

[2:39]

through our problem statement in our agent. So, for the purposes of this session, we're going to be focusing on an agent called Stock Pilot. This is an inventory management agent that was designed by and for a mid-size uh uh retailer.

过一遍我们这个 agent 的问题陈述。在这场分享里,我们要聚焦的是一个叫 Stock Pilot 的 agent。这是一个库存管理 agent,是为一家中型零售商量身设计的。

[2:56]

The agent that you see on the screen can do several things. It can flag low levels of stock. It can forecast demand. It can pick suppliers. It can file POs. And ultimately, it can write weekly reports for the employees of this retailer. Now, none of these capabilities are particularly complex on their own. But again, the issue is that we've essentially bolted capabilities

你们在屏幕上看到的这个 agent 能做好几件事。它能标记出库存过低的商品,能做需求预测,能挑选供应商,能开采购单(PO),最后它还能为这家零售商的员工写周报。这些能力单看每一项都不算特别复杂。但问题还是那句话——我们是随着时间推移,把一项项能力硬塞

[3:20]

onto our agent over time without modernizing our architecture. This complexity has started to cause some problems. Let's take a look at the actual architecture today of the agent. Folks, today the agent is facilitated by a single orchestrator. So, you see the Stock Pilot orchestrator sitting at the top of the screen.

到这个 agent 上,却没有同步升级它的架构。这种复杂度开始引发一些问题。我们来看看这个 agent 今天实际的架构长什么样。各位,今天这个 agent 是由一个单一的编排器(orchestrator)来驱动的。你们看到屏幕顶部坐着的就是 Stock Pilot 这个 orchestrator。

[3:41]

The agent has a system prompt, as I mentioned, that's grown to be about 400 lines long. It has 12 different tools. Three of those tools happen to be wrappers around sub agents with completely isolated context windows. So, if you have the repo pulled up, which we'll go into more detail in just a bit, there's an agent that's under a folder called before, which essentially

正如我前面说的,这个 agent 有一个已经长到大约 400 行的 system prompt。它有 12 个不同的 tool。其中 3 个 tool 其实是对 subagent 的封装,这些 subagent 拥有完全隔离的 context window。如果你已经把代码仓库打开了——稍后我们会更详细地讲——里面有个放在 before 文件夹下的 agent,它基本上

[4:05]

walks through this this agent exactly. So again, orchestrator, long system prompt, a lot of tools, we have a lot of sub agents. The result of this is that our evals have started to dip. So, let's imagine how we got here for a moment. Again, like we built that agent up front to solve a really specific problem.

把这个 agent 完完整整地走了一遍。所以再强调一次:一个 orchestrator、一个超长的 system prompt、一大堆 tool、一大堆 subagent。结果就是,我们的 eval 分数开始往下掉。我们花点时间设想一下我们是怎么走到这一步的。同样地,我们最初搭这个 agent,是为了解决一个非常具体的问题。

[4:26]

We received business requirements to say add maybe some forecasting capability to our inventory management agent. So, what we decided to do was essentially just spin up a forecaster as a sub agent. Again, later on we we received more requirements to add report writing capability to our agent. So, we decided to add another sub agent for that report writing capability.

后来我们收到业务需求,比如说要给库存管理 agent 加上一点预测能力。于是我们决定的做法,基本上就是直接起一个预测器(forecaster)当作 subagent。再后来,我们又收到更多需求,要给 agent 加上写报告的能力。于是我们又为写报告这件事加了一个 subagent。

[4:50]

Again, our evals started to dip over time because we added more and more complexity while just bolting this capability on. So, let's take just a little bit of time and talk about evals specifically. For this agent, folks, we have 12 different eval tasks across five different types of graders. So, my colleague gave a talk on evals shortly before this. Evals will have a component

就这样,我们的 eval 分数随着时间不断往下掉,因为我们一边硬塞能力,一边不断堆叠复杂度。那我们就花点时间专门聊聊 eval。各位,对这个 agent,我们一共有 12 个不同的 eval 任务,横跨 5 种不同类型的评分器(grader)。我有位同事在这之前刚做过一场关于 eval 的分享。eval 在这个 workshop 里也会占一部分,

[5:18]

within this workshop, but it won't be the main focus. I'll give you a quick summary of the tactical evals that we're using for this agent. On the left side of the screen, you see some IDs. You see several evals that start with the letter R. This stands for regression. These are more realistic single-turn tasks that we grade the model's capability on. So, imagine I

但不会是重点。我快速给你们总结一下我们给这个 agent 用的这套实操层面的 eval。在屏幕左侧,你能看到一些 ID。你会看到好几个以字母 R 开头的 eval。R 代表 regression(回归)。这些是更贴近真实场景的单轮任务,我们用它们来给模型的能力打分。设想一下,我

[5:41]

give the model a task, the model comprehends that task in the form within the agent, uh calls some tools, and then provides a response back to me. We're essentially evaluating that response. We also have some more complex tasks that we're grading the model on. So, you see those F IDs, the IDs that start with F on the left side of the screen. That stands for failure mode. In this case, we're

给模型一个任务,模型在 agent 里理解这个任务,调用一些 tool,然后给我返回一个回复。我们本质上就是在评估这个回复。我们还有一些更复杂的任务也在给模型打分。你看左侧那些以 F 开头的 ID,F 代表 failure mode(失效模式)。在这种情况下,我们

[6:03]

evaluating the model over a more complicated multi-turn task that we're grading. Now, again, I won't go into evals too specifically. We have a number of different types of graders that are both deterministic and non-deterministic.

评估的是模型在一个更复杂的多轮任务上的表现。再说一遍,我不会把 eval 讲得太细。我们有好几种不同类型的 grader,既有确定性的(deterministic),也有非确定性的(non-deterministic)。

[6:20]

Right? When I talk about not when I talk about deterministic evals, we're grading things like turn count and like latency and like the number of tokens that are used as our agent is completing a particular task and we're tracking those deterministic metrics over time.

对吧?当我说到确定性 eval 时,我们评的是这些东西:轮数(turn count)、延迟(latency),以及 agent 在完成某个特定任务时所消耗的 token 数量。我们会持续追踪这些确定性指标随时间的变化。

[6:37]

We're also using the idea of LLM as a judge to evaluate the non-deterministic characteristics of our agent. So, personality and tone and style and output quality, we're using a non-deterministic grader as a part of our eval to evaluate our agent's non-deterministic characteristics.

我们还用到了「LLM 当裁判」(LLM as a judge)这个思路,来评估 agent 那些非确定性的特征。比如人格、语气、风格和输出质量,我们就用一个非确定性的 grader 作为 eval 的一部分,来评估 agent 这些非确定性的特征。

[6:58]

Now, we're going to run the evals for our agent in just a bit, but when you do, you'll find that the agent is struggling a bit. I'll talk about some of these evals in just a little bit more depth. So, F1 on the screen, third from the bottom, this is essentially simulating a daily low stock sweep. this is an inventory agent. We're simulating our ability to look through all of our

稍后我们就会真的跑一下这个 agent 的 eval,等你跑的时候,会发现这个 agent 表现得有点吃力。我把其中几个 eval 再讲深一点。屏幕上的 F1,从下往上数第三个,它本质上是在模拟一次每日的低库存巡检。这是个库存 agent。我们模拟的是它能不能遍历我们所有的

[7:19]

inventory and pull the low levels of stock. This eval you'll find will actually fail because the agent is going to do the right thing, but it's going to take a very winding path to do so. So, instead of taking the straightest line from point A to point B, the agent is going to take a very inefficient path. It's going to get to the right end, but it's going to fail the eval because it's not

库存,把库存过低的商品挑出来。你会发现这个 eval 其实会失败——因为 agent 最终会把事情做对,但它会绕一条非常曲折的路才做到。也就是说,它不走从 A 点到 B 点的最短直线,而是走了一条非常低效的路。它最后会到达正确的终点,但还是会过不了这个 eval,因为它没达到

[7:41]

at the efficiency that we'd like. F2 on the screen is another eval that you'll see fail. This eval actually evaluates the ordering process under a particular promotion package. This is going to fail because we are using a sub agent for this particular task. The sub agent is actually getting the task right, but there's a communication breakdown between our sub agent and our

我们想要的那个效率。屏幕上的 F2 是另一个你会看到失败的 eval。这个 eval 评估的是在某个特定促销套餐下的下单流程。它会失败,是因为我们在这个特定任务上用了一个 subagent。这个 subagent 其实把任务做对了,但在我们的 subagent 和

[8:07]

orchestrator. This is a really common point of failure that we see when customers have have really complicated systems with a lot of sub agents. It's important to get the communication between your sub agents and your orchestrator just right. In the case of F2 like you see on the screen, this is an eval that's going to fail because we have a breakdown in that communication.

orchestrator 之间出现了沟通断层。这是我们在客户那边看到的一个非常常见的失效点——当客户的系统很复杂、挂着一大堆 subagent 时就容易这样。让你的 subagent 和 orchestrator 之间的沟通恰到好处,这一点非常重要。就像屏幕上 F2 这个例子,它之所以会失败,就是因为这条沟通链路出现了断层。

[8:29]

The last one that I'll highlight that you'll see fails is R8 on the screen. R8 will essentially check the forecasting during a particular promotion month. This eval is also going to fail because we have two different policies that live in very different parts of our system prompt and actually end up contradicting each other. So, I mentioned over time our system prompt has grown. We start to

我要重点讲的最后一个、你也会看到它失败的,是屏幕上的 R8。R8 本质上是检查在某个特定促销月里的预测。这个 eval 同样会失败,原因是我们有两条不同的策略,分别写在 system prompt 里相距很远的两个地方,结果它们彼此矛盾。我前面提过,随着时间推移我们的 system prompt 越长越长,于是开始

[8:53]

have some conflicts and the model gets confused leading towards a failure for this particular eval. Now, in the repo, you'll see it in the read me when we run these evals, you'll see that they're going to pass up front at about 83% which is okay, but if you work in the world of manufacturing, that is not okay. 17% failure is a really expensive failure percentage.

出现一些冲突,模型被搞糊涂了,最终导致这个 eval 失败。在代码仓库里,你会在 README 里看到:当我们跑这些 eval 时,一开始大概能过 83% 左右,这成绩还算凑合,但如果你身处制造业这行,这个成绩可不行。17% 的失败率,是一个代价非常高昂的失败比例。

[9:18]

Now, let's double click on R8 again just so that we can understand a little bit about what's happening behind the scenes. Again, R8 is where we're essentially calculating the forecast during a particular month with a promotion. And so, in my uh on my screen here on the right side where you see kind of the simulated terminal window.

那现在我们再次放大看一下 R8,好让我们稍微理解一点幕后到底发生了什么。再说一遍,R8 就是要在一个带促销的特定月份里计算预测值。在我屏幕右侧,你看到的这个模拟出来的终端窗口里——

[9:39]

Uh within the first block under the commented text, we can see that the agent pulled the right forecasting baseline and also pulled the right promotion multiplier. So, forecasting baseline 12 units a day, promotion multiplier 3.1x.

在注释文本下方的第一个代码块里,我们能看到 agent 拉到了正确的预测基线,也拉到了正确的促销乘数。所以:预测基线是每天 12 个单位,促销乘数是 3.1 倍。

[9:55]

This is all correct. But, in the calculation part below that, we can see that there was actually some kind of hallucination that happened. Instead of using that 3.1x promo multiplier, the agent actually ended up using 1.35.

这些全都是对的。但在下面的计算部分,我们能看到这里其实发生了某种幻觉(hallucination)。agent 没有用那个 3.1 倍的促销乘数,而是用成了 1.35。

[10:09]

So, something happened along the way. A hint here is that the reason for this is that we have context problem. So, this isn't a model problem. It's an issue with our the information that we're surrounding the model with. Our system prompt has grown to be really long and is very confusing for the model and has some conflicts in it, which lead to the issue that shows up within this

所以在这个过程中出了岔子。这里给个提示:出问题的原因是我们有个上下文(context)问题。所以这不是模型本身的问题,而是我们给模型周围铺设的信息出了问题。我们的 system prompt 已经长到非常长,对模型来说非常容易混淆,里面还有一些冲突,这些就导致了这个 eval 里出现的这个问题。

[10:31]

eval. So, folks, our objective in this workshop will first be to run our suite of evals. We're going to triage the issues and we're going to update the design of our agent accordingly. And then, we're going to do something that we call internally hill climbing towards eval improvement.

好,各位,在这个 workshop 里,我们的目标首先是跑一遍我们整套 eval。我们会对这些问题做分诊(triage),然后据此更新 agent 的设计。接着我们要做一件我们内部称之为「爬坡」(hill climbing)的事——朝着 eval 改善的方向往上爬。

[10:50]

Right? So, we run our evals, we get a baseline. It's going to be about 83%. We're then going to optimize the architecture of our agent and we're going to continue then running our eval so that we climb on them, hopefully seeing the success percentage uh improve over time. In this lab, we're also going to start with an agent that is self-created on our messages API. Um

对吧?也就是说,我们先跑 eval,拿到一个基线,大概是 83% 左右。然后我们去优化 agent 的架构,接着不断重复跑 eval,让分数一点点往上爬,希望随着时间推移,通过率能逐步提升。在这个实验里,我们还会从一个直接基于我们 messages API 自建的 agent 开始。嗯——

[11:14]

again, if you have the repo and you click on the before folder, I'll show you this in just a bit. This is an agent that is built from scratch on our messages API. We're going to actually migrate that that agent to Claude managed agents.

再说一遍,如果你拿到了这个 repo,点开 before 文件夹,等下我会演示给你看。这是一个完全从零搭起来、跑在我们的 messages API 上的 agent。我们接下来会把这个 agent 迁移到 Claude managed agents 上。

[11:31]

Claude managed agents essentially allows us to offload the messiness that comes with maintaining an agentic harness in scaling agents safely and securely to thousands and tens of thousands of users, right? Like if I want to build my agent locally and run it locally, I can do that pretty quickly and pretty easily. But the moment that I need to take that agent, I need to host it

Claude managed agents 本质上让我们把维护一套 agent 运行框架时那些麻烦事都甩出去,从而能安全、可靠地把 agent 扩展到成千上万的用户。你想想,如果我只是在本地搭一个 agent、本地跑一跑,那很快很简单就能搞定。但一旦我要把这个 agent 拿出去,我得把它托管起来,

[11:57]

remotely, and I need to allow hundreds and and and and thousands of users to at the same time engage with that agent, there's an infrastructure problem, there's a scaling problem, there's memory, there's security, there's so much that I have to account for. So, in order to offload that, so I can just worry about the architecture of my agent itself and make decisions around tools,

部署到远端,还得让成百上千的用户同时来用这个 agent,这时候就有基础设施的问题、有扩展的问题,还有内存、安全等等一大堆我都得操心的事。所以,为了把这些事甩出去,让我只需要专注于 agent 本身的架构、专心做关于 tool、

[12:21]

skills, and sub-agents, I'm going to offload everything else to Claude managed agents. So, again, to break that down just a bit, um there's been a few talks on CMA so far today, but this is really where we're able to separate the agent from the session details, from the sandboxed environment where tool calls are actually happening.

skill 和 subagent 的决策,我会把其它所有的东西都交给 Claude managed agents 去处理。再稍微展开讲一下,今天已经有几场演讲聊过 CMA 了,但它真正的价值在于:我们能把 agent 本身,跟会话细节、跟 tool 实际执行所在的沙箱环境给分离开。

[12:46]

Um again, this allows us to offload particular parts of the stack to then worry about the to then only worry about the design of our agent itself. All right. I mentioned that we're going to get hands-on in this workshop. We are going to go ahead and do that right now.

再说一遍,这让我们能把整个技术栈里特定的那些部分甩出去,从而只需要操心 agent 本身的设计。好,我之前说过这个 workshop 我们会动手实操,现在我们就来动手。

[13:08]

Now, what you see on the screen here is the workshop URL as well. Um if you haven't had a chance to grab it, feel free to go ahead and do so. Um this is where we're keeping all of the different workshops throughout Code with Claude within London, so you can go back and revisit them if helpful. Within this workshop, we're going to be working on agent decomposition. So, that's going to

你们现在屏幕上看到的也是这个 workshop 的网址。如果还没来得及拿,可以现在就去拿一下。我们把 Code with Claude London 上所有的 workshop 都放在这里了,所以如果有用的话,你之后可以回来再看。这个 workshop 里我们要做的是 agent 拆解(agent decomposition)。所以那也

[13:29]

be the name of the folder that we're actually going to be working within. Great. Let me jump forward here. Perfect. So, the first thing that we're going to do as a part of this workshop is we're first going to get a baseline. So, when you open up that link, you'll first clone the repo. So, we're going to clone the repo locally. We have a UV project that's set up. So, we're

正是我们实际要操作的那个文件夹的名字。好,我往前翻一下。完美。作为这个 workshop 的第一步,我们要先拿到一个基线。你打开那个链接之后,首先要 clone 这个 repo。所以我们先在本地把 repo 克隆下来。我们配好了一个 UV 项目,所以我们

[13:58]

going to run UV sync in order to make sure that we have all of our packages and our dependencies to be able to invoke the Anthropic SDK and then eventually deploy our agent to Claude managed agents. So, we can run UV sync to do that. I mentioned previously that we're going to need an API key for this workshop as well. So, using those credits that you got at the start of

要跑 uv sync,来确保我们装齐了所有的包和依赖,这样才能调用 Anthropic SDK,最终把我们的 agent 部署到 Claude managed agents 上。我们跑一下 uv sync 就行。我前面提过,这个 workshop 还需要一个 API key。所以,用这场开始时发给你的那些

[14:18]

this session, you can go to your Claude console account and create an API key. If you copy the ENV example, you'll just have to manually copy your API key into the ENV file that's created for you. Now, all the 12 evals that I previously walked you through, we have all of those set up already. So, in order to get a baseline and run those evals, you have to run UV run evals {dash} {dash} agent

额度(credits),你可以去你的 Claude 控制台账户里创建一个 API key。把 ENV 示例文件复制一份,然后手动把你的 API key 填进给你生成的那个 ENV 文件里就行。我前面带你们过了一遍那 12 个 eval,我们已经把它们全都配好了。所以,要拿到基线、跑这些 eval,你需要运行 uv run evals --agent

[14:43]

before. This is all in the read me, but if you just run that command, you will be able to actually go about running your evals. Now, in terms of our building here, we're going to take a number of steps to actually go about running our evals using Claude code to triage the results of them and then climbing accordingly on our agent.

before。这些 README 里都写了,但你只要跑这条命令,就能真正开始跑你的 eval 了。在我们这次的构建过程里,我们会分好几步走:用 Claude Code 来跑 eval、分诊(triage)它们的结果,然后相应地一步步往上爬(hill climbing),优化我们的 agent。

[15:09]

So, we're first going to take some We're going to take a look at our the system prompt that we have for our agent itself. So, I mentioned earlier that our system prompt is currently sitting at about 400 lines long. We've been stacking information on our system prompt over and over again as we've continued to get more business requirements. So, our system prompt is

所以我们第一步先看一下我们 agent 本身的 system prompt。我前面提过,我们的 system prompt 现在大概有 400 行那么长。随着业务需求越来越多,我们一遍又一遍地往 system prompt 上堆信息。所以我们的 system prompt

[15:27]

very long. We'll take a look at that. We are then going to take some time to evaluate the tools that we're using. Right now, as I mentioned, we have 12 different tools. Three of them are actually kind of wrapped sub-agents, so we'll take a look to see what we can do to make that more efficient. And then lastly, if there are any sub-agents that we really need to make our agent

非常长。我们会看一下它。接下来我们会花点时间评估一下我们正在用的那些 tool。就像我说的,我们现在有 12 个不同的 tool。其中有三个其实是包装过的 subagent,所以我们会看看能做点什么让它更高效。最后,如果确实有哪些 subagent 是我们为了让 agent

[15:46]

effective, we're going to take a look at the best way um to actually construct sub-agents with Claude managed agents. I'm going to jump back just for a moment. There's one thing that I forgot to mention for you as you get started.

好用而真正需要的,我们会看一下用 Claude managed agents 来搭建 subagent 的最佳方式是什么。我往回跳一下。有件事我忘了在你们开始之前跟你们说。

[16:02]

Um within the repo folder, there's two different uh folders that you'll see. There's a before folder, and then there's a starter folder. Those contain two separate agents. So, if you want to view the messages API version of the agent, again, this is just me building my own agent loop and my own agent harness around the Anthropic messages API to invoke Claude, you'll see that

在这个 repo 文件夹里,你会看到两个不同的文件夹。一个是 before 文件夹,另一个是 starter 文件夹。它们里面装的是两个不同的 agent。如果你想看这个 agent 的 messages API 版本——再说一遍,这就是我自己围绕 Anthropic 的 messages API 搭的一套 agent 循环和 agent 框架,用来调用 Claude——你就能在

[16:27]

within the before folder. If you want to view what that agent looks like when deployed on Claude managed agents, um you can look in the starter folder, which exists right below that. If you want to deploy your agent on Claude managed agents, you can run UV run deploy a starter. So, again, run your evals using the messages API version uh {dash} {dash} agent before,

before 文件夹里看到它。如果你想看这个 agent 部署到 Claude managed agents 上是什么样子,可以看下面紧挨着的 starter 文件夹。如果你想把你的 agent 部署到 Claude managed agents 上,可以运行 uv run deploy starter。所以再说一遍,用 messages API 版本跑你的 eval 用 --agent before,

[16:53]

you can then deploy your agent on Claude managed agents. We already had it built for you, and it's really easy to use Claude code to kind of compare the two um and understand exactly what's going on and what some of the differences are uh with Claude managed agents.

然后你就可以把你的 agent 部署到 Claude managed agents 上。我们已经帮你把它搭好了,而且用 Claude Code 来对比这两个版本特别方便,能让你搞清楚到底发生了什么、Claude managed agents 都有哪些不一样的地方。

[17:09]

Okay, so I'm going to jump over here, and we're just going to open up Claude code, and we are going to build together. I'm I'm to zoom in very far so that you can see everything and so that I can see everything and we'll just talk through exactly what happens when I run some of these evals and we'll talk through the process that we usually go through to do what I just called hill climbing on the

好,我切到这边来,我们打开 Claude Code,一起来动手搭。我会把界面放得很大,好让你们能看清所有东西、也让我自己能看清所有东西,然后我们就边做边讲,看看我跑这些 eval 的时候到底发生了什么,讲讲我们平常对 eval 做我刚才说的那种爬坡优化(hill climbing)时,通常走的是怎样一个流程。

[17:30]

evals themselves. Okay, so if you're looking at Claude code here, again, I just used Claude code to actually run my evals because I want Claude's help in triaging what's going on. Um, so this is me. I'm using Claude code. I have Opus 4.7 running as you can see on the screen. Uh, my effort level is set to extra high. I usually set effort as extra high with Opus 4.7 and I

好,你们现在看着 Claude Code,再说一遍,我就是用 Claude Code 来实际跑我的 eval 的,因为我想让 Claude 帮我分诊到底是哪儿出了问题。这就是我,我在用 Claude Code。你们能在屏幕上看到,我跑的是 Opus 4.7。我的 effort level(努力等级)设成了 extra high。我用 Opus 4.7 的时候一般都把 effort 设成 extra high,然后

[17:54]

forget about it. That's the effort level that I usually stand. We find that it gets great performance um, with extra high effort all together. Now, you can see on the screen the first thing that I did was I ran my eval. So, I used the bash capability in Claude code and I ran UV run evals {dash} {dash} agent before.

就不管它了。那是我通常用的 effort 等级。我们发现整体来说,用 extra high effort 能拿到非常好的表现。好,你们能在屏幕上看到,我做的第一件事就是跑了我的 eval。我用了 Claude Code 里的 bash 能力,跑了 uv run evals --agent before。

[18:14]

Claude actually went ahead and ran my eval. So, I'm going to scroll down and we're going to look at what Claude found while actually running those. So, you can see the response that we got, the results that we got from this eval run was actually lower than what I told you before. So, we ran them and we got 62% which is worse than the 83% that we started with.

Claude 就真的去帮我把 eval 跑了起来。所以我往下滚动一下,我们来看看 Claude 在真正跑这些 eval 的时候发现了什么。你们能看到我们拿到的回复、这一轮 eval 跑出来的结果,其实比我之前跟你们说的还要低。我们跑完得到的是 62%,比我们一开始的 83% 还要差。

[18:36]

So, we passed seven out of 12 of them and it looks like Claude has provided us with a diagnosis for the different evals that we actually failed. Let's scroll down just a bit more. And we are going to use Claude to understand a little bit more about why this actually happened. So, you can see here I am using Claude to provide me some of the themes around why we actually failed some of these

所以我们 12 个里通过了 7 个,而且看起来 Claude 已经针对我们实际没通过的那几个 eval 给出了一份诊断。我们再往下稍微滚一点。我们接下来会用 Claude 来更深入地理解一下这到底是为什么发生的。你们能看到,这里我在用 Claude 帮我归纳出我们实际没通过这几个 eval 背后的一些

[19:09]

evals. Again, this is a a technique if you have evals for your agent. Um again, as Geary showed before this session, you can use Claude actually go about triaging these. So, it looks like there's a few different themes that Claude is figuring out based on this agent. So, the first thing Claude is seeing that our model is taking on a lot of work that it should have tools in

主题(themes)。再说一遍,如果你给你的 agent 准备了 eval,这是个很有用的技巧。就像 Geary 在这场之前演示过的那样,你可以让 Claude 来帮你分诊这些。所以看起来,Claude 基于这个 agent 找出了几个不同的主题。Claude 看到的第一件事是:我们的模型在干很多本该有 tool 去干的活。

[19:29]

order to do. So, our model is doing a lot of reasoning across information that it just doesn't have the tools to be able to complete. It looks like there is some issues that we have with the enforcement of output structure. So, our model and our sub-agents are producing information in a particular output structure that doesn't align um exactly with uh what we're looking for with um

也就是说,我们的模型在对一堆信息做大量的推理,而它根本就没有相应的 tool 能去把这些事完成。看起来我们在输出结构(output structure)的约束这块也有点问题。我们的模型和 subagent 产出的信息,其输出结构跟我们想要的、能从 agent 身上压榨出最佳表现所需要的那个结构,并不完全对得上。

[19:54]

to to pull the best performance from uh from our agent. If I continue to scroll down here, you can see there was some policy issues, etc. Um as I mentioned before, we have a system prompt that's really long right now. Um and so, Claude is seeing some confusions based on the information that's found within the system prompt.

我接着往下滚,你们能看到还有一些策略(policy)方面的问题等等。就像我前面说的,我们现在的 system prompt 真的很长。所以 Claude 看出来,基于 system prompt 里塞的那些信息,产生了一些混淆。

[20:14]

So, again, you can see Claude has found some root causes. Now, we're going to do a few different things here. Again, we're going to go one-by-one and address some of the areas um that we're seeing issues on within our agent. So, I'm going to scroll down here and we are going to use um Claude code to triage some things within our agent. Okay. So, the first thing that

再说一遍,你们能看到 Claude 找出了一些根因(root causes)。现在我们要做几件不同的事。我们会一个一个来,逐一处理我们在 agent 里看到的几个出问题的地方。所以我往下滚动,我们要用 Claude Code 来分诊我们 agent 里的一些东西。好,我要做的第一件事是

[20:36]

I'm going to ask Claude to do, we're going to talk through this. Claude is making some changes, which is great. Um system prompts tend to get very, very long when we accumulate agents over time. So, the first prompt that I ran, if you're following along, feel free to go ahead and do this. I encouraged Claude to look at my agent.py file, which is where our main CMA um agent

我要让 Claude 去做的第一件事——我们边做边讲。Claude 正在做一些改动,这很好。当我们随着时间不断往 agent 上堆东西,system prompt 往往会变得非常非常长。所以我跑的第一个 prompt,如果你在跟着做,欢迎你也照着来。我让 Claude 去看我的 agent.py 文件,那里就是我们主要的 CMA agent

[20:58]

loop is located. Again, that's agent.py. And I essentially said, "Hey Claude, do you have any thoughts on the system prompt? Maybe I can use skills instead of a long-running system prompt for progressive disclosure. So, the first thing that we'll talk about is skills.

循环所在的地方。再说一遍,就是 agent.py。然后我基本上就跟它说:"嘿 Claude,你对这个 system prompt 有什么想法吗?也许我可以用 skill 来做渐进式披露(progressive disclosure),而不是一个长长的 system prompt。" 所以我们要聊的第一个东西就是 skill。

[21:13]

There's been a few other sessions on skills. The short definition that I like to use is that skills are packaged and composable information that Claude has the ability to pull into context whenever Claude realizes that it needs that information to complete a particular task. Right? Skills are really useful with Claude code. Like if you need to provide Claude information

前面已经有几场分享讲过 skill 了。我喜欢用的简短定义是:skill 就是打包好、可组合的信息,Claude 能在它意识到自己需要某些信息来完成某个特定任务时,把这些信息拉进 context。对吧?skill 配合 Claude Code 用起来真的很有用。比如你需要给 Claude 提供一些信息,

[21:37]

on your testing process, or if you want to package up your brand and your UI components, and bundle them into a skill that Claude can pull into context whenever needed, skills are fantastic. Skills are also useful within the agents that you're building for your customers.

讲清楚你的测试流程,或者你想把你的品牌和 UI 组件打包起来,捆成一个 skill,让 Claude 在需要的时候随时拉进 context——这种场景下 skill 棒极了。skill 在你为客户构建的 agent 里同样很有用。

[21:55]

So, if you're building a product, and you are going to give that product to customers, you're building an agent, skills are great within that. In the case of the agent that we have on the screen here, um again, we have a lot of different policies and a lot of procedures that go into our inventory management system. As I accumulated requirements over time, instead of building skills, I decided to

所以,如果你在做一个产品,你要把这个产品交给客户,你在搭一个 agent,那么 skill 在这里面就非常好用。拿我们现在屏幕上这个 agent 来说,再说一遍,我们的库存管理系统里有一大堆不同的策略、一大堆流程规程。随着我把需求一点点堆起来,我当时没有去搭 skill,而是决定

[22:23]

take all of that information and keep appending it to my system prompt. So, my system prompt got longer and longer and longer over time. This is not something that we recommend you do based on the introduction of skills. Right? Leave the system prompt only for the information that Claude needs in its mind, regardless of the task that you give it.

我把所有这些信息全都不断往 system prompt 里追加。于是我的 system prompt 越变越长、越变越长。在引入了 skill 之后,我们其实不建议你这么做。对吧?system prompt 里应该只放那些不管你交给 Claude 什么任务、它都必须时刻记在脑子里的信息。

[22:46]

Skills are fantastic for packaging information that Claude is going to need some of the time, not all of the time. Right? So, if I ask Claude to go build a forecast, Claude is going to um go ahead and do that. Let's see, I lost my computer just for a second. Here we go.

skill 特别适合用来打包那些 Claude 只在某些时候需要、而不是一直都需要的信息。对吧?比如我让 Claude 去做一个预测,Claude 就会去做这件事。我看看——我的电脑刚刚卡了一下,好了,回来了。

[23:05]

If I ask Claude to go ahead and build a forecast, right, Claude is not going to need forecasting information unless I specifically ask it to go ahead and and build that forecast, right? So, in the case of that particular task, I want Claude to pull forecasting information into its context window. Skills are also fantastic for making sure that you are being efficient with context because if

如果我让 Claude 去做一个预测,对吧,那 Claude 平时是用不到预测相关信息的,除非我专门让它去做这个预测。所以针对这种特定任务,我才希望 Claude 把预测相关的信息拉进它的 context window。skill 还有一个特别棒的地方,就是能帮你高效利用 context,因为如果

[23:31]

you stuff all of this information into the system prompt, you're polluting that context window with information that Claude does not need >> [snorts] >> um, in order to complete a particular task. So, again, the first thing that I did, I'll zoom in just a bit more so that you can see this, and I'll scroll up just a bit.

你把所有这些信息一股脑全塞进 system prompt,你就等于在用 Claude 完成某个具体任务时根本不需要的信息去污染那个 context window。所以,我再说一遍,我做的第一件事——我放大一点让你们看清楚,再往上滚一点。

[23:51]

I said, "Hey Claude, can you help me take a look through my system prompt? Can I use skills instead? My system prompt is too long and I need some help." And so, Claude did an analysis of this and realized that I have some pre-built skills that I can use to supplement information in my system prompt. So, the first correction or fix that we're going to make to

我说:嘿 Claude,你能帮我看一遍我的 system prompt 吗?我能不能改用 skill?我的 system prompt 太长了,我需要点帮助。于是 Claude 做了一番分析,发现我有一些现成的 skill 可以用来替代 system prompt 里的部分信息。所以,我们要做的第一个修正、第一个改进,

[24:15]

modernize our architecture here >> [snorts] >> is we are actually going to um, remove uh, many of much of the system prompt and we're going to put that information into skills. And so, you can see here the first thing that we're doing with Claude is we are activating a number of different skills that previously were not there before, and we're actually swapping our system prompt to be a short

为了把这套架构现代化,就是把 system prompt 里很大一部分内容去掉,把这些信息搬进 skill。你可以看到,我们和 Claude 一起做的第一件事,就是激活了一批之前没有的 skill,然后把我们的 system prompt 从长的换成短的。

[24:39]

prompt instead of a long one. So, if you're curious, if you feel like you have a long system prompt within the agents that you're building, feel free to take a look at this to see the differences between what was like a 400-line system prompt compared to about a 50-line system prompt, we've supplemented that and we've switched a lot of that information to skills.

所以如果你好奇,如果你觉得自己在构建的 agent 里 system prompt 太长了,欢迎来看看这个对比:从原来大概 400 行的 system prompt,变成现在大约 50 行的 system prompt,我们做了补充,把其中大量信息都转移到了 skill 里。

[25:00]

Great. I am now going to continue working with Claude. You can see we made those changes here, which is fantastic. There's some evals that I can go rerun. I'm going to ask Claude to do one more thing, and then we're going to we're going to rerun some of our evals to see where we've improved.

很好。接下来我要继续和 Claude 一起干活。你能看到我们刚才做的这些改动,效果很棒。有一些 eval 我可以重新跑一遍。我打算再让 Claude 做一件事,然后我们就重新跑一些 eval,看看我们改进了多少。

[25:17]

So, I mentioned before that we have 12 different tools. You saw those on the screen in the second slide that I shared. As a part of this inventory management agent, we have we have tools that we've created for everything. So, whenever Claude needs to retrieve data, we have a tool. Whenever Claude needs to analyze data, we have a tool for that.

前面我提到过,我们有 12 个不同的 tool。你们在我分享的第二张幻灯片上看到过。作为这个库存管理 agent 的一部分,我们几乎给所有事情都做了对应的 tool。所以每当 Claude 需要取数据,我们就有一个 tool;每当 Claude 需要分析数据,我们也有对应的 tool。

[25:36]

We have tools for everything. So, I'm going to ask Claude to take a look at the tools that my agent has and help me think through how I can optimize here. So, right now Claude is running an analysis across the different tools that I have for my agent.

我们什么都有 tool。所以现在我要让 Claude 看一看我这个 agent 拥有的这些 tool,帮我想想怎么在这里做优化。此刻 Claude 正在对我这个 agent 拥有的各个 tool 做分析。

[25:55]

And we're going to get to see what some of the results for. Now, while this is working, I'll give you a tip when it comes to building agents that we carry with us at Anthropic for our agents internally and the agents built with customers.

等会儿我们就能看到一些结果。趁它还在跑,我先给你们分享一个在构建 agent 时的小窍门,这是我们在 Anthropic 内部构建自己的 agent、以及和客户一起构建 agent 时一直贯彻的原则。

[26:10]

Whenever we build agents, we lean into the same primitives that we as humans have access to. So, imagine yourself when you show up to work, right? You have a computer that's sitting in front of you. You have the ability to navigate files on a file system. You can type in the browser and you can search the web.

每当我们构建 agent 时,我们都会倾向于复用我们作为人类所拥有的那一套基础能力。想象一下你来上班时的样子:你面前摆着一台电脑,你能在文件系统里浏览文件,你能在浏览器里打字,你能上网搜索。

[26:27]

If you're an engineer, you have the ability to write and execute code. When you think about Claude code as an agent, we've effectively given Claude access to all of the same primitives that you and I have access to when we show up to work every single day. Like Claude code is a great coding agent because Claude is really good at code, but essentially what we've done with

如果你是工程师,你还能写代码、跑代码。当你把 Claude Code 看作一个 agent 时,我们其实就是把你我每天上班时拥有的这同一套基础能力,全都交给了 Claude。Claude Code 是个很棒的编程 agent,因为 Claude 本身就很擅长写代码,但本质上我们用 Claude Code 做的事,

[26:48]

Claude code is we've just given Claude access to a computer. Right? And this is really powerful because this allows us to drop in better versions of Claude as we continue to release new models, and Claude just uses those primitives better than it did before.

就是把一台电脑交给了 Claude。对吧?这一点非常强大,因为这意味着每当我们发布更强的新模型,我们就能直接把更好版本的 Claude 替换进来,而 Claude 用这些基础能力的水平也会比以前更好。

[27:07]

Right? Like imagine yourself after this conference compared to yourself when you walked in. You're going to have the same tools at your fingertips, but you're theoretically, your brain's going to be a little bit bigger, you're going to be smarter based on what you learned here, and you're going to be more effective while using the same tools. Claude works the same exact way.

对吧?就好比想象一下这场大会结束后的你,和刚走进会场时的你做对比。你手头能用的工具还是同一批,但理论上你的脑子会变大一点,因为今天学到的东西你会变得更聪明,用同样的工具也能做得更出色。Claude 的运作方式一模一样。

[27:25]

Right? And so whenever we build agents, we lean into human-like primitives first. These primitives are things like code execution and the navigation of a file system, the keeping of a to-do list, the ability to search the web. These are foundational tools that we always start with when we build agents, and we remove them as needed. An example that I like to give

对吧?所以每当我们构建 agent 时,我们都优先采用这些类人的基础能力。这些能力包括:执行代码、在文件系统里导航、维护一份待办清单、上网搜索。这些是我们构建 agent 时永远的起点,是最基础的 tool,然后我们再按需把它们去掉。我喜欢举的一个例子

[27:50]

is with file, uh, like document analysis. If you're building an agent that requires document analysis, maybe you have a lot of CSVs or Excel sheets that your agent is going to be looking over, code execution, so the ability to write and run code is one of the best ways of uh, uh, doing data analysis and working across lots of documents, right? Like if you need Claude to look across the CSV,

是关于文档分析的。如果你在构建一个需要做文档分析的 agent,可能你的 agent 要处理大量 CSV 或 Excel 表格,那么代码执行——也就是写代码、跑代码的能力——就是做数据分析、处理大量文档最好的方式之一。比如你需要 Claude 去通览一堆 CSV,

[28:20]

giving Claude a bash tool so that Claude can write a quick Python script and reason across the results after running that Python script is much more effective than just uploading the entire CSV into Claude's context window.

给 Claude 一个 bash tool,让 Claude 写一段简短的 Python 脚本,跑完后再基于结果去推理,这比直接把整个 CSV 上传到 Claude 的 context window 里要有效得多。

[28:37]

Right? So again, we lean into these, uh, computer-like primitives first when building an agent. So if If scroll down here, that's exactly what we did here. You can see we took a lot of steps and we actually removed most of the tools that exist within our agent and we replaced them with some of the primitives that I talked through previously. This is an inventory management agent that leans

对吧?所以我再说一遍,构建 agent 时我们优先采用这些类似电脑的基础能力。我往下滚一下,这正是我们这里做的事。你可以看到我们做了不少步骤,实际上把 agent 里大部分 tool 都去掉了,换成了我前面讲的那些基础能力。这是一个特别适合这么做的库存管理 agent,

[29:02]

really well to this. Um I have the ability to consolidate and remove a lot of the tools that I'm using to reason across Excels and reason across forecasting data and just give Claude access to the same tools that Claude code has in order to do that. What's cool about this is that when you build using Claude managed agents, these tools are actually included by

用这套思路效果非常好。我可以把大量原本用来处理 Excel、处理预测数据的 tool 整合掉、删掉,直接让 Claude 用上 Claude Code 拥有的那同一套 tool 来干这些活。有意思的是,当你用 Claude managed agents 来构建时,这些 tool 其实是默认就内置好的,

[29:29]

default. So if you want to give Claude access to those same tools that Claude code has and use them to drive powerful capability within your agent, you don't have to worry about writing a tool that gives Claude the ability to write and run code. Or you don't have to write a tool that gives Claude the ability to use the file system. You can just rely on those built-in tools

所以如果你想让 Claude 用上 Claude Code 那同一套 tool,并用它们在你的 agent 里发挥强大能力,你根本不用操心去写一个让 Claude 能写代码、跑代码的 tool,也不用去写一个让 Claude 能用文件系统的 tool。你直接依赖那些内置的 tool 就行,

[29:55]

that we have built ourselves for Claude code that we just make available through Claude managed agents. I'm going to ask Claude to rerun an eval. The evals to see if we are getting better. Now with your agent, there's always going to be the need to add some custom tools as well. Like you're not you're only going to get so far by giving your agent the same tools that we give Claude

这些是我们专门为 Claude Code 打造、并通过 Claude managed agents 直接提供给你的。我现在要让 Claude 重跑一个 eval,看看我们有没有变得更好。当然,对你的 agent 来说,永远都会有需要再加一些自定义 tool 的时候。光靠给你的 agent 套用我们给 Claude Code 的那套 tool,

[30:23]

code. So we always start with those primitives like code execution and web search and to-do lists, etc. We always start there and then we either remove those tools as we don't need them. Right? There might be some agents where we just don't need web search, so we'll go ahead and remove that tool. Um Um and then we'll add custom tools whenever we need them, right? So again,

你能走的路是有限的。所以我们总是从那些基础能力起步,比如代码执行、网页搜索、待办清单等等。我们永远从这里开始,然后要么把不需要的 tool 去掉——有些 agent 可能根本用不到网页搜索,那我们就把那个 tool 删掉——要么在需要时加上自定义 tool。所以再说一遍,

[30:48]

when you think about tools, I encourage you to start with those Claude code primitives, those human-like primitives, and then add custom tools only as you need them. In the case of this specific inventory agent, um we were able to remove most of the tools and replace them with Claude codes. So you can see right now, Claude is redeploying my agent to Claude managed agents. So

当你在考虑 tool 的时候,我建议你先从 Claude Code 的那些基础能力、那些类人的基础能力起步,然后只在确实需要时才加自定义 tool。就这个具体的库存 agent 而言,我们能够把大部分 tool 都删掉,换成 Claude Code 的那套。所以你现在能看到,Claude 正在把我的 agent 重新部署到 Claude managed agents 上。

[31:12]

again, I have my agent locally. I am redeploying it based on some of the changes that we've made, and now I can rerun some of my evals to see the result. So you can see in that last command, I'm rerunning uh the F1 eval, and we're going to see what happens as a result.

再说一遍,我的 agent 是在本地的,我正基于我们刚做的一些改动把它重新部署上去,然后我就能重跑一些 eval 看结果了。你可以看到在最后那条命令里,我正在重跑 F1 这个 eval,我们来看看跑出来会是什么结果。

[31:31]

Now, we always get a lot of questions when it comes to MCP. So in the case of CMA here, you have a couple different options when it comes to tools. You can first lean on those Claude code primitives, things like web search and code execution and file system. Again, that's what we start with. You can then create uh just custom tools, so standalone tools that only your agent

我们关于 MCP 总会收到很多提问。就拿这里的 CMA 来说,在 tool 这件事上你有好几种不同选择。你可以先依靠 Claude Code 的那些基础能力,比如网页搜索、代码执行、文件系统。再说一遍,这是我们的起点。然后你可以创建自定义 tool,也就是只有你的 agent 才能用的独立 tool。

[31:52]

has the ability to use. Then you can connect your agent to MCP. We see a lot of folks run towards MCP first, and a lot of our customers end up in this ecosystem where there's a lot of kind of chaotic MCP servers that exist.

再然后,你可以把你的 agent 接到 MCP 上。我们看到很多人一上来就直奔 MCP,结果不少客户最后陷进了这样一种生态:存在着一大堆杂乱无章的 MCP server。

[32:08]

A lot of times they have overlap, um which can create some problems. So when we build agents, again, we start with those Claude code tools. We then create local tools only for our agent. We don't run to MCP. And then only in the case where we have a common collection of tools that multiple clients will benefit from accessing, do we go about the process of collecting

这些 server 很多时候功能相互重叠,这会带来一些问题。所以我们构建 agent 时,再说一遍,先从 Claude Code 的那些 tool 起步,然后只为自己的 agent 创建本地 tool,我们不会一上来就奔向 MCP。只有当我们有一批通用的 tool、而多个客户端都能从访问它们中受益时,我们才会着手去把

[32:32]

those and publishing them as an MCP server. So only when we have multiple agents, maybe multiple Claude code clients that need to access the same set of standardized and governed tools, we run towards MCP.

这些 tool 收集起来、发布成一个 MCP server。所以只有当我们有多个 agent、或者多个 Claude Code 客户端,都需要访问同一套标准化、受治理的 tool 时,我们才会奔向 MCP。

[32:48]

Something else that's becoming increasingly common throughout the industry is leaning on Claude's ability to effectively use code execution as a means of executing tools. So, we see a lot of capabilities coming out around just giving Claude access to uh use CLIs and invoke APIs using code and actually run tools using code instead of MCP. One of the drawbacks of MCP is that it does

还有一件事在整个行业里正变得越来越常见,那就是借助 Claude 的能力,直接用代码执行来作为调用 tool 的方式。我们看到很多新能力涌现出来,核心就是让 Claude 直接去用 CLI、用代码调用 API,真正用代码来运行 tool,而不是走 MCP。MCP 的一个缺点在于,它确实

[33:16]

um cause some uh it can cause some context issues just in terms of polluting context and taking up a lot of space. So, there may be some cases where you can just rely on code execution either through CLIs or just by giving Claude the ability to invoke APIs using code as a means of um creating more flexibility for your agent where you do not have to use MCP. So, just something

因为它会带来一些 context 上的问题——会污染 context、占用大量空间。所以在某些场景下,你完全可以靠代码执行来搞定:要么通过 CLI,要么直接让 Claude 用代码去调用 API,以此给你的 agent 创造更多灵活性,这样就不必非得用 MCP。这点

[33:42]

to keep in mind as you're building. Great. Okay, so Claude just got done. Um looks like we have the before and after from some of the changes that we've made and I think that this is pretty compelling, right?

在你构建的时候值得记在心里。好,Claude 这边刚跑完。可以看到我们做的这些改动有了前后对比,我觉得这个对比相当有说服力,对吧?

[33:56]

The first thing that jumps off the screen to me is the token usage. So, before I was using over 200,000 tokens for a particular task, after leaning in on some of those file system primitives, you can see that that went down dramatically. This is a direct result of giving my agent code execution. So again, imagine instead of giving my agent a full CSV that needs to be read

第一个最抓我眼球的就是 token 用量。改动之前,某个任务我要用掉超过 20 万个 token;而在用上这些文件系统原语之后,可以看到这个数字大幅下降。这是给我的 agent 加上代码执行能力的直接结果。再想象一下,与其把一整个需要读进 context 的 CSV 喂给 agent,

[34:21]

into context, I just give my agent the ability to write and run Python as a means of kind of navigating across all of that information. The agent uses a lot less tokens when it can write code and then run code and then read the results instead of having to consume all of that data in Claude's mind and then use all of that kind of collective brainpower to then make decisions based

我只要给它写和运行 Python 的能力,让它以这种方式去穿梭于所有这些信息之中就行。当 agent 能写代码、跑代码、然后读结果时,它用的 token 要少得多——而不是把所有数据都塞进 Claude 的脑子里,再调动全部脑力去基于这些结果做决策。

[34:44]

on the results. Um a few other things. Um we can see that our cost went down as well because we're just not using as many tokens, which makes sense. Um our our our execution time went down as well. So, this was a pretty good case where I think we got better overall, but this is not something that will happen all the time, right? We like we might see some cases where we regress, but

还有几点。可以看到我们的成本也下来了,因为用的 token 少了嘛,这很合理。我们的执行时间也缩短了。所以这是一个相当不错的例子,整体上我们做得更好了。但这种事不会每次都发生,对吧?有些情况下我们可能反而会退步。不过

[35:06]

this was the case where using some of those primitives as opposed to some of our more stagnant tools was clearly the uh the right decision. Great. Okay, we're going to jump back and we're going to talk about subagents for just a bit.

这个例子里,用这些原语来取代那些更死板的 tool,显然是正确的决定。好。接下来我们往回跳一下,稍微聊聊 subagent。

[35:23]

I'm going to copy another prompt to Claude and we are going to um investigate subagents. Now, I mentioned before that we had 12 different tools. Three of them were effectively wrapping subagents. So, if I'm Claude, I have the ability to call on a tool, that tool's a wrapper for a subagent. I can then go and invoke that subagent.

我再复制一段 prompt 给 Claude,我们来研究一下 subagent。前面我提到过我们有 12 个不同的 tool,其中三个本质上就是在包装 subagent。也就是说,如果我是 Claude,我有能力去调用一个 tool,而这个 tool 是某个 subagent 的封装,我接着就能去调起那个 subagent。

[35:45]

See, Claude's doing a lot here. I'll scroll up just a bit and then we'll talk through it. The two main use cases where or the two main instances where we see subagents initially as being really effective is first when you want to throw a lot of Claude at a problem, right? So, let's say that you're trying to do deep research or like web search.

你看,Claude 在这儿干了不少活。我往上稍微滚一点,然后我们来过一遍。我们最初看到 subagent 真正有效的两个主要用例:第一个,是当你想用大量 Claude 的算力去攻一个问题的时候,对吧?比如说你要做 deep research,或者网页搜索。

[36:04]

Um let's say that you're trying to do in the case of Claude code code-based exploration. That's a great case where like having many different minds running at the same problem makes sense. So, subagents are a great way to parallelize and throw a lot of Claude at a problem to get it done faster and more effectively.

再比如说在 Claude code 这个场景里,你要做代码库探索。这就是一个绝佳的例子——让很多个大脑同时去解同一个问题是说得通的。所以 subagent 是一种很好的方式,可以并行化、用大量 Claude 去攻一个问题,从而更快、更高效地完成它。

[36:23]

The second case where it's really common to use subagents is when you need a fresh mind to look at a problem. So, I use the Claude code example first. If I'll use my example as a developer, if I am writing code, I do not want to be the same person that is writing and also reviewing my code. I'm going to have somebody else review my code. So, in the case of Claude code, it makes a lot of

第二个常用 subagent 的场景,是当你需要一个全新的大脑来审视一个问题的时候。我先用了 Claude code 的例子,这次我用我自己作为开发者的例子:如果我在写代码,我可不想让写代码的人同时也是审我代码的人。我会找别人来 review 我的代码。所以在 Claude code 的场景里,很合理的做法是

[36:47]

sense to have one instance of Claude doing the writing of the code and then another instance of Claude coming over the top and reviewing that that does not have context about the initial instance of Claude. This is a great case for a subagent. Using just a code review subagent and layering it over the top is a great way to do this.

让一个 Claude 实例负责写代码,然后让另一个 Claude 实例凌驾其上来做 review——这个实例不带有第一个 Claude 实例的 context。这就是 subagent 的绝佳用例。用一个专门的代码 review subagent 叠在上层,就是实现这个目的的好办法。

[37:07]

We also have a subagent within our our agent here, our inventory management agent, that we've actually kept as the result of of some of the changes that we made and that's for forecasting specifically. So, again, I have a forecasting capability that's within my inventory management agent. I do want to keep my forecasting separate from my main instance of Claude. I don't want anything in my

在我们这个 agent 里,也就是这个库存管理 agent 里,我们也保留了一个 subagent——这是我们做完那些改动后特意留下来的,专门用于预测。所以再说一遍,我有一个预测能力内置在库存管理 agent 里。我确实想让预测和我的主 Claude 实例分开。我不希望我

[37:36]

initial context window to distort the forecasting process. I do have a skill that kind of walks through the step-by-step sequence and the guidelines that I prefer Claude use when writing and building forecasts, but again, I don't want the same Claude that I'm that my customer is talking with to also be the Claude that writes the forecast, right? So, I want to divide that. So,

初始 context window 里的任何东西去干扰预测过程。我确实有一个 skill,它把我希望 Claude 在写预测、做预测时遵循的步骤顺序和准则一步步走了一遍。但同样地,我不想让那个正在和我客户对话的 Claude,同时也是那个写预测的 Claude,对吧?所以我想把它们分开。所以

[37:58]

I'm leaning on that second example of when to use a subagent as the place where we'd like to go about doing this. So, in this case we've removed our other subagents and we've just replaced them with primitive tools, but we are going to leave the forecasting subagent. Now, we're not going to expose our subagent as a tool.

我这里依据的就是「何时该用 subagent」的第二个例子,把它当作我们想这么做的地方。所以在这个案例里,我们把其他的 subagent 都去掉了,换成了原语级的 tool,但我们会把预测 subagent 留下来。注意,我们不会把这个 subagent 暴露成一个 tool。

[38:21]

Using Claude managed agents, there's a native capability for sub agents um that allows the logging and the observability of your sub agents um to be really effective. One of the problems with sub agents is that when you have multiple instances of Claude running, first off, it's difficult to make sure that the communication between your orchestrators and your sub agents is

用 Claude managed agents 的话,它有一个针对 subagent 的原生能力,能让你对 subagent 的日志记录和可观测性做得非常到位。subagent 的一个难题在于,当你有多个 Claude 实例在跑的时候,首先,你很难保证编排器和 subagent 之间的通信是

[38:45]

accurate and is seamless, right? There's a lot that can get lost in translation. Just like when I'm talking to one of my colleagues, I might be thinking something, they might be interpreting it completely differently. The same thing happens with orchestrators and sub agents.

准确且无缝的,对吧?中间有很多东西会在「翻译」过程中丢失。就像我跟同事说话一样,我心里想的是一回事,他们理解到的可能完全是另一回事。编排器和 subagent 之间也会发生同样的事。

[38:59]

The um the other thing that can happen is logging is really difficult in some cases, right? Because then you have to worry about collecting the transcripts from multiple different agents. So, within Claude managed agents, we've added this native sub agent capability.

另一件可能发生的事是,在某些情况下日志记录非常难搞,对吧?因为你得操心怎么从多个不同的 agent 里把各自的对话记录收集起来。所以在 Claude managed agents 里,我们加进了这个原生的 subagent 能力。

[39:16]

I saw it on here. Let me scroll up just a bit. I think Claude found it. Yes. So, there's this callable agents capability that exists within Claude managed agents, which is essentially just like managed sub agents, so that within your session information, you have observability and metrics about what exactly your sub agents are doing that is as accurate as your initial

我在这儿看到它了。让我往上稍微滚一点。我想 Claude 找到了。对。Claude managed agents 里存在一个叫「可调用 agent」(callable agents)的能力,本质上就像是被托管的 subagent,这样在你的会话信息里,你就能拥有关于 subagent 到底在干什么的可观测性和指标,而且其准确度和你那个初始的

[39:39]

orchestrator. Right? Um this is again meant to solve one of the common problems of just having uh a lot of information that is hard to track with sub agents. We did some building. Again, I'm going to skip through these because we spent some time talking about them. We just talked about sub agents. Again, there's a few different cases where you can use them. We just talked about callable

编排器一样高。对吧?这同样是为了解决一个常见难题:subagent 会产生大量难以追踪的信息。我们做了一些搭建。我还是会跳过这些,因为我们前面已经花了时间讲过它们了。我们刚聊完 subagent。再说一遍,有几种不同的场景可以用到它们。我们刚讲了可调用

[40:00]

agents. You can also just define your sub agent as a tool, which is what we did previously, but we actually moved away from that and we decided to use the CMA native capability. Um there are a lot of cases where you can just now scrap the sub agent entirely and just give more flexibility and capability to your main agent. So, what we have a lot of customers doing is

agent。你也可以把你的 subagent 直接定义成一个 tool——这正是我们之前的做法,但我们后来放弃了那种方式,决定改用 CMA 的原生能力。还有很多场景下,你现在完全可以把 subagent 整个砍掉,转而给你的主 agent 更多的灵活性和能力。所以我们看到很多客户实际在做的是

[40:20]

actually just consuming capability into their main, in this case, orchestrator, because frontier models have gotten intelligent enough to manage across more information where you just don't need as many sub-agents. So, again, when you're thinking sub-agent, I had a lot of cla- or I have a big problem that I want to throw a lot of Claude at, or I want a separate Claude to kind of look at um

干脆把能力吸收进它们的主体——在这个案例里就是编排器——因为前沿模型已经变得足够聪明,能够跨越更多信息进行管理,你根本不需要那么多 subagent 了。所以再说一遍,当你想到 subagent 时:我有一个大问题,想用大量 Claude 算力去攻它;或者我想要一个独立的 Claude 来审视

[40:41]

the work of either me or of a different instance of Claude. Two great times to use sub-agents. Okay. So, let's look at the architecture that we ended with. Again, refreshing us, we started with an orchestrator system prompt of about 400 lines long.

我自己、或另一个 Claude 实例所做的工作。这是用 subagent 的两个绝佳时机。好。那我们来看看我们最终落地的架构。再帮大家回顾一下,我们一开始的编排器 system prompt 大约有 400 行那么长。

[40:59]

We had 12 tools, three of them were sub-agents. What did we end with after this exercise? We still have an orchestrator, but we deployed that on Claude managed agents because I didn't want to have to worry about infrastructure, scaling, security, etc. I just wanted to worry about my agent, right? Like in in Will's simple terms, like that is when I reach for Claude managed agents because I just

我们有 12 个 tool,其中三个是 subagent。那这一通操作下来我们最终是什么样?我们仍然有一个编排器,但我们把它部署在了 Claude managed agents 上,因为我不想去操心基础设施、扩容、安全这些事。我只想操心我的 agent,对吧?用 Will 我自己的大白话说,这就是我会拿起 Claude managed agents 的时刻,因为我只

[41:20]

want to worry about building the best thing possible and not all the messiness that comes with it. We simplified our tools. We now have uh we have right now three different tools. So, we actually simplified everything to just use bash, read, and write.

想专注于打造尽可能好的东西,而不去管随之而来的那一堆烂摊子。我们简化了 tool。我们现在有三个不同的 tool。其实我们把一切都简化成了只用 bash、read 和 write。

[41:35]

[snorts] >> Now, when our agent starts executing, we sync some data into the Claude managed agent's environment so that it can reason across that data. We actually simplified our system prompt to 15 lines long, and we replaced all of our business logic with skills. So, again, I was just stuffing requirement after requirement into my system prompt.

现在,当我们的 agent 开始执行时,我们会把一些数据同步进 Claude managed agent 的环境里,好让它能跨着这些数据进行推理。我们实际上把 system prompt 简化到了只有 15 行,并且把所有业务逻辑都换成了 skill。所以再说一遍,我之前是把一条又一条需求往 system prompt 里硬塞。

[41:56]

I decided to take that, package it up as skills so that Claude could pull that information into its brain only when Claude realized that it needed it in order to solve a problem. As a result of this, we showed how we can then start hill climbing on evals to see improvements over time.

我决定把那些东西拿出来,打包成 skill,这样 Claude 只在它意识到解决某个问题需要时,才把这些信息拉进自己的脑子里。这么做的结果是,我们展示了接下来怎么在 eval 上「爬坡」,看到随时间推移的改进。

[42:13]

So at the end of this, my eval score is about 92%. I've simplified my design. I'm leaning into some of the primitives um that make uh Claude great um and I'm seeing the positive results after doing so.

所以到最后,我的 eval 得分大约是 92%。我简化了我的设计,我用上了一些让 Claude 出彩的原语,而且在这么做之后我看到了正向的结果。

[42:31]

Again, some of the eval results, you see that here after running this, um we're getting faster, we're using fewer tokens because we're leaning into code execution. Um our turn count is remaining sort of the same, but again, because the token usage and the cost is going down, I'm actually okay with Claude taking more turns. There are some cases where we'll see the latency not

再看看其中一些 eval 结果,你在这儿能看到,跑完之后我们变得更快了,token 用得更少了,因为我们用上了代码执行。我们的轮次数(turn count)基本保持不变,但因为 token 用量和成本都在下降,我其实是可以接受 Claude 多花几轮的。有些情况下我们会看到延迟并没有

[42:54]

drop maybe as much as you would expect, but for some of these more sophisticated high-intelligence agents where like forecasting is at play, I'm willing to take a little bit higher latency um at the expense of seeing my performance improve and my costs go down.

像你预期的那样降那么多,但对于这些更复杂、更高智能的 agent 来说——比如涉及预测的场景——我愿意承受稍微高一点的延迟,换来的是看到我的性能提升、成本下降。

[43:10]

All right, let's wrap with some some takeaways here in our last minute. When we build agents, we start with a single agent loop that it that is equipped with very simple primitives that give Claude some of these human-like capabilities like the ability to use the file system like the one that you have on your computer, web search, code execution, um sometimes a to-do list. Again, we start

好,在最后这一分钟里,我们用几条要点来收尾。我们构建 agent 时,起点是一个单一的 agent 循环,它配备了非常简单的原语,给 Claude 赋予一些类人的能力:比如使用文件系统(就像你电脑上那个一样)、网页搜索、代码执行,有时还有一个待办清单。再说一遍,我们的起点是

[43:38]

there and then we build accordingly. The next thing that we did is we use progressive disclosure through skills. Instead of stuffing our system prompt with a lot of information, we made information accessible to Claude whenever Claude realized that it needed that information in order to solve a problem. This is great because we can run more efficiently and uh we're not polluting our context

然后我们再据此来构建。我们做的下一件事,是通过 skill 实现渐进式信息披露。我们没有把一大堆信息一股脑塞进 system prompt,而是让这些信息变得可以被 Claude 随取随用——只在 Claude 意识到自己需要某条信息来解决问题时才去取用它。这样做的好处很大,因为我们的运行效率更高,也不会去污染我们的 context

[44:00]

window um and we're giving Claude more flexibility to make decisions. The last thing that I want you to walk away with, write evals in general. This idea of hill climbing is a concept that we lean on really heavily at Anthropic. Right, you have evals as you establish a baseline, you then tweak your architecture and you rerun evals and you get better over time. Now, as a result,

窗口,同时还给了 Claude 更大的决策灵活性。最后我想让大家记住的一点是:无论如何都要写 evals。这种「爬坡式优化」的思路,是我们在 Anthropic 非常依赖的一个理念。对吧,你先用 evals 建立一个基线,然后调整你的架构,再重新跑一遍 evals,这样你就能不断变好。而正因为如此,

[44:23]

it's important to make sure that your evals are updated as your product capability expands. Always make sure that your evals are encompassing the things that you care about and that you're measuring within your agent so that you can actually make sure that your agent is accomplishing the thing that you set out to accomplish.

很重要的一点是:随着你的产品能力不断扩展,要确保你的 evals 也同步更新。一定要让你的 evals 始终覆盖你真正在意、并且会在 agent 里去衡量的那些东西,这样你才能真正确认你的 agent 确实完成了你最初设定要完成的目标。

[44:41]

With that, folks, we're going to go ahead and wrap. I really appreciate your time today. I'll be in the back after the session just outside of this room in case you have any questions at all. Thank you for spending your day at Code with Claude in London. I hope you have a great rest of your day. Appreciate it.

那么各位,我们就讲到这里吧。非常感谢大家今天抽出时间。散场之后我会待在后面、就在这个房间外面,有任何问题都可以来找我。感谢你们把这一天的时间花在伦敦的 Code with Claude 上,祝你们接下来的一天过得愉快。谢谢大家。

[44:58]

[music]

[音乐]