The prompting playbook

频道: Claude
视频: https://www.youtube.com/watch?v=G2B0YWuJUgI
原文语言: en
统计: 共 81 轮

[0:20]

Hello everyone. Um, thank you so much for joining me this afternoon in the breakout room. The last session today of code with Claude. I hope you've all had a fantastic day so far. My name is Margot Vanlar. I am an applied AI engineer at Anthropic here in London.

大家好。非常感谢各位下午抽空来分会场听我讲。这是今天 Code with Claude 大会的最后一场了。希望大家今天过得都很愉快。我叫 Margot Vanlar,是 Anthropic 伦敦办公室的一名应用 AI 工程师。

[0:38]

And this afternoon, we're going to be talking about the prompting playbook. And prompting is arguably one of the first skills, if not the first skill that we had to learn as engineers when we first started to work with LLMs. and even now it continues to be one of the most critical um skills to building effective AI systems. So today we're going to discuss some best practices

今天下午,我们要聊的主题是「提示工程实战手册」。可以说,prompting 是我们当工程师、刚开始接触 LLM 时学的第一批技能之一,甚至可能就是第一个。直到今天,它依然是构建高效 AI 系统最关键的技能之一。所以今天我们会一起过一遍一些最佳实践。

[1:05]

um in the context of two practical scenarios that you're probably encountering at work. The first is where you have an existing prompt in production that you've been maintaining for some time um and possibly you're migrating it to a new model or making a change to the architecture and for some reason it's no longer working as well.

我会结合两个你们工作中很可能会遇到的实际场景来讲。第一个场景是:你手头有一个已经上线、维护了一段时间的 prompt,现在你可能要把它迁移到新模型上,或者改了改架构,结果不知道为什么,它的表现突然就不如以前了。

[1:28]

The second scenario is where we're building an entirely new agentic use case from the ground up and we need to build the prompt from zero to one. Now, in order to illustrate these best practices, I don't just want to give you a list of dos and don'ts. I want to walk through a practical example that's been inspired by real prompts that um I've seen some of our customers work with who

第二个场景是:我们要从零开始搭一个全新的 agent 用例,得把 prompt 从无到有地写出来。为了把这些最佳实践讲清楚,我不想只是甩给你们一份「该做什么、不该做什么」的清单。我想带大家走一个实际的例子,这个例子是受我见过的一些客户真实使用的 prompt 启发而来的——

[1:54]

are building on Claude. So, the prompt that we'll look at today is a miniaturized example. the prompts that you're working with are probably a lot longer and more complex than the one we'll see today. Um, but it's representative of some common problems that you might encounter when maintaining a prompt. So, imagine that we have a prompt that multiple people have been collaborating on, contributing

这些客户都是在 Claude 上做开发的。我们今天看的这个 prompt 是个精简版的例子。你们实际在用的 prompt 大概率比今天这个要长得多、复杂得多。但它很有代表性,涵盖了维护 prompt 时常见的一些问题。设想一下:我们有一个 prompt,好几个人一起在上面协作、添砖加瓦,

[2:21]

to. There's no clear owner. It covers a lot of different areas like policy, like tone, processes. um we have some patches for kind of previous models that we've migrated to all mixed together. Um it's built up and it's complex. And when we're migrating to a new model, we're finding that suddenly a lot of our test cases are no longer working as well as we expected. So what's actually going on

却没有一个明确的负责人。它涉及的内容五花八门——政策、语气、流程都有。我们还为之前迁移过的几个老模型打了些补丁,全都混在一起。它就这么越堆越厚、越来越复杂。等我们要迁移到新模型时,突然发现一大堆测试用例的表现都不如预期了。那这到底是怎么回事?

[2:51]

here? Well, in order to start unpacking that question, um we need a starting point. And that starting point is evaluations. We need evaluations to provide that rigor um to understand whether a change to our prompt is actually correlating to an improvement in its performance.

要开始拆解这个问题,我们得有个起点。这个起点就是 eval(评估)。我们需要 eval 来提供那种严谨性,好让我们搞清楚:对 prompt 做的某个改动,是不是真的对应着性能的提升。

[3:12]

And we have different models which have different capabilities and different behaviors. And when you migrate to a different model, it could be that your system is no longer working as well for two reasons. First of all, if um the new model might be capable, but it's behaving differently and therefore we can tune our prompting to fix that behavior. The second case is where

不同的模型有不同的能力、不同的行为方式。当你迁移到另一个模型时,系统表现变差可能有两个原因。第一,新模型其实是有能力的,只是行为方式变了,那我们就可以通过调 prompt 来纠正这个行为。第二种情况是,

[3:38]

actually the model that we're changing to isn't as capable and no amount of prompting is going to fix that. So we need to have an eval suite to act as a way of testing that regression so that we can apply our prompting best practices to that.

我们要换过去的这个模型本身能力就不够,那再怎么调 prompt 也救不回来。所以我们需要一套 eval 来充当回归测试的手段,这样才能把我们的 prompting 最佳实践应用上去。

[3:56]

So in the example that we're going to be looking at today, as I said, it's going to be a miniaturized example. We'll have five test cases in our eval. In reality, you'll have a lot more test cases in your EVA suite, but the key thing here is that it's representative of three key cases that we need to cover. Those three key cases include having a control case, which is a case

在今天要看的这个例子里,就像我刚说的,它是个精简版。我们的 eval 里只有五个测试用例。现实中你的 eval 套件里会有多得多的测试用例,但关键在于,这五个用例很有代表性,覆盖了我们必须照顾到的三类关键情况。这三类里,第一类是「对照用例」(control case),

[4:22]

which should always pass. It's something that the we know the model handles well. It's unambiguous. The second is edge cases. And these are cases where we've seen the model fail before. And by including instructions into the prompt, we're making sure that same behavior doesn't slip through again in the future. And finally, and critically, we need to make sure that

也就是那种应该永远能通过的用例。它是我们知道模型处理得很好、毫无歧义的东西。第二类是「边缘用例」(edge cases),也就是我们之前见过模型在这上面翻过车的情况。通过把相应的指令写进 prompt 里,我们确保同样的问题以后不会再溜过去。最后,也很关键的一点是,我们要确保

[4:47]

the model has a good understanding of the extent of its capabilities, where it should be handing off to a human or where actually maybe it should be point blank refusing to answer a request. So, in the example that we're going to be looking at today, um we'll be using a um prompt for a customer support bot for a telco company called Meridian Mobile.

模型对自己能力的边界有清晰的认识——它什么时候该把问题转交给人工处理,或者什么时候干脆就该直接拒绝回答某个请求。在今天要看的这个例子里,我们用的是一个客服机器人的 prompt,服务对象是一家叫 Meridian Mobile 的电信公司。

[5:14]

And these are the five test cases that we are going to be looking at today. We have a simple control case looking at um you know what's the data limit in the basic plan. Uh we're also looking at edge cases such as its ability to do calculations such as calculating proration bills. If I switch my bill halfway through the month uh or if I switch my plan halfway through the

这就是我们今天要看的五个测试用例。第一个是简单的对照用例,比如问「基础套餐的流量上限是多少」。我们还看了一些边缘用例,比如它做计算的能力——像按比例计算账单(proration)。如果我月中换了套餐,我的账单会怎么算?

[5:40]

month, what will my bill look like? We want to check that it's accurately addressing key questions which are covered by our policy. um we need to make sure that it's escalating to a human whenever there is um a billing error. Um and finally, we want to make sure that our model isn't withholding any information that it has access to, which it should be handing

月中换套餐的话,我的账单会变成什么样?我们想检验它能不能准确回答那些被我们政策覆盖到的关键问题。我们要确保它在遇到账单错误时会及时转交给人工。最后,我们想确认模型没有把它本该提供给客户的、它手头有权访问的信息给藏着掖着。

[6:04]

over to the customer. So what we're going to do in this process is we'll take our prompt and we'll run it on our v0ero um of of the eval and we'll see what our failure modes are and systematically target those failure modes one at a time to see if we can resolve those failure modes by prompting and along the way we'll learn a little bit more about the kind of

所以接下来我们要做的是:拿我们的 prompt,在 eval 的第 0 版(v0)上跑一遍,看看有哪些失败模式,然后系统性地一个一个去攻克这些失败模式,看能不能靠 prompting 把它们解决掉。这一路上,我们还会顺带多了解一些

[6:28]

antiatterns uh um and traps to avoid and this is representative of how we would apply apply these best prompting techniques in practice. Right? We are rarely writing a prompt from scratch. We're often debugging an existing prompt.

需要避开的反模式(antipatterns)和陷阱。这个过程很能代表我们在实践中是怎么运用这些 prompting 最佳实践的——对吧,我们很少从零写一个 prompt,更多时候是在调试一个已有的 prompt。

[6:46]

And best practice before we start targeting those failure modes specifically is to kind of apply our general prompting 101 best practices, applying general hygiene to clean up before we do the eval run. So let's have a little look at the example that we're going to be using. So what we're looking at here first of all before we look at the prompt is just this vibecoded web

在开始专门攻克这些失败模式之前,最佳做法是先把我们最基础的「prompting 入门级」最佳实践套上去,做一遍通用的「卫生清理」,在跑 eval 之前先把东西收拾干净。我们来看看今天要用的这个例子。在看 prompt 之前,我们先看的是这个我「vibe coding」(凭感觉随手撸出来)的网页应用,

[7:10]

app that I've made for the presentation today so that we can look at how we're iterating on the prompt together. Uh um in this page here I can easily run my evals on all five test cases and inspect the results in a little bit more detail.

这是我为今天的演示做的,这样我们就能一起看到我们是怎么一轮一轮迭代这个 prompt 的。在这个页面上,我可以很方便地在全部五个测试用例上跑 eval,并稍微深入地检查一下结果。

[7:28]

So before we have a look at the prompt I'm just going to run the evals in the background. This is a pretty good first pass at a prompt. When we look at this, we've defined the bot's role at the top. When we scroll down, um we've given it some data. We've given it some information on how to reason over uh um the answers that it should be giving to the customer. It's giving some critical

在看 prompt 之前,我先让 eval 在后台跑起来。这个 prompt 作为第一版已经相当不错了。我们看一下,顶部我们定义了机器人的角色。往下滚,我们给了它一些数据,还给了它一些关于该怎么推理、该给客户什么样回答的信息。它还给出了一些关键的

[7:58]

instructions around the tone it should use, um how to do calculations, etc. And then finally we're passing in our customer account context and our user message. So let's have a look at how our first pass the evals did. So we can see as we expect our control case all of our test cases have passed. This is what we expect for this unambiguous test case.

指令,比如该用什么语气、该怎么做计算等等。最后,我们把客户的账户上下文和用户的消息传进去。我们来看看第一版在 eval 上跑得怎么样。可以看到,跟我们预期的一样,对照用例——我们所有的测试用例都通过了。对于这种毫无歧义的测试用例,这正是我们期望的结果。

[8:24]

But it's performing pretty poorly in these other areas. Now, before we zoom in on those specific failure modes here, let's do some general cleanup of our prompt. So, as we mentioned, when we look through this prompt, there's a couple oddities here already. So, for example, first one is we're telling the bot that it's um a human, which just isn't true, right? We can see as we scroll down

但在其他这几个方面,它表现得相当糟糕。现在,在我们放大去看这些具体的失败模式之前,先对 prompt 做一些通用的清理。就像刚才提到的,我们通读这个 prompt,会发现里面已经有几处怪怪的地方了。比如说,第一处是我们告诉机器人它是「一个人类」,这显然不符合事实,对吧。我们往下滚会看到,

[8:57]

there's clearly some information here that has been copied directly from a website. So the key giveaway here is a reference to a hero image. Um there's even some references to cookies um at the bottom.

这里明显有些信息是直接从某个网站上复制粘贴过来的。最明显的破绽是,这里居然提到了 hero image(网页主视觉大图)。底部甚至还提到了 cookies。

[9:10]

So we need to remove a bit of redundant information. When we look at the instructions here, they're all grouped into one big paragraph. So we've got some reasoning here. We've got instructions about the role. uh um some critical instructions as well without a real way of unpacking um policy from guidelines from tone etc.

所以我们得删掉一些冗余信息。再看这里的指令,它们全都堆在一大段里。我们这里有一些推理逻辑,有关于角色的指令,还有一些关键指令,却没有真正把政策、指导原则、语气这些东西拆分开来的办法。

[9:34]

So let me just I've preempted some changes we want to make to this prompt and this is just a diff view of some of those changes. So what we've done is first of all added some structure. So you can see that we've added XML tags here to define the role to separate general guidelines to separate policy to separate tone of voice um etc.

我提前对这个 prompt 做了一些改动,这里是这些改动的 diff 对比视图。我们做的第一件事是加了结构。你可以看到我们加了 XML 标签,用来界定角色、把通用指导原则单独分出来、把政策分出来、把语气分出来,等等。

[10:08]

So if we run that eval then on this new updated prompt, we should hopefully see an improvement in the output as is. So we can see just by clearing up the prompt, we've already improved the model's performance on this prepaid scenario.

那我们拿这个更新后的新 prompt 再跑一遍 eval,顺利的话应该能看到输出上的改善。可以看到,光是把 prompt 收拾干净,我们就已经在这个预付费(prepaid)场景上把模型的表现提上去了。

[10:36]

There's an interesting regression there in that fifth hotspot case and I don't want to worry too much about that now. There's going to be some natural level of variance in the different runs of the eval and we'll come back to that case specifically to see if we can make the prompt consistently better in that area.

有意思的是,第五个 hotspot(热点)用例这里出现了一处回退。现在我不想太纠结这个。eval 的不同运行之间天然会有一定程度的波动(variance),我们待会儿会专门回到这个用例,看看能不能让 prompt 在这块儿稳定地表现得更好。

[10:54]

So what did we learn from this then? Um, simply clearing up the prompt with a better structure, with a better role description has improved the performance. And this is a best practice that you can return to at any stage of writing and maintaining your prompt, especially as your prompts get more detailed and more complex. A general rule of thumb that I like to follow is

那我们从这里学到了什么呢?很简单——光是把 prompt 收拾得结构更好、角色描述更清楚,就把性能提上去了。这是一个你在写 prompt、维护 prompt 的任何阶段都可以反复回头去做的最佳实践,尤其是当你的 prompt 变得越来越细、越来越复杂的时候。我喜欢遵循的一条经验法则是:

[11:17]

if you're reading a prompt and you can't tell guidelines from policy, from data, most likely the model isn't able to either. So before looking at some of those cases in more detail, there's a little bit more general cleanup we can do. Um specifically here looking at creating an output contract. This is a key best practice to follow if you're struggling with your output format consistency. Now

如果你在读一个 prompt 时,自己都分不清哪是指导原则、哪是政策、哪是数据,那模型大概率也分不清。所以在更细致地看那几个用例之前,我们还能再做一点通用的清理。具体来说,这里要看的是建立一个「输出契约」(output contract)。如果你正被输出格式不一致的问题困扰,这是一条要遵循的关键最佳实践。不过呢,

[11:46]

in this case, we have a customer support bot. We want it to reply in a conversational tone. So it's unlikely to be a big issue in this case, but it's something to bear in mind if you you're dealing with more complex output structures like nested JSONs for example.

在我们这个例子里,客服机器人本来就该用对话式的口语语气回复。所以输出格式在这个场景里不太可能是个大问题。但如果你处理的是更复杂的输出结构,比如嵌套的 JSON,那这一点就值得记在心里了。

[12:04]

So again, if we go back to the prompt and see what fixes we can apply here. First of all, we've added a section uh um at the end where we've defined an output format for the model telling it to use uh um XML tags to output the response. But the prompt is not always the most effective way of handling issues. We can also change things in the harness to ensure consistency to a

那我们再回到 prompt,看看这里能加什么修复。首先,我们在末尾加了一个小节,给模型定义了一个输出格式,告诉它用 XML 标签来输出回复。但 prompt 并不总是处理问题最有效的办法。我们还可以在 harness(承载框架)那一层做些改动,来确保一致性——

[12:34]

higher degree. So what we've added here to the API call is a stop sequence which is going to detect that closing XML tag and tell the model to stop generating a response at that point. Now when I run the eval here I don't necessarily expect to see any clear improvement in performance. Um, but it's a general best practice that we should be following and as I said is something that we should

所以我们在这次 API 调用里加了一个 stop sequence,它会检测到那个 XML 结束标签,然后让模型在那个点上停止生成。现在我跑这个 eval 的时候,其实并不指望性能上能有什么明显提升。但这是一个我们应该遵循的通用最佳实践,就像我刚说的,这是我们应该

[13:02]

remember in particular when we have more complex output schemas. One thing to point out here as well if you do have a more complex output schema something like structured outputs can be incredibly helpful to ensure that consistency in a more programmatic way.

记住的,尤其是当输出 schema 更复杂的时候。这里还有一点值得提一下,如果你的输出 schema 确实比较复杂,像 structured outputs 这样的东西会非常有用,能用一种更程序化的方式来保证那种一致性。

[13:21]

Okay. So after the cleanup then we can see that we now have two test cases which are consistently passing but we have three key failure modes the proration the billing error and the hotspot. So let's isolate these one by one um to iterate on the prompt and and see the effect of that.

好,清理完之后我们可以看到,现在有两个测试用例是稳定通过的,但还有三个关键的失败模式:按比例计费、账单错误,还有热点流量。那我们就一个一个把它们拎出来,逐个迭代 prompt,看看效果怎么样。

[13:50]

First of all then the hotspot question. So the question is how much hotspot data is on my unlimited plan. What we expect the model to do is state directly the amount of hotspot data that the customer has.

首先来看热点流量这个问题。问题是:我的无限套餐有多少热点流量?我们期望模型做的,是直接说出这个客户拥有的热点流量额度。

[14:04]

And the reason this is a slightly complex case is because the customer test case that we're dealing with is on a legacy plan. So actually the current policy doesn't apply to them. So if we see what's going on in the actual test case here, the customer data which we're feeding uh um to the prompt includes the amount of hotspot data that customer has. They have 5 gigabytes, right? But

这个之所以稍微复杂一点,是因为我们处理的这个客户测试用例用的是老套餐,所以当前的政策其实并不适用于他们。如果我们看一下这个实际的测试用例,我们喂给 prompt 的客户数据里其实是包含了这位客户的热点流量额度的,他们有 5 GB,对吧?但是

[14:29]

they also have a grandfathered plan. So what we're seeing uh the model is actually telling the customer is the general um the unlimited plan includes 4 GB um but since you're on a legacy plan you should go check this out yourself.

他们用的是被保留下来的老套餐(grandfathered plan)。所以我们看到模型实际上是这么告诉客户的:一般来说无限套餐包含 4 GB,但既然你用的是老套餐,你得自己去查一下。

[14:55]

So let's have a look at the prompt then to see um why the model is deflecting this question to the customer account URL rather than actually giving the information itself. Now if we read this prompt originally it said we changed our plans recently and the policy doc shows the current plan data and customers on grandfather's plan have different rates. Never give a

那我们来看看 prompt,搞清楚为什么模型把这个问题踢给客户账户的 URL,而不是自己直接给出信息。我们读一下最初的这个 prompt,它写的是:我们最近调整了套餐,政策文档显示的是当前套餐的流量,而老套餐的客户费率不同。绝对不要给

[15:20]

customer the wrong plan details. instead point them to the URL. So it's clear that this instruction, this latter one, never give customer the wrong information, is the instruction that the bot has been optimizing for. And you might recognize this as being very similar to a patch that you might have introduced in a previous model that you were using to avoid where the model was

客户错误的套餐细节,而是把他们引导到那个 URL。所以很清楚,后面这条指令——绝不给客户错误信息——就是这个 bot 一直在为之优化的那条指令。你可能会发现,这跟你之前用某个旧模型时为了避免它出错而打的补丁非常像,当时那个模型会

[15:48]

giving the customer the wrong information about that plan. Now, as our models have evolved, they have gotten much better at instruction following. So, it's likely that instructions like these have now become redundant and are actually being overfitted to.

给客户提供关于套餐的错误信息。但随着我们的模型不断进化,它们在遵循指令上已经做得好太多了。所以像这样的指令现在很可能已经变得多余,反而成了过拟合的对象。

[16:06]

So, what we're going to tell the model instead is give this balanced view uh um where it says, you know, customers on grandfather's plan have different allowances, but it's captured in the customer information that's given and that is the accurate source of truth.

所以我们改成这样告诉模型:给出一个平衡的视角,也就是说,你要讲清楚老套餐的客户有不同的额度,但这个信息已经在给出的客户资料里体现了,而那才是准确的事实来源(source of truth)。

[16:26]

So running the eval here we should hopefully be addressing uh um all of the test cases for the hotspot case. Now I am running this live so there could be some variability here but we see here that now clearly all of our test cases are are passing.

所以在这里跑 eval,我们应该有望把热点流量这个用例的所有测试都搞定。我现在是现场实时跑的,所以可能会有一些波动,但我们看到现在所有测试用例显然都通过了。

[16:44]

So what did we learn from this? Well, we worry a lot about hallucinations or the invention of facts and numbers, but actually the opposite can also happen. The model can withhold information that it actually has access to. Now, we saw here that this is likely a result of a patch that we introduced for a previous model. And a best practice that we could follow here is actually using version

那我们从中学到了什么?我们常常很担心幻觉,也就是模型编造事实和数字,但其实相反的情况也会发生:模型可能会把它本来掌握的信息藏着不说。我们这里看到,这很可能是我们为某个旧模型打的补丁导致的。这里我们可以遵循的一个最佳实践,其实就是用版本

[17:11]

control where wherever we are making defensive changes in the prompt, we are tracking the reason why we've introduced these. Sometimes they're necessary, but in the future these kind of changes can produce unwanted effects so that we can backtrack on them.

控制(version control):只要我们在 prompt 里做了防御性的改动,就把我们为什么引入这些改动的原因记录下来。有时候这些改动是必要的,但在未来,这类改动可能会产生意料之外的副作用,这样我们就能回退它们。

[17:31]

So the next failing test case then is this proration calculation where a customer asks what if I upgrade to the 30 gigabyte plan? What will my next bill be? And what we want the model to do is to perform some calculation and return exactly uh um what their next bill would be rather than giving some sort of vague output which is what we can see it's doing right now. Uh um if we look at

那下一个失败的测试用例,是这个按比例计费的计算。客户问:如果我升级到 30 GB 的套餐,我下一期账单会是多少?我们希望模型做的是进行一些计算,精确返回他们下一期账单的金额,而不是给出某种含糊其辞的输出,而后者正是它现在在做的。如果我们看一下

[18:00]

what the model is returning, it's clearly reasoning through it. It's doing a little bit of mental maths here and there, but it's not really giving the customer a concrete answer. And I wouldn't rely on this as being able to accurately give the customer a response.

模型返回的内容,它显然是在一步步推理,这里那里做了点心算,但它并没有真正给客户一个确切的答案。我也不会指望靠它来准确地给客户回复。

[18:20]

So if we look at the prompt then to see how we can fix this in the original prompt we can see that all the instructions that were given to it is telling it don't ever give a customer a vague answer. Uh um critical always calculates any pr-rated amounts correctly. Now, telling the model to do a good job isn't particularly helpful when we don't give the model the capability to actually do

那我们看一下 prompt,看看怎么修这个问题。在原始 prompt 里我们能看到,给它的所有指令就是告诉它:绝对不要给客户含糊的答案。关键!永远要正确计算任何按比例分摊的金额。但是,当我们没有给模型真正完成这件事的能力时,光是告诉模型要好好干

[18:55]

a good job. We want to avoid the model doing mental math. So, what we're going to introduce is give the model a tool. So, we're saying in the prompt whenever you're doing any calculations, please use the calculate proration tool to do so.

是没什么用的。我们想避免让模型去做心算。所以我们要引入的是:给模型一个 tool。也就是在 prompt 里说:只要你要做任何计算,请用 calculate proration 这个工具来做。

[19:10]

In order to introduce that tool, we need to introduce it into the API to tell the model you have access to this tool. We need to define um the tool schema which tells the model what this tool does and when to use it. And then finally we need to actually implement the tool which is the maths behind how it should be doing that calculation.

为了引入这个 tool,我们需要把它加进 API 里,告诉模型你可以使用这个工具。我们需要定义这个 tool 的 schema,它告诉模型这个工具是干什么的、什么时候该用。最后我们还得真正去实现这个工具,也就是它背后做这个计算的数学逻辑。

[19:38]

So running that eval then for another pass we can see that all the test cases are now passing. It's clearly done uh um the maths using the tool in the background and returning the correct response. So the key lesson to take away here is instructions don't add capability.

那再跑一遍这个 eval,我们可以看到现在所有的测试用例都通过了。它显然是在后台用这个工具把数学算好了,然后返回了正确的回复。所以这里要带走的关键一课是:指令并不会增加能力。

[20:06]

Telling the model it's critical to do a calculation right doesn't make it better at mental math. So the correct approach was to give it a tool. Overall giving it the ability to reason over harder problems and using tools to actually execute them reliably.

告诉模型把计算做对很关键,并不会让它的心算变得更厉害。所以正确的做法是给它一个 tool。总的来说,就是给它在更难的问题上进行推理的能力,再用工具去可靠地把它们执行出来。

[20:25]

So now we have one final failing test case which we need to address which is this billing error here. In this scenario there is a billing conflict and what we really want is the agent to escalate this to a human. And what we're seeing it doing instead is it's trying to explain to the customer what the reason behind it might be.

那现在我们还剩最后一个需要处理的失败测试用例,就是这里的账单错误。在这个场景里存在一个账单冲突,我们真正想要的是让 agent 把它升级转交给人工。而我们看到它实际在做的,却是试图向客户解释背后的原因可能是什么。

[20:54]

and it's trying to kind of diagnose the problem itself. So in order to fix this behavior, let's again have a look what it was told in the prompt. We see in the initial instructions it was giving, it says, "Avoid escalating or transferring to a care specialist unless absolutely necessary as it cost approximately $8 and it counts against our team's fast contract resolution."

它在试图自己去诊断这个问题。所以为了修复这个行为,我们再来看看 prompt 里是怎么跟它说的。在最初给它的指令里我们看到,它写着:除非绝对必要,否则避免升级或转交给客服专员,因为这大约要花 8 美元,而且会拖累我们团队的快速一次性解决率。

[21:29]

Now, this is only giving one side of the story, right? We're telling it what the cost is to escalating, but not the benefit, which means it's going to overfit again to not escalating this scenario. And second of all, we've got this clear conflict between what we've defined in the eval in terms of what we want the model to do to do this escalation versus what we're actually telling it to do.

现在,这只讲了故事的一面,对吧?我们告诉了它升级的成本,却没讲收益,这就意味着它又会过拟合到不去升级这个场景上。其次,我们这里有一个明显的冲突:我们在 eval 里定义的、希望模型去做的(也就是做这个升级)跟我们实际告诉它去做的,是矛盾的。

[21:56]

And the fix that's relevant here is to give it both sides of the story by saying it costs $8 uh um to escalate a case, but actually if you get this wrong, then it's going to cost you a refund as well as customer trust.

这里相关的修复办法,就是把故事的两面都讲给它听:虽然升级一个 case 要花 8 美元,但如果你这件事搞错了,那代价会是一笔退款,外加客户的信任。

[22:16]

Again, here we observed how the model optimizes for a goal. And this kind of instruction is a common instruction to give. It's quite similar to the one we saw earlier where we didn't want it to overfitit to a certain type of behavior.

在这里我们又一次观察到模型是怎么为一个目标去优化的。而这种指令是很常见的一类指令。它跟我们前面看到的那条很像,当时我们不希望它过拟合到某种特定行为上。

[22:31]

But it's the kind of instruction that can be followed quite differently by different generations of models. And specifically, as models become more intelligent, we need to remember to state both sides of the trade-offs because our models are becoming better themselves at making those tradeoffs themselves.

但这种指令,不同代际的模型在遵循时可能会有相当不同的表现。具体来说,随着模型变得越来越聪明,我们要记得把权衡的两面都讲清楚,因为我们的模型自己在做这些权衡上也越来越在行了。

[22:57]

So if we just go back to our eval then and um run our final test case, we should see that all of our evals are now passing correctly. So overall, we looked at applying general hygiene principles, how that can provide an initial uplift to the prompt, making sure we're removing any redundant instructions which were initially intended as patches for previous models

那我们回到 eval,跑一下最后这个测试用例,应该会看到现在所有的 eval 都正确通过了。所以总体来看,我们做了这些事:运用通用的卫生原则,看它如何给 prompt 带来一次初步的提升;确保移除掉那些当初是为旧模型行为打补丁而引入的、如今已经多余的指令;

[23:26]

behavior, making sure we're giving it tools to do certain tasks reliably. Now there's one other scenario that we uh introduced at the start which is one that you might also encounter in your work which is where we're building a new agent from scratch. And the example that we'll look at here is um an agent whose purpose it is to create a week-long retail staff schedule based on employee

还有确保给它配上工具,好让它能可靠地完成某些任务。现在还有另一个场景,是我们一开始就提到过的,你在工作中也可能会碰到,那就是我们要从零开始构建一个新的 agent。我们这里要看的例子,是一个 agent,它的任务是根据员工的

[23:57]

availability and other constraints. And when we're building a new agent from scratch, we need to consider not just the prompt, but also the model that we're using and the harness that we're using. So in this next example, we're going to compare a number of approaches to explore the impact of those three different areas.

可用时间和其他一些约束,排出一份为期一周的零售员工排班表。当我们从零开始构建一个新 agent 时,需要考虑的不只是 prompt,还有我们用的模型,以及我们用的 harness。所以在接下来这个例子里,我们要对比好几种做法,来探究这三个不同方面各自带来的影响。

[24:24]

So again, I've just vcoded up this web app so that we can walk through this problem. Uh um in this demo here I've just laid out what the problem is that we're addressing. We have our eight employees. Um on the right we have this schedule that we need to staff with the headcount and we have our constraints that must be satisfied in every scenario.

同样地,我用 vibe coding 快速搭了个 web app,这样我们就能把这个问题走一遍。在这个 demo 里,我把我们要解决的问题摆了出来。我们有八名员工。右边是我们需要按人头配齐的排班表,还有在每种场景下都必须满足的约束条件。

[24:50]

Now, because we have these hard rules, rather than using an LLM judge like we did in the previous case to do the grading, we can actually use a just a Python function which programmatically checks for every schedule that's generated how many violations were made.

因为我们有这些硬性规则，所以不像前面那个例子那样用 LLM judge 来打分，这里我们其实可以直接写一个 Python 函数，对每一份生成的排班表，用程序自动检查它违反了多少条规则。

[25:13]

So to begin with in a we want to start simple. We're going to use a simple prompt. We're going to use the bare bones that we think we'll need with a model set 46 to see how it performs and how we're going to hill climb against that. Um, so here is our baseline prompt. We've already applied some of that general hygiene and those best practices that we saw earlier on using

所以一开始我们想从简单的做起。我们用一个简单的 prompt，只放上我们认为最起码需要的内容，配上 4.6 这个模型，看看它表现如何，再以此为基础一步步往上爬。这就是我们的基线 prompt。我们已经应用了前面讲过的一些通用规范和最佳实践，比如用

[25:36]

XML tags to structure the prompt. We've given it an output format as well. Now that we're giving a schedule, uh we're asking it to output a JSON which if we don't give that output structure might lead to passing errors uh downstream.

XML tag 来组织 prompt 的结构。我们还给它规定了输出格式。既然要输出一份排班表，我们就要求它输出 JSON，如果不规定这个输出结构，后续环节解析时可能会出错。

[25:59]

When we run the simple model on a first iteration of the evals, all cases fail. Now, just what we're looking at here is in our test set, we're essentially repeating uh um we're doing five trials here. Uh and these numbers are showing how many violations were made in each trial.

用这个简单模型跑第一轮 eval 时，所有用例都失败了。我们这里看到的是，在测试集里我们基本上是在重复——其实是跑了五次试验，这些数字显示的是每次试验里违反了多少条规则。

[26:25]

In the outputs, we can see that it's made a decent attempt at reasoning through the problem, but it's burning a lot of tokens. Uh and it's clearly not checking its work um as it's not getting to the right impact. So let's try a larger model uh a model which we know is better at reasoning. So we're going to run it through uh Opus 4.7 instead keeping everything else the same. Now,

从输出里可以看到，它对这个问题做了一番还算像样的推理，但烧掉了很多 token。而且它显然没有检查自己的工作，因为它没得出正确的结果。那我们试试更大的模型，一个我们知道推理能力更强的模型。所以我们改用 Opus 4.7 来跑，其它一切都保持不变。

[27:00]

interestingly, whilst all test cases are still failing, you can see that the overall number of violations that Opus has made has reduced significantly from Sonnet 4.6. So, we're possibly on to something here, right? This isn't good enough to ship because it's still failing, but clearly giving it more reasoning capability is helping drive it towards a better result. So what we're

有意思的是，虽然所有测试用例还是失败，但你可以看到 Opus 违反规则的总数相比 Sonnet 4.6 大幅下降了。所以我们可能找对路子了，对吧？这还不够好，没法上线，因为它仍然在失败，但很明显，给它更强的推理能力正在把它推向更好的结果。所以接下来我们要

[27:28]

going to try next is using opus with adaptive thinking instead. So it can decide for itself how much thinking it needs how much reasoning it needs to use to solve this issue. So no change to the prompt really just uh a change to the API here.

尝试的是改用带 adaptive thinking 的 Opus，让它自己决定需要多少思考、需要用多少推理来解决这个问题。所以 prompt 基本没变，只是改了一下这里的 API 调用。

[27:54]

So this now seems to reliably generate compliance schedules, but it requires a lot more tokens. We're tripling essentially in the number of tokens that we're using here, and we're tripling the latency. So we want to try and see if we can optimize that cost latency trade-off a little bit more. Um, this is latency 100 seconds.

现在它似乎能稳定地生成合规的排班表了，但需要多得多的 token。我们用的 token 数量基本翻了三倍，latency 也翻了三倍。所以我们想看看能不能在成本和延迟这个权衡上再优化一点。这里的 latency 是 100 秒。

[28:18]

Obviously, I'm running this one async. uh for the purposes of time, Opus 47 hasn't magically gotten much faster since the last time uh you used it. Um so let's see if we can optimize a little bit more for the token latency tradeoff.

显然为了节省时间，我这一个是异步跑的。Opus 4.7 从你上次用它到现在并没有奇迹般地变快很多。那我们来看看，能不能在 token 和延迟的权衡上再优化一点。

[28:39]

What we haven't tried yet is using Sonnet 46. So a smaller model but with a better prompt. We looked a lot at the prompt optimization uh um in that last section. So I've added a couple details to the prompt here. We're talk we're in particular how to uh reason through this problem and most critically telling it to check its work before outputting it.

我们还没试过的是用 Sonnet 4.6，也就是一个更小的模型，但配一个更好的 prompt。前面那一节我们花了很多时间研究 prompt 优化。所以我在这里给 prompt 加了几个细节，特别是怎么一步步推理这个问题，最关键的是告诉它在输出之前先检查自己的工作。

[29:10]

So when I ran that EAL we see that it passes in two out of the five cases. Now the failure modes that we're seeing is actually not violations of the scheduling requirements but the model hasn't been able to finish the tasks within the output limit that we set. So whilst we could increase the max tokens that this model is able to use to get all five test cases passing,

跑完这个 eval 后，我们看到五个用例里通过了两个。现在我们看到的失败情况其实并不是违反排班要求，而是模型没能在我们设定的输出上限内把任务做完。所以虽然我们可以提高这个模型能用的 max tokens，让五个测试用例全部通过，

[29:45]

we see here that we're using even more tokens and this run has an even higher latency. So this is probably not the route that we want to go down. Now as a final pass, then we want to look at doing this a little bit more agentically.

但这里可以看到我们用的 token 更多了，这一轮的 latency 也更高了。所以这大概不是我们想走的路子。那么作为最后一招，我们想试试用更 agentic 的方式来做这件事。

[30:03]

So we're going to use this generate evaluate repair loop where essentially the generator now creates a first draft of the schedule and then we have a separate prompt which reports any specific violations that it made. So not programmatically checking it but checking it with an LLM.

所以我们要用这个「生成—评估—修复」的循环：基本上 generator 先生成排班表的初稿，然后我们有一个单独的 prompt 来报告它违反了哪些具体规则。所以不是用程序来检查，而是用一个 LLM 来检查。

[30:24]

So we're checking for every rule and we're providing evidence of every violation. And we then have a third repair prompt which receives uh any violations that were made and tries to make targeted fixes to it. So we have three very simple prompts but they're now running independently rather than trying to do everything in one large prompt.

我们逐条规则地检查，并为每一处违规提供证据。然后我们还有第三个修复 prompt，它接收所有发生的违规，并尝试做针对性的修正。所以我们有三个非常简单的 prompt，但它们现在是各自独立运行的，而不是想用一个大 prompt 把所有事情都干完。

[30:57]

So we can see in this case our agentic approach has solved all of our test cases um with a much lower number of tokens and with a lower latency than trying Sonet 46 with a better prompt. So going forward, it seems like there's two appropriate approaches to take here. Using Opus 47 with adaptive thinking or using this agentic loop. Now moving forward, we'd probably want to do a little bit more

可以看到，在这个例子里，我们的 agentic 方法解决了全部测试用例，而且用的 token 数量低得多，latency 也比用更好 prompt 的 Sonnet 4.6 更低。所以往后看，这里似乎有两条合适的路可走：用带 adaptive thinking 的 Opus 4.7，或者用这个 agentic 循环。再往下，我们可能还想做更多

[31:30]

optimization on this loop to try and get it to be more efficient. But there's one key benefit as well from using this generate evaluate repair loop. And that is that you can put in soft requirements at runtime. So in the evaluation prompt, we can say Harry doesn't like working with Sally. So as much as possible, try and separate them from working together or we need a third shift uh um on

优化，让这个循环更高效。但用这个「生成—评估—修复」循环还有一个关键好处，那就是你可以在运行时塞进软性要求。比如在评估的 prompt 里，我们可以说 Harry 不喜欢和 Sally 一起工作，所以尽可能把他们的排班分开；或者我们说周三需要加一个班次，

[31:57]

Wednesday, for example. So it means that you're not having to make changes to the um Python function which is doing the evaluation in the back end every time to satisfy for any soft constraints which might depend just on a case-byase basis.

举个例子。这就意味着，每次为了满足那些可能因情况而异的软性约束，你不必去改后端那个负责评估的 Python 函数。

[32:16]

So to wrap up then pulling all of those learnings together, what did we see? Well, we looked at two scenarios. Two scenarios which I, as an engineer, see most in my day-to-day, which is where we're maintaining a prompt. We're migrating to a new model which has some different behaviors and we're and building a new use case from scratch. We saw that general hygiene

那我们来收个尾，把所有这些经验串起来，我们看到了什么？我们看了两个场景，这两个场景是我作为工程师在日常工作中最常遇到的：一个是我们在维护一个 prompt，迁移到一个行为有些不同的新模型；另一个是从零开始搭建一个新用例。我们看到，遵循通用的

[32:42]

principles following those can and immediately uplift the performance against um a set of evals and that we need those eval to be able to rigorously see any impacts of changing our prompt on the output. Then we saw this process of targeting failure modes one by one, adding structure, avoiding long band lists, etc. were all things that helped push our model to the correct behavior.

规范原则，能立刻提升模型在一组 eval 上的表现；而我们需要这些 eval，才能严格地看出改动 prompt 对输出造成的各种影响。然后我们看到了这个过程：逐个攻克失败模式，加上结构，避免又长又乱的禁止事项清单等等，这些都是把模型推向正确行为的有效手段。

[33:10]

And then finally with our new agentic bot that we were building, we saw the impact of splitting into three separate prompt systems. So rather than using one prompt to address everything, we're actually isolating different tasks where it's easy and repeatable to separate out the steps that it needs to take every time.

最后，在我们搭建的那个新 agentic 机器人上，我们看到了把它拆成三个独立 prompt 系统所带来的效果。所以与其用一个 prompt 解决所有事情，我们其实是在隔离不同的任务——在那些每次都需要走的步骤容易拆分、可重复的地方，把它们分离出来。

[33:36]

Thank you so much for attending this afternoon. I hope you have a fantastic rest of your day.

非常感谢大家来参加今天下午的分享，祝你们今天剩下的时间过得愉快。