Evals for taste: Hill-climbing a slide-generation agent

频道: Claude
视频: https://www.youtube.com/watch?v=v9FTCvkV_a0
原文语言: en
统计: 共 125 轮

[0:03]

[music] >> Hello. Hello. Hello. Good afternoon, everyone. I hope you all had a wonderful lunch. Um there's so many of you as well. I'm actually kind of surprised by this. Um happy to see that there's that much interest in talking about uh Evals. Um I personally am a big fan of anything Evals related.

大家好，大家好。下午好啊各位。希望你们午饭都吃得很愉快。哇，来了这么多人，说实话我还挺意外的。很高兴看到这么多人对聊 eval 这个话题感兴趣。我个人是任何跟 eval 相关的东西的铁杆粉丝。

[0:38]

But I know not everyone's That's not everyone's cup of tea, right? Um so very happy to see this many people of you. Um So yeah, this So today's session is really going to be about Evals. Um and I guess my goal for this session is for you all to be afterwards to be inspired to build Evals, to be like, "Okay, Evals are actually really useful."

但我也知道，不是每个人都对这个感冒，对吧？所以看到来了这么多人真的很开心。今天这场分享主要就是讲 eval。我这场的目标，是希望大家听完之后能受到启发，愿意自己去构建 eval，会觉得「好，原来 eval 真的挺有用的」。

[0:59]

Um and how you can act on them, right? Like, we're going to be building Evals. I want you to get a better sense of like, "Okay, how should I be thinking about building Evals? What are useful type of Evals? And then also, how can we use and take these Evals to then make better agents, right?" So that's the main goal of this session.

以及，怎么把 eval 用起来，对吧？我们待会儿会真的动手构建 eval。我希望你们能更清楚地理解：「该怎么思考构建 eval 这件事？哪些类型的 eval 是有用的？然后，又怎么拿这些 eval 去做出更好的 agent？」这就是这场分享的主要目标。

[1:20]

Um and the way we're going to do this is by building a slide generation agent and then finding out like, "Okay, what are some good Evals? What do we want to measure? And then, how can we build now better agents based on the feedback that we're getting from our Evals?"

我们的做法是：构建一个幻灯片生成的 agent，然后去摸索「好，哪些是好的 eval？我们想衡量什么？然后怎么根据 eval 给我们的反馈，做出更好的 agent？」

[1:37]

And the first thing that we all need to set the stage on is, "What are Evals, right?" So Evals are systematic tests that measure how well an AI system performs on a specific domain or use case, right?

我们都得先把一个概念立清楚：「到底什么是 eval？」eval 就是一套系统化的测试，用来衡量一个 AI 系统在某个特定领域或使用场景下表现得有多好。

[1:53]

So they give you information about like, "What's the quality of the results? Um what did it do well? What was it not good at? How can we improve, right? And evals they are made up of tasks define certain scenarios that then encode certain expectations through the grading logic. So, one way that we're thinking about evals is if you for example are building an AI

所以 eval 会告诉你这些信息：「结果的质量怎么样？哪里做得好？哪里做得不行？我们怎么改进？」eval 是由一个个任务（task）组成的，这些任务定义了某些场景，再通过打分逻辑把某些预期编码进去。举个例子，我们对 eval 的一种理解是：假如你在构建一个 AI

[2:17]

system and AI agent and you want to make sure that the output adheres to like a certain type of quality or you want need to make sure like this must always be present. Evals are a way to kind of encode this behavior in a way where then afterwards, if your evals fail, you know like, okay, my agent is not doing or behaving the way it is intended, right? So, that's

系统、一个 AI agent，你想确保它的输出符合某种质量标准，或者你需要确保某些东西必须始终存在，那么 eval 就是一种把这种预期行为编码下来的方式。这样一来，之后如果你的 eval 没通过，你就知道：好，我的 agent 没有按照预期那样工作或表现。所以

[2:41]

the way how we can use these evals. And then evals is also the bridge between things like it seems to work or like we know it works or maybe it's all like, ah, it kind of feels a little bit worse today for some reason.

这就是我们使用 eval 的方式。同时，eval 也是一座桥梁，连接「它好像能用」「我们确定它能用」和「啊，不知道为什么今天感觉差了点」这几种状态之间。

[2:55]

It's always very hard to act on these types of vibes, right? Like I think vibes definitely have their own place. I think they're useful just to get like a general sense check of like how people are feeling. But they're not very actionable, right? And that's kind of what you want to get out of evals. We want to have something that's actionable.

凭「感觉（vibes）」是很难采取行动的，对吧？我觉得感觉当然有它的位置，它有用，能让你大致了解一下大家是什么体感。但它不太能转化成行动，对吧？而这正是你想从 eval 里得到的东西——我们想要的是可以据此行动的东西。

[3:15]

So, then we always ship eval. Like we always once we release like a model, we always have this accompanying benchmark scorecard, right? And then we always list like, oh, these are like a bunch of evals. This is what we achieve, what our models achieve. We compare them to other models. We compare them to competitor models, right? Um And there's like always a few usual

所以我们每次都会随模型一起发布 eval。每次我们发布一个模型，都会配一份 benchmark 成绩单，对吧？上面会列出：哦，这是一堆 eval，这是我们达到的成绩、我们模型达到的成绩，我们拿它和其他模型对比，和竞品模型对比。这里面总有几个老面孔，

[3:36]

suspects, right? Like for example, um SweetBench is a very famous one which measures agentic coding abilities. Terminal Bench is one that's also quite popular. But we also have other types of evals, right? We have like tool use and agents like for example like Tower Bench, OSWorld, which are some other evals that measure different things. And then we also have like reasoning and knowledge

对吧？比如说 SWE-bench，这是一个非常有名的，衡量 agentic 编程能力的 benchmark。Terminal-Bench 也是个挺流行的。但我们还有其他类型的 eval，比如衡量 tool 使用和 agent 能力的，像 Tau-bench、OSWorld，这些是衡量不同方面的 eval。然后我们还有推理和知识类的，

[3:58]

um like Arc-AGI too. Um Now, this is all fine and dandy, right? And then you look at these evals and we always every time a new model releases like, "Oh, we stopped off the benchmark for these and these um evals, right?" Um and they give us like a general general sense of like how well is the model and how much did we improve upon previous versions, right?

比如 ARC-AGI 2。这一切看起来都挺好的，对吧？你看着这些 eval，每次有新模型发布我们就说：「哦，我们在这些那些 eval 上刷新了 benchmark 成绩。」它们大致告诉我们：这个模型有多好，相比之前的版本我们提升了多少。

[4:20]

But for you guys, if you're like building something, if you're building an agentic system, this doesn't really say much usually, right? Like because like we we don't measure for example like a very specific use case that you guys are building on, right? We measure these generic general benchmark that measure a lot of capabilities, but they might not be applicable to your

但对你们来说，如果你们在构建某个东西、构建一个 agentic 系统，这些数字通常说明不了太多，对吧？因为我们并没有去衡量你们正在做的那个非常具体的使用场景。我们衡量的是这些通用的、泛化的 benchmark，它们涵盖很多能力，但不一定适用于你们

[4:42]

specific use case, right? So, that's why we always say build your own evals, benchmark the different models, benchmark your AI agent, and make sure that you get the most out of the models, and make sure that you're also using the right model for the job.

那个具体的使用场景，对吧？所以我们一直说：构建你自己的 eval，拿不同的模型来 benchmark，给你的 AI agent 做 benchmark，确保你把模型的价值榨到最大，也确保你为这个活儿选对了模型。

[4:56]

Right? And so, why are these evals specifically important? So, this is my pitch to you of start using evals, right? So, without evals, suppose you don't have evals. I think we've all been into the scenario where you have like this agent and it's working fine, and then you get like this feedback of like a customer who is like saying like, uh it's it's not really up to par of

对吧？那么，为什么这些 eval 这么重要？这就是我向你们安利「开始用 eval」的理由。没有 eval 的话，假设你没有 eval——我想我们都遇到过这种情况：你有这么个 agent，它运行得好好的，然后你收到客户的反馈，说：呃，它好像没达到

[5:20]

like this new model switch, ma it's something is off, right? It's very hard like to do anything with that information, right? It's just like, "Okay, um do you have some logs maybe that we can take a look at some specific instances, right?" And then you try to like debug it manually, right?

应有的水平，比如换了新模型之后，感觉哪里不对劲。拿到这种信息你很难做点什么，对吧？你只能说：「好吧，你有没有日志，我们能看看某些具体的实例？」然后你试着手动去 debug。

[5:37]

But in a way, you're still flying blind, right? You're always in a reactive loop, so you wait for the feedback and then you're like, "Okay, let's see what we can do about this, right?" So, you basically only catch issues in production. You can fix, for example, like one issue, which then might, for example, create multiple more down the line by making, I don't know, prompt

但某种程度上，你还是在盲飞，对吧？你永远处在被动的循环里：等着反馈进来，然后说「好，我们看看能拿它怎么办」。所以你基本上只能在生产环境里才抓到问题。你修了一个问题，结果这个改动可能在后面又制造出更多问题——比如你改了个 prompt，

[5:54]

change tweak that suddenly degrades the capabilities on like other tasks that you haven't even considered. Um it's it's also quite annoying to distinguish like genuine feedback from noise, right? Um which is always you don't want to act on every single thing that you see because people have also some um biases in the way they perceive these things, right? Um

做了点微调，结果突然让另外一些你压根没考虑到的任务的能力退化了。而且，要把真正有价值的反馈和噪音区分开也挺烦的，对吧？你不想看到什么就照单全收去改，因为人们在感知这些事情时也带着各种偏见。

[6:16]

And then finally, I think which is the most important one is there's no way to verify improvements or regressions on anything that you're building or that you've done, right? So, you like need a way to make sure that the changes that you are making to your agent are actually impacting the quality and making sure that you improve upon the previous versions, right?

最后，我觉得最重要的一点是：你没有办法去验证，你做的任何东西到底是改进了还是退化了，对吧？所以你需要一种方式来确认，你对 agent 做的这些改动确实在影响质量，确实相比之前的版本有所提升。

[6:33]

And so, this is basically what evals do give you. If you add evals, you have clarity. You need to define what does success look like, right? Because like if let's say you don't have evals, right? And you're not even able to articulate like this is how the agent should behave. This is what the successful end product would look like for my agent, then how can you make sure

而这基本上就是 eval 能给你的东西。一旦你加上 eval，你就有了清晰度。你必须定义「成功长什么样」，对吧？因为如果你没有 eval，你甚至都没法清楚地表达「agent 应该这样表现，对我的 agent 来说成功的最终产物应该是这个样子」，那你又怎么能确保

[6:53]

that your agent's actually behaving properly because you can't even vocalize it to yourself like this is what it should be. So, building these evals forces you to define, formalize in a way what you expect your agent to do.

你的 agent 真的表现得当呢？因为你连对自己都说不清「它应该是什么样」。所以构建这些 eval 会逼着你去定义、去某种程度上把「你期望 agent 做什么」给形式化下来。

[7:05]

Um It also allows you, as I said, to iterate on optimal agent configs. Um you can also adopt new models faster, right? Instead of like saying like oh, we might test out this new model and then see if it's like okay. You now have like some clarity to say like okay, this is better on this and this is not better on this and this is why we should or should not migrate to a new model. Which

正如我说的，它还让你能够迭代出最优的 agent 配置。你也能更快地采用新模型，对吧？不再是「哦，我们也许试试这个新模型，看看行不行」，现在你有了清晰度，可以说「好，它在这方面更好，在那方面没更好，这就是我们该或不该迁移到新模型的原因」。

[7:27]

especially I think is quite relevant with the pace of new models coming out. I think it's also like just taking this load off your back of always constantly having to find like okay, what is the new frontier, right? Um And then finally, making problems visible before launch, right? So, you know, like, oh, if there's like a few um cases that you have that we always do

考虑到新模型出现的速度，我觉得这一点尤其重要。它也算是帮你卸下了一个负担——你不用一直没完没了地去搞清楚「现在最前沿的是哪个」。最后，它能让问题在上线之前就暴露出来，对吧？你知道的，如果你有那么几个一直表现稳定、

[7:49]

well or that you trust to provide a lot of insight, that's where you get the most value out of evals. And so, how how do evals really fit in? So, originally, um when we were like thinking about like prompt engineering, we had this basically this flow of like how you should optimize a prompt, right?

或者你信得过、能提供大量洞见的案例，那就是你从 eval 里能榨出最大价值的地方。那么，eval 到底是怎么融进整个流程的？最早，当我们思考 prompt engineering 的时候，我们有这么一套流程，讲的是你该怎么优化一个 prompt，对吧？

[8:08]

So, first you develop your test cases, which are the evals in the end. Then you write like a prompt, you test out the prompt against the tasks, you refine the prompt a little bit, and then it goes back, you run the prompt again, you refine it until you're like, "Okay, I'm doing good great on my evals. I'm confident that my system is working properly." And then finally, you can

首先你开发你的测试用例，这些用例最终就是 eval。然后你写一个 prompt，拿这个 prompt 去跑这些任务，稍微改改 prompt，再回去重新跑一遍，反复打磨，直到你觉得「好，我在 eval 上表现得很棒，我有信心系统运行正常了」。然后最后你就可以

[8:28]

like ship the post polished prompt, right? Um over time, systems have gotten a lit little Oh. Ooh, can I go back? Can we go back one slide, please? Thank you. Um so, over time, it has gotten a little bit more complex with now agents coming into the loop with like tool calls, skills, all the different ways to optimize your contacts and all that stuff. So, over time, these

把这个打磨好的 prompt 发布出去，对吧？随着时间推移，系统变得稍微……噢，能往回翻吗？能麻烦翻回上一张幻灯片吗？谢谢。随着时间推移，事情变得更复杂了：现在 agent 进来了，带着 tool 调用、skill，还有各种优化你 context 的方式等等。所以随着时间推移，这些

[8:50]

systems get more and more complex. So, it's also way more levers that you can pull to make changes to your agents, which makes it once again then more important to have evals that forces you to have concrete way of identifying these are the things that we can change, and these are the things that impact the system in a positive So, once again, like with agents, it's the

系统变得越来越复杂。这也意味着你能拉动的「杠杆」更多了，能对 agent 做的改动更多了。这反过来又让 eval 变得更重要——它逼着你用一种具体的方式去识别：这些是我们可以改的东西，这些是真正能正面影响系统的东西。所以，对 agent 来说也是

[9:13]

same flow, right? Um except now with just way more and way more complex things. Um evals, when you create them, there's basically a few graders. A grader is what we consider basically a way how we can judge the output, right? And like one of those ways is, for example, a code-based grader, which is pretty similar to, for example, a unit test, as you might know in like

同样的流程，对吧？只不过现在东西多得多、复杂得多。当你创建 eval 时，基本上会有几种 grader（打分器）。grader 就是我们认为可以用来评判输出的一种方式。其中一种方式，比如说，是基于代码的 grader，它和软件工程里你可能熟悉的

[9:37]

software engineering, right? It can be like a string match, rejects, maybe a fuzzy fuzzy match, but it's like a strict analysis, right? It finds static and tool code checks and the advantages of this one is it's fast, cheap, deterministic, but it has a big drawback which is that it's brittle and it also lacks in nuance, right?

单元测试很像，对吧？它可以是字符串匹配、正则表达式，也可能是模糊匹配，但本质上是一种严格的分析。它做静态检查和工具代码检查，它的优点是快、便宜、确定性强，但它有个大缺点：它很脆（brittle），而且缺乏细微的拿捏。

[9:59]

Um And with this we mean like especially brittle is quite an interesting one in my opinion because like these deterministic checks they force a certain deterministic behavior, right? But with and sometimes this is absolutely the way we want an agent to behave, right? Like for example, let's say you have an agent that creates a slide deck for example, you want to make

我们说的「脆」——在我看来这一点特别有意思——因为这些确定性的检查会强制某种确定性的行为，对吧？有时候这绝对正是我们想要 agent 表现出来的样子。比如说，假设你有一个 agent，专门生成幻灯片，你会想确保……

[10:20]

sure that in the end there is a slide deck present, deterministic check. But then if you want to have like a check on what's the quality of the slide deck, this is way more nuanced, right? Like you cannot easily encode this in like some deterministic checks, right?

确保最后确实生成了一份幻灯片，这是个确定性的检查。但如果你想检查这份幻灯片的质量怎么样，那就微妙多了，对吧?你很难把这种东西用一些确定性的检查编码出来。

[10:35]

And that's why we also have the second type of graders which is the model-based graders, right? And this is like rubric-based reasoning. Um So you for example say like is this slide high quality? Very generic, but that might be for example a rubric or like for example is this text coherent?

这也是为什么我们还有第二类 grader，也就是基于模型的 grader。它做的是基于 rubric 的推理。比如你可以问:这张幻灯片质量高吗?这很笼统,但它就可以作为一条 rubric;或者比如问:这段文字连贯吗?

[10:55]

Also a way to get some intel on how well your agent's performing. Um you can do some interesting things with this as well. Um pairwise comparison is in my opinion quite underrated. Let's say you have two examples, two outputs, um and then you basically ask the model which one of the two do you prefer and why?

这也是一种了解你的 agent 表现如何的办法。用它你还能玩出一些有意思的东西。在我看来,两两对比(pairwise comparison)是相当被低估的。假设你有两个样本、两份输出,你基本上就是问模型:这两个你更喜欢哪一个,为什么?

[11:13]

That's also quite interesting to get some information out of especially especially for these scenarios where you don't really have a clear way of of defining what makes the better one, right? Um And and then another one is the multi-judge consensus which is just for example you take like best of three and you say like three judges score independently and say like majority wins

这也很有意思,能从中挖到不少信息,尤其是在那些你没法明确定义"什么才算更好"的场景里。还有一种叫多评委共识(multi-judge consensus),比如就是取三选最优,让三个评委各自独立打分,然后少数服从多数。

[11:36]

for example, right? Once again, that this multi-judge consensus is interesting because it allows you to introduce some more determinism in a way where if you have like we know that an LLM is undeterministic, right?

对吧。这种多评委共识之所以有意思,是因为它能在某种程度上引入更多的确定性——我们都知道 LLM 是不确定的。

[11:49]

And the same would be happening for this model-based graders, right? Like if you run them like 100 times, a few times it might say, "Oh, this is great." And a few other times it might say, "Ah, it's not that good." If you have like this multi-judge consensus, you basically are assuming, let's put more compute into this and let's see what the majority of our graders consensus is, right?

基于模型的 grader 也是同样的情况。比如你跑它 100 次,有几次它会说"哦,这个很棒",另外几次又会说"嗯,不太行"。如果你用这种多评委共识,你基本上就是在说:咱们多投入点算力,看看这一群 grader 大多数人的共识是什么。

[12:08]

And this is unlocks a lot of things, right? Like this is flexible, this is scalable, this is nuanced. But as I said, it's non-deterministic, it costs more money, and also it requires some calibration, which we will see is not easy at all.

这就解锁了很多东西。它灵活、可扩展、又细腻。但正如我刚说的,它是非确定性的,更费钱,而且还需要做一些校准——后面我们会看到,校准一点都不简单。

[12:22]

And then finally, the most expensive one are the human graders. And these are probably the graders that when you're building these agentic systems, you will be using the least, right? Because they're like incredibly expensive.

最后,最贵的是人工 grader。在你搭建这些 agentic 系统的时候,这大概是你用得最少的一种 grader,因为它们实在太贵了。

[12:36]

Um you have like a whole subject matter expert that will do like a whole review of the system. It will It's expensive, it's slow as well, but it is more It's the highest quality. It is very nuanced. Um and yeah, it's like really good for like some AB testing and some spot checking, right?

你得请一位领域专家来对整个系统做一遍完整的评审。它又贵又慢,但质量最高、最细腻。是的,它特别适合用来做一些 A/B 测试和抽查。

[12:52]

So, I'm not sure like how many of you were able to clone the repo beforehand and have this all set up. Um I actually wanted to do this session a little bit differently, but given the amount of people, I will probably do a little bit more um myself instead of like letting you um think about all of the things. Um but I'll quickly give you an overview of of what's in the repo, right? Um let me

所以,我不太确定在座有多少人提前 clone 了这个 repo 并且都配置好了。其实我本来想用稍微不一样的方式来做这次环节,但考虑到人数,我可能会多自己来演示一点,而不是让你们去逐项思考。不过我先快速给你们过一遍 repo 里都有些什么。我来——

[13:13]

make this a little bit bigger. Um I have made some pre-made uh slides that I will show you in a bit. Um The resources is the main thing where you guys would be working in. So, you have the I'm actually closing this session for now. Um so if like the agent.yaml and this is basically where you would define your agent, right? Like um I think before we did a session.

把这个放大一点。我做了一些预先准备好的幻灯片,待会儿给你们看。resources 是你们主要会动手的地方。这里有个——我先把这个 session 关掉。比如 agent.yaml,这基本上就是你定义 agent 的地方。我想我们之前做过一个 session。

[13:34]

So this is basically what we're going to do use like the manage agent. So for the people who attended that session um before lunch, it's basically the same thing. We define here like an an an agent in this case um and we have given this uh the system prompt, right? So this is a system prompt that we're giving. So basically you are a slide generation agent and when the user gives

所以我们接下来基本上就是用这个 managed agent。对于参加过午饭前那个 session 的人来说,这其实是同一套东西。我们在这里定义一个 agent,这里就是,然后我们给了它 system prompt。这就是我们给它的 system prompt。基本上就是:你是一个幻灯片生成 agent,当用户给你——

[13:52]

you a topic, create a PowerPoint file at this location. And then also we tell it you have a shell um with Python PPTX uh pre-installed, right? Um so that's all we give it for now. And then we also have like an environment which we've defined um with like few packages um what is it what it needs to complete this session. Um And then basically that's it. We also

——一个主题时,就在这个位置创建一个 PowerPoint 文件。然后我们还告诉它:你有一个 shell,里面预装了 Python 的 python-pptx。我们现在就给它这些。然后我们还定义了一个 environment,里面装了几个包,就是完成这个 session 所需要的。基本上就这些。我们还——

[14:13]

have some other things defined, but I will get to that. I think maybe the first question that I have for the audience today is we want to make a slide generation agent, right? What do you guys think is a good eval? What are you trying to measure? What would be some good information that you want to get out of evals?

——定义了一些别的东西,这个我待会儿再讲。我想今天我要问大家的第一个问题是:我们想做一个幻灯片生成 agent,你们觉得什么才算一个好的 eval?你想衡量的是什么?有哪些是你希望从 eval 里得到的有用信息?

[14:36]

Sorry? Number of words on slides is is it a useful uh thing to track? And anyone else with some ideas? Sorry? Yeah, absolutely valid. Absolutely valid. Yeah, yeah, yeah. Um I I And this actually I like these two examples because they immediately give you like a different sense of um how you can use the type of grades. Like for example, the number of words on a slide is

抱歉,没听清?幻灯片上的字数——这是个值得追踪的有用指标吗?还有其他人有想法吗?抱歉?对,完全成立,完全成立。对对对。我喜欢这两个例子,因为它们一下子就让你感受到该怎么用不同类型的 grader。比如说,一张幻灯片上的字数是——

[15:04]

quantifiable, right? It's like easy to say you can count the number of words with like a deterministic grader with like a code grader. >> [snorts] >> The one if it's like overlapping or if it's overspilling, that one is harder to at um um encode in code, right? So for this one you might for example use a model grader. And that's exactly what we did, right?

——可量化的,对吧?用一个确定性的 grader、一个代码 grader,你很容易就能数出字数。而另一个,文字是不是重叠了、有没有溢出框外,那个用代码就更难编码出来了。所以对这种情况,你可能就会用一个模型 grader。而这正是我们做的。

[15:21]

So we have actually defined for you guys already a few graders beforehand. Two specific directories we have the code and we have judge. So, the code one is as I said it's like these these code graders they're quite deterministic like for example if we take a look at emoji count for example is one that we have defined where we basically just count the number of emojis present

所以我们其实已经预先帮你们定义好了几个 grader。具体分两个目录,一个是 code,一个是 judge。code 这个,正如我说的,这些代码 grader 相当确定性。比如我们定义了一个叫 emoji count 的,基本上就是数一下出现了多少个 emoji——

[15:43]

in the slide deck. Um because we we just noticed that it's quite prevalent. Like for example if I open the slide deck um let me go with environment one in this case. Um so these are the slides that I it's basically the agents running. Um it's it's done beforehand just because it takes can take quite a while um to get the agents running. Um but this is for example the result of the initial

——在整份幻灯片里。因为我们注意到这个现象挺普遍的。比如我打开这份幻灯片,这次就用 environment one 吧。这些就是 agent 跑出来的幻灯片。这是提前跑好的,因为让 agent 跑起来可能挺花时间的。不过这个就是最初那批 agent 的结果——

[16:07]

agents, right? So, this is slide number one. Um slide number two slide number three um with some weird things on the bottom left. Um slide four and slide five. Now, I I think we can all agree like this is not the best slide deck you guys have ever seen. Um but it's a good start. At least it does a slide deck um that's five slides. I think that's exactly the prompt that we sent it. Um

——对吧。这是第一张幻灯片,这是第二张,这是第三张——左下角有些奇怪的东西。这是第四张,第五张。我想咱们都能同意,这不是你们见过最好的幻灯片。但它是个不错的起点。至少它做出了一份五页的幻灯片。我想这正是我们给它的 prompt 要求的。

[16:36]

so we have a few slides. There's a few content on here it's like few boxes. It's you know, it's a slide deck. Um given these slides is there anything else that you guys are seeing that like this is something that we would never want in a slide deck?

所以我们有了几张幻灯片。上面有一点内容,有几个框。你知道的,它就是一份幻灯片。看着这几张幻灯片,你们还看到别的什么——这种东西是我们绝对不希望出现在幻灯片里的吗?

[16:58]

What was that? I No teal No teal. We can if you absolutely want to avoid teal that's absolutely right. I think in this it doesn't do that for every single slide. Like let me see for the career one. Um let me see what it is. Oh, okay maybe it does always use teal actually. Um but for example in this one we see like this overlap of like words and and and this this horizontal

刚才说什么?不要青色(teal)?不要青色。如果你真的非常想避免青色,那完全没问题。我觉得在这里它并不是每张幻灯片都用青色。我看看那张讲职业的。我看看是什么。哦,好吧,可能它确实总是用青色。不过比如这一张,我们看到文字重叠了,还有这条水平的——

[17:20]

what else do we have some weird coloring. Yeah, there's there's there's a few weird things happening generally, right? So yeah, based on this we take you take a look at what it is what the results are and you're like hmm what type of graders do I want to define for this specifically, right? And so we did that and we noticed for example emoji counts is one that's quite prevalent.

——还有什么,我们看到一些奇怪的配色。对,总的来说这里发生了不少奇怪的事。所以,基于此,你去看看结果是什么样的,然后你就会想:嗯,针对这个我想定义哪些类型的 grader 呢?于是我们就这么做了,我们注意到比如 emoji count 就是相当普遍的一个问题。

[17:44]

We want to check how many times do we see an emoji popping up. Another one is for example cluttered slides like how many shapes do we see on the slides like if there's just so too many things it becomes cluttered.

我们想检查一下 emoji 出现了多少次。另一个比如是幻灯片是否太杂乱——我们数一下幻灯片上有多少个形状(shape),如果上面东西实在太多,它就变得杂乱了。

[17:56]

Counting the number of slides for example we always ask for five slides making sure that you have five slides. Do we have slides with image, small fonts, text heavy slides. Now this is this is in this case it's quite arbitrarily chosen, right? These were just like things that we like thought were like this is quite representative of what a slide deck might have for graders,

再比如数幻灯片的页数——我们总是要求五张,那就确认你确实有五张。还有:有没有带图片的幻灯片、字号是不是太小、文字是不是太多。当然在这个例子里,这些选得相当随意。这些只是我们觉得"挺有代表性"、一份幻灯片可能会用到的 grader——

[18:18]

right? It really depends like I want really want to stress this like it really depends from use case to use case what makes a good grader, right? I think generally the way I think about this thing is if you have a grader that you get no useful out of information out of then you should not have that part of your evil, right? Like each thing you should be able to tell like for each

——对吧。这真的因人而异,我特别想强调这一点:什么才算一个好的 grader,真的是一个用例一个样。我大致的思路是:如果一个 grader 你从里面得不到任何有用的信息,那它就不该出现在你的 eval 里。每一项你都应该能说清楚——对于每一个——

[18:39]

single scenario that you're testing you should be able to say like this is the information that I want to get out of this. This is the type of this is the part of the system that I'm testing and this is how I can act on if it's being degraded, right?

——你正在测试的单独场景,你都应该能说出:这是我想从中得到的信息,这是我正在测试的系统的哪一部分,以及如果它出现退化我该如何应对。

[18:52]

So those were just like a few codes ones and then we also have a few judge ones for example the color judge which basically judged what's the color contrast and then it gives a score from like zero to five. Same with image, the layout, text. And this is the prompt that we give. Let me close this one real quick.

所以那些就是几个代码类的 grader,然后我们还有几个 judge 类的。比如颜色 judge,它基本上就是判断色彩对比度怎么样,然后给出一个 0 到 5 的分数。图片、布局、文字也是一样。这就是我们给它的 prompt。我快速把这个关掉。

[19:14]

Oh. So, let me keep it like this. So, this is basically the system prompt that we give it. Um no. So, we're saying, "Please evaluate the slide based on each of the following criteria. Text, the title should be simple and clear to indicate the main points. For main content, avoid too many texts and keep words concise. Use a consistent and readable font size, style, and color."

哦。我就让它保持这样吧。这基本上就是我们给它的 system prompt。呃,不是。我们说的是:"请根据以下每一条标准来评估这张幻灯片。文字:标题应当简洁清晰,点明主旨。正文部分:避免文字过多,用词要精炼。字号、字体样式和颜色要保持一致且易读。"

[19:36]

And I mean, it it goes on and on, right? So, we give like for each of the different things that we want to measure we give like a little information of like "This is what you should be focusing on when you want to measure this."

然后它就这么一条一条往下写,对吧?所以对于我们想衡量的每一项不同的东西,我们都给出一点说明,告诉它:"当你想衡量这一项时,你应该重点关注的是这些。"

[19:45]

Right? Okay, cool. So, we have these evals. Let's say you have now created a slide deck and you now want to see like, "Okay, what are the results, right? And how can we act on these results?" So, in this wrapper we also have created this nice little script that will automatically score your slide deck for you. And so, at the top here we basically have it all

对吧?好,挺好。所以我们有了这些 eval。假设你现在已经生成了一份幻灯片,接下来你想看看:好,结果怎么样?以及我们该如何根据这些结果采取行动?于是在这个 wrapper 里,我们还写了一个不错的小脚本,它会自动帮你给幻灯片打分。在最上面这里,我们基本上把所有——

[20:08]

listed out. So, we have like the slide count which is being counted the slides, the number of slides with image, text-heavy slides, cluttered slides, small font size, and so on. We also have our judges over here which are saying like they give a score from like zero to five uh based on like how good is the text, how good is the image at the layout and the color, right? Um

都列出来了。比如这里有幻灯片数量,就是统计有多少张幻灯片、有多少张带图、文字过多的、排版杂乱的、字号过小的等等。我们这边还有几个 judge,它们会从 0 到 5 打分,评判文字写得怎么样、图片怎么样、布局和配色怎么样,对吧?嗯。

[20:29]

uh honestly like these scores you can immediately note that these scores are quite high. So, as we said like we calibrated between zero and five. And as we see like the scores are being given you are like between 2.8 and four which honestly I think are quite high given the slide deck that we have seen, right?

说实话,你一眼就能看出来这些分数其实给得挺高的。前面说过我们把分数校准在 0 到 5 之间。可以看到这里打出来的分大概在 2.8 到 4 之间,但凭我们刚才看到的那套幻灯片来说,我觉得这分确实给得偏高了。

[20:45]

So, that's like the part of the calibration that needs to happen as well, right? I think there's also like one thing that I maybe want to stress. It's not because you have set up your evals once that they are now like the ground truth, you know? Um evals over time they can evolve. They need to be a living artifact. It's not like something you make once and then

所以这也是校准这一步需要做的事情。还有一点我想特别强调一下:并不是说你的 eval 一旦搭好了,它就成了不可动摇的标准答案,对吧?eval 是会随时间演进的,它得是一个活的东西,而不是你做一次就一劳永逸的。

[21:03]

forget and then use it like to make all of your future decisions on, right? Because like we will see over time like as I go through all of the different examples that we have, we will see like there needs to be a way also how we can see how we can make sure that the evals that we create are actually still measuring something useful for us, right?

不能做完就丢一边,然后以后所有决策都靠它来拍板,对吧?因为接下来我把我们手头各种例子过一遍的时候你会发现,我们还需要有办法去检查、去确认我们做的这些 eval 是不是真的还在衡量对我们有用的东西。

[21:21]

If you ever hear people talk about saturation of evals, that's basically what they mean in a way that like the eval is not giving any more relevant information that we can act on due to several reasons.

如果你听到有人说 eval 出现了 saturation(饱和),基本上指的就是这个意思:由于种种原因,这个 eval 已经给不出更多我们能据此采取行动的有效信息了。

[21:32]

Cool. So, we see this and I guess maybe the first thing that we want to do in this case is we want to make an agent that is a little bit more polished, right? And so for this, we actually just update our system prompts.

好。看完这些,我想这种情况下我们第一步要做的,大概就是把这个 agent 做得更精致一点,对吧?为此,我们其实只要更新一下 system prompt 就行。

[21:46]

So in instead of just having like, "Oh, you are slide generation agent, make a slide deck." We now give it a little bit more information of like what are the expectations that we have of you in terms of typography, right? Because as we noticed, we said like, "Oh, the font is too small. There's too many words on there. It's not readable or it's too big." Right? So we give it a little bit

所以不再只是简单地写一句"你是一个幻灯片生成 agent,做一套幻灯片出来",我们现在给它多一点信息,告诉它我们对排版(typography)有哪些期望。因为我们刚才注意到了嘛,说"哎,字太小了"、"上面字太多读不下去"、"或者字又太大了"。所以我们多给一点。

[22:04]

more of information. So we say like, "Slide title should be this size, section header should be this size, body this size, caption this size." Right? And we also give it some information on the layout and density. Like here are the things that we expect from the layout and density point of view. For example, we say, "Keep the body text concise, leave breathing room, and left

多给一些信息。比如我们说:"幻灯片标题该多大、小节标题该多大、正文多大、图注多大。"对吧?我们还给它一些关于布局和密度的说明,告诉它在布局和密度方面我们的期望是什么。比如我们说:"正文保持简洁、留出呼吸空间、段落左对齐。"

[22:23]

align paragraphs." Right? And then also we I think everyone kind of I mean I am at least getting like ticked off like if I read something that's clearly AI written, I'm always a little bit skeptical of if I can completely trust the content and if the person sending me this text is like has like read it himself or themselves and is standing behind that content, right? So we also say like

对吧?然后还有一点,我想至少我自己是这样:如果我读到一段一看就是 AI 写的东西,我总会有点怀疑——我能不能完全信任这内容,以及发给我这段文字的人到底有没有自己读过、是不是真的认可并为这内容背书。所以我们还加了一条。

[22:45]

avoid these AI generated tells as well. So never use a thin accent lines in the titles and don't pepper slide with emojis as decorative icons, right? So this is based on the things that we have seen in our eval, right? So we have seen as I Let's go back a little bit.

也就是避免那些一眼就能看出是 AI 生成的痕迹。比如:标题里绝不要用那种细细的装饰线条,也别在幻灯片上撒一堆当装饰图标用的 emoji。这些都是基于我们在 eval 里观察到的现象来定的。我们之前就看到了——咱们往回翻一点。

[22:59]

We've looked at this slide deck. And we're like, "Oh, this is not properly done. These fonts are a little bit off. Um, there's some emoji used in here. It's like a little bit all over the place." And then based on the score, we were like, "Okay, these are the things that we're clearly failing at, right?" So, we have like emoji counts four in this case, small font slides also four as

我们看了这套幻灯片,心想:"哎,这做得不太行,字体有点不对劲,这里还用了几个 emoji,整体有点乱七八糟的。"然后根据分数,我们就明白了:"好,这些就是我们明显没做好的地方。"所以你看,这里 emoji 数量是 4,小字号的幻灯片也是 4。

[23:20]

well, cluttered slides two, and text heavy slides, right? So, based on the information that we have gotten from the eval that we have run, we have made these changes to our new agent, right? Let me now pull up the result of the new agent that we have created in this case, right? Um, so this is slide one, which I think is immediately way more enjoyable to look at. Like, there's no

排版杂乱的有 2 张,文字过多的也有,对吧?所以根据我们跑这个 eval 拿到的信息,我们对新的 agent 做了这些改动。现在我把新做的这个 agent 的结果调出来给大家看。嗯,这是第一张,我觉得一眼看上去就顺眼多了。没有那种

[23:41]

overlapping stuff. Um, there's no dollar sign. There's just generally it's cleaner. This once again, I think this one still has like quite small text, but at least once again, we're like getting a little bit more consistent with the coloring as well.

互相重叠的东西,没有那个莫名其妙的美元符号,总体上就是更干净了。这张嘛,我觉得字还是偏小,但至少配色这块又稍微统一了一点。

[23:55]

Um, once again, like the whole slide deck is more consistent. This third slide, the fourth slide, and the fifth slide, right? And this is just by basically identifying, "Here's a few failure modes of our original one. Here's how we now make changes based on these things that we found in the system prompt, and now we run it back."

整套幻灯片又更协调了一些。第三张、第四张、第五张都是这样,对吧?而这一切,基本上就是靠识别出"原版有这么几个失败模式",然后"根据我们发现的这些问题在 system prompt 里做相应改动",再重新跑一遍。

[24:15]

And now once again, we can do the same thing. So, this is we're now basically in this loop of finding what's wrong, iterating, finding what's wrong, running it again, and making improvements over time.

然后我们又可以再做同样的事。所以现在我们基本上就进入了这么一个循环:找出哪里不对、迭代、再找出哪里不对、再跑一遍,就这样持续改进。

[24:25]

So, now we can take a look back at what we found over here. Oh, and this is actually way worse suddenly. We see like emoji count 20. I'm wondering where they are. I haven't seen them actually. Wondering where that is at.

现在我们回头看看这边发现了什么。哦,这下反而突然变得糟糕多了。我们看到 emoji 数量是 20。我都纳闷它们在哪儿,我实际上一个都没看见,搞不懂在哪儿。

[24:36]

Hmm. Wonder if this is like a mistake in this case. Um, but generally we see like, okay, small font slides, we've seen that, but we've we've improved upon the cluttering. Um, and let's see, text heavy, is that still the case?

嗯,我怀疑这是不是哪里搞错了。不过总体上看,好,小字号的幻灯片,这个我们之前看到过,但在排版杂乱这块我们改进了。再看看,文字过多——还是这样吗?

[24:51]

Um, I think I think that's fine. I mean, those are a little bit text heavy, but I think it's acceptable, right? So, now we're like So, this once again shows the value of like human review as well, right?

嗯,我觉得这个还行吧。这几张是有点文字偏多,但我觉得可以接受,对吧?所以现在我们就……这又一次说明了人工 review 的价值,对吧?

[25:04]

Because now we see, oh, these things that we have defined in our evals are maybe not as well defined as we hoped them to be, right? Because now I'm here arguing like, oh, this is not as text-heavy as I expected it to be, right?

因为现在我们发现,哦,我们在 eval 里定义的这些东西,可能并没有我们指望的那么严谨,对吧?因为我现在就在这儿争辩说:哎,这其实没我预期的那么"文字过多"嘛。

[25:15]

So, that means that something is actually wrong with the way we're grading. So, now we go back then, we would go back, go to our grader, change the grader, update it, and make sure that it better reflects the actual thing that we want to measure, right?

这就意味着我们的评分方式其实有点问题。所以现在我们就该回头,回到 grader 那里,改一改 grader、更新它,确保它能更好地反映我们真正想衡量的东西,对吧?

[25:28]

And this is also not something to be underestimated, like this calibration of how your agent should behave um and how your judges should judge the specific output is really something very fickle, right? Like you should spend like proper time trying to find the ways on how you should make this happen.

而且这一点也千万别小看:校准你的 agent 该怎么表现、以及你的 judge 该怎么去评判具体的输出,这事儿其实非常微妙、非常容易翻车。你真得花上像样的时间去琢磨怎么把这事做对。

[25:45]

Um let's say now that we want to have an agent. Like I I I I think with this one, I mean, it's fun. I think it's nice, um but it's still quite text-heavy and it's only text, right? Um let's say now that we want to have an agent. Let's say that's one of our requirements, right? That we have an agent that we always want to have includes diagrams. Once again, we go

嗯,假设现在我们想要一个 agent。这一版嘛,我是说挺好玩的,我觉得不错,但它还是文字偏多,而且全是文字,对吧?那假设现在我们想要一个 agent——假设这是我们的一条硬性要求,对吧?我们想要一个永远都会带上图表的 agent。那我们再一次回到

[26:05]

back to our system prompt, we update it, and we now say every slide must include at least one generated diagram or chart inserted as an actual image, right? Um so, once again, we update the system prompt or any part of the agent that you can tune, and then we go again and we check what do we get.

我们的 system prompt,更新它,这次我们写:每一张幻灯片都必须至少包含一张生成的图示或图表,并且要作为真正的图片插进去,对吧?所以我们再一次更新 system prompt,或者 agent 里任何你能调的部分,然后再跑一遍,看看结果如何。

[26:26]

Okay, so this one is quite interesting. I'm I guess personally I'm not a fan of having an image on the opening slide, but once again, it is what we defined that it should do, right? So, I'm going to let it slide. But it's it's a nice nice graph. What is saying? It's like no negotiation and active negotiation. So, it's arguing that if you do active negotiation for your salary, you can see

好,这张挺有意思的。我个人不太喜欢在开篇那张幻灯片上放图,但话说回来,这正是我们要求它做的事,对吧?所以我就睁一只眼闭一只眼。不过这图挺好看的。它说的是什么?"不谈判"和"主动谈判"。它想表达的是,如果你为自己的薪资主动去谈判,你会看到

[26:46]

over time the gap widens between no negotiation and yes negotiation negotiation. Some extra benchmarks. I I I this looks immediately way better just in the way that it's like kind of grounded into some actual facts right now instead of just waffling its way through the slide deck, right?

随着时间推移,"不谈判"和"谈判"两者之间的差距会越拉越大。还附了一些额外的 benchmark 数据。我觉得这一下子就好看多了,就因为它现在有真实的事实做依托,而不是在整套幻灯片里满嘴跑火车、空话连篇,对吧?

[27:04]

Yeah, this one I'm not a big fan of. I feel like it's a little bit stretched, but that might also just be the screenshot. Um Yeah, and this one also not the best one either, right? Let me see like let's see what the score.json also says, okay. No emojis, great. No cluttered slides. Still quite text heavy slides surprisingly. Um still small font slide. I think that's fine. I

嗯,这张我就不太喜欢了。我感觉它有点被拉伸了,不过这也可能只是截图的问题。嗯,这张也不算最好的那种,对吧?我来看看 score.json 又怎么说。好,没有 emoji,很好;没有排版杂乱的幻灯片。但文字过多的幻灯片居然还是挺多的,有点意外。嗯,还是有小字号的幻灯片。我觉得那个没事。我

[27:27]

think we just say like with images I think Yeah, I think we we accept like these types of things are fine. Um so once again, shows you some um questions regarding the grader that we have set up. But now we can also take a look at like the judges, right? Like for example because now we have images that we have created, so now we can also consider how does the image judge uh I think this is.

觉得我们就这么说吧——配上图之后,我觉得……嗯,我觉得这类情况是可以接受的。所以这又一次让你对我们搭的这个 grader 产生一些疑问。不过现在我们也可以来看看那些 judge,对吧?比如说,因为我们现在生成了图片,那我们就可以看看图片 judge 是怎么评的——我想这就是。

[27:48]

And it says it's 3.8 out of five. Um doesn't say give us a lot to go off, right? It just gives us a random number. What does it mean? How can we improve upon this? But that's fine for now. Now one thing that we always see that works just generally quite well, and that's like it's transversal over every single use case is adding a QA loop, right? Um for coding this is

它给的是 5 分里的 3.8 分。嗯,它没给我们太多可以参考的东西,对吧?就甩给我们一个随机数字。这数字什么意思?我们怎么照着它改进?但现在这样也行。还有一件事,我们总能看到它普遍效果相当好,而且几乎适用于每一种 use case,那就是加一个 QA 循环,对吧?在写代码这块,这个

[28:12]

quite intuitive. That's basically saying like you create an agent that actually is writing the code, right? And then you add the second agent that is then looking at the code that has been written and just criticizes it. So it's basically saying this is bad, this is bad, this is bad. This introduces a bug, this introduces a bug, this is not according to standards, whatever, right?

就相当直观了。基本上就是:你做一个真正负责写代码的 agent,对吧?然后再加第二个 agent,专门去看已经写好的代码,纯粹地挑毛病。它基本上就是说:这儿不行、那儿不行、这里又不行,这会引入一个 bug、那会引入一个 bug、这不符合规范,诸如此类,对吧?

[28:30]

So it basically is criticizing the the thing that has been created. And then that part of the feedback you give back to your original agent, the creation agent. The creation agent goes off again, does the creation, does the fine-tuning, makes the changes that were informed by the criticizing, and then once again after that is done, it goes back to the criticizing agent. And that loop

所以它本质上就是在批判已经做出来的那个东西。然后这部分反馈你再交回给你原来那个 agent,也就是创作 agent。创作 agent 又重新上阵,做创作、做微调,按批判 agent 给出的意见改动;改完之后,又再次交回给那个批判 agent。如此这般这个循环……

[28:50]

basically goes on and on and on until both sides are like, "Okay, this is fine. We can ship this." And that's basically what we now do in this next step. Um so, we basically say like, "Okay, require QA loop. Um assume there are no problems.

基本上就这么一轮一轮地来回过招，直到双方都觉得「行了，这没问题，可以发布了」。这其实就是我们接下来这一步要做的事。所以我们大致会这么说：「好，要求一个 QA 循环。假设没有问题——」

[29:06]

Um oh, assume there are problems, and then your job is to find them. Approach QA as a bug hunt, not a confirmation step." And this is quite interesting, because we're like actively instructing the agent to behave in a way adversarially adversarially, right? Like we're saying like, "There are issues.

呃不对，是「假设存在问题，而你的任务就是把它们找出来。把 QA 当成一场抓 bug 的狩猎，而不是走个确认的过场。」这一点挺有意思的，因为我们是在主动指示这个 agent 用一种对抗性的方式去做事，对吧？我们等于在说：「这里有问题。」

[29:21]

You need to find them." It's not It's not like, "Oh, there might be something. You might be interested in finding something." No, it's actively saying, "There are issues. Go find them." Um and then we say like we instruct after writing the deck, "Okay, convert it to images. Inspect every slide image yourself. Fix the issues, re-render, re-inspect, and then do not stop until

「你必须把它们找出来。」不是那种「哦，也许有点什么，你也许有兴趣去找找看」。不，是明确地说：「这里就是有问题，去把它们找出来。」然后我们在写完这套幻灯片之后会指示它：「好，把它转成图片。自己逐张检查每一张幻灯片的图。修掉问题，重新渲染，重新检查，并且不要停下来，直到——」

[29:39]

you've completed at least one fixed and verified cycle." Cool. Now, as I said, I think for coding this is quite intuitive, but I think it's also quite intuitive if you take a look at like um the the slides that we have created, right? Because that's basically what we did. We have looked at the slides, and we're like, "Ah, this is not good. This is not good. Let's take that feedback,

「你至少完成了一轮完整的『修复并验证』循环。」很好。正如我刚才说的，我觉得这套做法对写代码来说相当直观，但其实如果你看看我们刚才生成的那些幻灯片，它同样很直观，对吧？因为我们做的就是这件事：我们看着这些幻灯片，心想「啊，这个不行，那个不行。把这些反馈拿过来，」

[29:59]

update our graders, update our system prompt, and let's run it back again, right?" So, let's now see if we can actually if this is actually showing some improvements. Um I think this is immediately a lot better. So, the the the image is way bigger now. I think it's way more readable even from a further distance away. Um still the slides are small, but it's like for example, it's source

「更新我们的评分器（grader），更新 system prompt，然后再跑一遍。」那我们现在就来看看，这是不是真的带来了一些改进。我觉得这一下子就好多了。这张图现在大多了，即使站得远一点也清楚得多。当然幻灯片本身还是偏小，但比如说，它有标注来源——

[30:20]

known. There's a source over here as well, which is quite good. Um I think this is also way better. It is more clearly structured. I think the image is also a little bit better as well, right? A quite interesting graph in this case, uh your value profile versus team average.

标了出处。这边也有个来源，这点挺好。我觉得这一张也好很多，结构更清晰了。图片我觉得也稍微好了一点，对吧？这个例子里有一张挺有意思的图表——你个人的价值取向画像对比团队平均值。

[30:37]

This one is still a little off in my opinion. Also, we now have like a little introduction of like these weird ticks. Um And this one is also a little bit better, I would say, but I think like the just the image taking is um kind of messing with the slide here. And so then we kind of know the drill by now. We take a look at the score.

在我看来这一张还是有点不对劲。而且现在还冒出来一些奇怪的小刻度线。这一张我会说也稍微好了一点，但我觉得就是截图这个过程有点把这张幻灯片搞乱了。那么到这里我们大概也都摸清套路了：我们去看分数。

[30:56]

We see like has it improved? Why do we see still gaps? And now we see like for all of the judges that we have created, it is higher than uh the ones before, right? We're now all good in the four .2 to 4.4 now. Um so we're on a we're on a good track, right?

我们看：它有没有改进？为什么还能看到一些差距？现在我们能看到，我们创建的所有那些评判器（judge），分数都比之前高了，对吧？现在都落在 4.2 到 4.4 这个区间了。所以我们走在一条不错的路上，对吧？

[31:11]

And um you can keep on doing this. You can keep on doing this. Um And you will always make like these little changes, but sometimes and this is I guess where it gets quite interesting and more like more like nuanced is you can also just go to a smarter model, right? Because like now you're like defining, oh, this is what a good slide should look like.

而且你可以这样一直做下去，一直做下去。你总能做出这些小小的调整，但有时候——我猜这里就开始变得相当有意思、也更微妙了——你也可以干脆换一个更聪明的模型，对吧？因为你现在是在定义「哦，一张好的幻灯片应该长这样」。

[31:33]

This is what it should do. This is what not what you should not do. But with these models getting smarter and better over time, you kind of expect them to be like able to figure that out on their own, right?

「这是它应该做的，这是它不应该做的。」但随着这些模型越来越聪明、越来越强，你其实可以期待它们自己就能搞明白这些，对吧？

[31:44]

Um I mean, that would at least be nice. So, that's what we tried out as well. So, now in the last one we basically just changed our model to Opus 4 7 instead of Sonnet 4 7, which we have used up to this point. If you can uh 4 6. Um So, now we have switched to Opus 4 7, and we have basically just given it a simple prompt again. Like you are a slide generation agent, and then when

我是说，那样至少会挺不错的。所以这也是我们试了一下的东西。在最后这一版里，我们基本上只是把模型换成了 Opus 4.7，而不是我们之前一直在用的 Sonnet 4.7——呃,是 4.6。所以现在我们换成了 Opus 4.7,而且我们基本上又只给了它一个很简单的 prompt:你是一个幻灯片生成 agent,然后当——

[32:07]

the user gives you a topic, create a PowerPoint file at whatever and then you have a shell. So, it's basically just the initial prompt that we gave to uh our Sonnet model in the beginning, right? And then once again, let's now consider taking a look at the results of those.

当用户给你一个主题时,在某某路径下创建一个 PowerPoint 文件,然后给你一个 shell。所以这基本上就是我们一开始给 Sonnet 模型的那个最初的 prompt,对吧?那么我们再一次来看看这些的结果。

[32:24]

And this is just a base prompt, right? Like you can immediately see like it's significantly better than the Sonnet one, right? I think there's still clear issues that we can iron out, but generally like it's It's more structured, right?

而这只是个基础 prompt 而已,对吧?你一眼就能看出来,它明显比 Sonnet 那一版好多了,对吧?我觉得还是有一些明显的问题可以再打磨,但整体来说它更有结构了,对吧?

[32:37]

And then we can take a look at the score as well. And I think this is quite interesting and quite telling. Like for example, Opus just does not use any emojis. Like it kind of knows like if you want to make a slide deck about salary increase, emojis are probably not right place to put them, right?

然后我们也来看看分数。我觉得这一点相当有意思、也相当说明问题。比如说,Opus 干脆一个 emoji 都不用。它差不多就知道——如果你要做一套关于加薪的幻灯片,emoji 大概不是该放上去的东西,对吧?

[32:54]

Um it also has like fewer small font slides because it's kind of has like this innate knowledge of, "Okay, it should be readable. This is how a slide deck should function. This is what people expect out of a slide deck."

它字体过小的幻灯片也更少,因为它天生就有那种「好,这东西得能看清。一套幻灯片应该这样运作。人们对一套幻灯片的预期就是这样」的内在认知。

[33:04]

Right? And then we get to these judge graders, right? Um we see a 4.4. We see a five for the images. Do we even have an image in this one? I don't think it we do actually. No, we don't. Okay, but then once again, we got a five in this one. Layout judge 4.2 and then the color judge 4.8 and title body coherence 4.4. So, this is like immediately giving like extremely

对吧?然后我们看到这些评判型评分器(judge grader)。我们看到一个 4.4,图片那项看到一个满分 5。这一版里我们到底有没有图片来着?我觉得没有……其实,确实没有。好吧,可即便这样,这一项我们还是拿了个 5。布局评判 4.2,然后配色评判 4.8,标题与正文一致性 4.4。所以它一上来就给出了极其——

[33:26]

high scores as well, right? Which I think is quite interesting because like this is once again showing that we might not be measuring the right thing. And this is not too unexpected for these types of um graders, right? Or for these judge graders. I think one of these things with like Okay, let's go to the code grades. I think those are quite straightforward. I

高的分数,对吧?我觉得这相当有意思,因为它再一次说明:我们衡量的可能并不是对的东西。而对这类评分器、或者说这类评判型评分器来说,这种情况也并不太出人意料。我觉得其中一件事就是……好,我们来看看代码型评分器(code grader)。我觉得那些相当直白。

[33:46]

think most people in the room would have understood by now like how they work and how what we can do with them. Like for example, emoji count, it's quite simple. Just count the number of emojis and that's it. But with this judging, what we have done here is actually quite problematic.

我想在座大多数人到现在应该都明白它们是怎么运作的、我们能拿它们做什么了。比如说 emoji 计数,这相当简单,就数一数 emoji 的数量,就这样。但说到这种「评判」,我们这里所做的其实相当有问题。

[34:03]

We basically say like give a score from zero to five and for text well, the text should uh the title should be simple and clear to indicate the main point for main content. Avoid too many text and keywords.

我们基本上是说:给出一个 0 到 5 的分数,对于文字嘛——标题应当简洁明了,点出主旨;对于正文内容,避免堆砌太多文字和关键词。

[34:14]

But it has nothing to anchor on, right? Like it doesn't really know what good looks like in this case. It doesn't know what bad looks like. So, there's still like this trade-off between like what does a model actually know and what do we need to give more information on to the model to make sure that it can give like a proper um proper judging of what we actually have

但它没有任何可以作为参照的锚点,对吧?它其实并不知道在这种情况下「好」到底长什么样,也不知道「差」长什么样。所以这里始终存在这样一种取舍:模型本身到底知道什么,以及我们需要再额外喂给模型多少信息,才能确保它对我们实际产出的东西给出一个像样的——一个像样的评判。

[34:35]

produced, right? So, for example, in this case, I would for example say what could help is say like, "Oh, this is a bad example. Like let's say you have a zero. Like everything is just awful. These are some telltale signs that you're dealing with an extremely badly formatted slide deck." And then like over time, different ranges you can kind of express.

对吧?所以举个例子,在这种情况下,我会说能起到帮助的做法是,告诉它:「这是一个差的例子。比方说这是个 0 分——一切都糟透了。出现这些征兆,就说明你面对的是一套排版极其糟糕的幻灯片。」然后随着时间推移,你可以把不同的分数区间一段段地描述出来。

[34:58]

And then once over time, and then once again, like that doesn't mean it will still be able to give like a good answer because we now have these results. We have this number that our LLM decided to output for some reason. Like for example, in this in this case, it just put out five.

然后随着时间推移——再说一次,这并不意味着它就一定能给出一个好答案,因为我们现在拿到了这些结果,拿到了这个我们的 LLM 出于某种原因决定输出的数字。比如说在这个例子里,它就直接给了个 5。

[35:17]

Okay, what do we do with that number now? Okay, it's a five. We we we just said there was no not a single image in the slide deck, right? So, how can we interpret this five? One way of doing this is just basically always asking your judge graders to give reasons why it came to that conclusion, right? And one thing that should be very like cautious about is the ordering, right?

好,那我们现在拿这个数字怎么办?好,是个 5。可我们刚刚说了,这套幻灯片里一张图片都没有,对吧?那我们要怎么去解读这个 5?一种办法就是,基本上永远要求你的评判型评分器给出它得出这个结论的理由,对吧?而有一件事你要非常非常小心,那就是顺序,对吧?

[35:41]

I've had it happen while I was like setting this up. And I did like this exact thing. So, I had like the number and then I said like, "Okay, give me also reasons why you did that." And so then it said like, "Oh, it's a four and the reasons for this are these these and these."

我在搭这套东西的时候就碰到过这种情况。我当时就是这么干的:我先让它给出那个数字,然后我说「好,再告诉我你为什么这么打分」。结果它就说「哦,这是 4 分,理由是这些、这些和这些」。

[35:53]

But we know that an LLM it works auto regressively, right? So, if it is anchored on like this four, it will do anything it can to argue why it should be a four, right? Anything. Even if it's like extremely bad. If it's like if it should be like a one, it will still say, "Oh, it is good for these and these reasons." Because it needs to justify the four that it put out.

但我们知道,LLM 是自回归(auto-regressive)地工作的,对吧?所以一旦它被锚定在「4 分」上,它就会想尽一切办法去论证为什么这应该是 4 分,对吧?不管怎样都行。哪怕它其实糟糕透顶,哪怕它本该是 1 分,它照样会说「哦,它在这些、那些方面都不错」。因为它得为自己刚刚给出的那个 4 分找理由。

[36:13]

So, once how you do it is you actually turn it around. So, first you say like, "Give me a bunch of reasons. Give me pros. Give me cons. Give me reasons why it should be high. Give me reasons why it should be bad." And then based on all of those reasons together, then you need to make your final decision on the output, right?

所以正确的做法是把顺序反过来。先说:「给我一堆理由。给我优点,给我缺点。给我它分数该高的理由,给我它分数该低的理由。」然后基于所有这些理由综合起来,你才去做出最终的输出判断,对吧?

[36:29]

And that's also that goes also back to like this QA loop as well. Um because then once again, you can get a little bit tricky where you have like multiple agents also doing the verification part where you have like one agent that is like finding all of the issues, and then the other one is like refuting those.

这一点也呼应回前面那个 QA 循环。因为你在这里同样可以玩点花样:让多个 agent 来做这个验证环节,比如让其中一个 agent 专门去找出所有问题,另一个 agent 则去反驳这些问题。

[36:45]

For example, one example that I can give, which I think is quite interesting. Let's say you want to make um a document for um where you need to like some analysis. First you need to get a lot of context from the internet, for example, like on a legal document, for example, right? Um and you ask the you ask a model to like make a summary of um a certain case, what was decided,

举个我能想到的例子,我觉得相当有意思。假设你要做一份文档,你需要在里面做一些分析。你先得从互联网上获取大量背景信息,比如说一份法律文书,对吧?然后你让模型去总结某个案件——里面判决了什么,

[37:07]

what does this have for legal implications for other cases, right? You need to be very careful with like all of these things that like legal cases are generally like quite tricky. And like an agent would love to create like, "Oh, this and this." and jump to conclusions like, "This is the reason."

这对其他案件又有什么法律层面的影响,对吧?这类东西你都得非常小心,因为法律案件通常都相当棘手。而一个 agent 特别喜欢张口就来「哦,这样这样、那样那样」,然后草率地下结论:「这就是原因。」

[37:19]

and that's it, right? And then the grader might be like, "Oh, this is unclear. This is um maybe not as this is maybe uh untrue. This is maybe uh maybe like glossing over the actual facts." All of those type of things, right?

「就这样了。」对吧?这时评分器可能就会说:「哦,这里说得不清楚。这里也许……可能不太对。这里也许是在掩盖真正的事实。」诸如此类的种种,对吧?

[37:34]

But then once again, you can like apply these multiple techniques. You can have like multiple graders, for example, seeing like um evaluating those and seeing like what are the main ones popping up. Um because once again, a grader might still hallucinate things as well, right? Especially in like the very nuanced scenarios, right? So, there's like different ways of how you then can

不过呢，你又可以把这几种技巧结合起来用。比如你可以设多个 grader，让它们分别去评估，然后看看主要冒出来的是哪些问题。因为说到底，grader 本身也可能会产生幻觉，对吧？尤其是在那些非常微妙、细致的场景里。所以接下来你有好几种不同的方式可以去

[37:53]

work with these judges to make sure that you actually get like good, consistent output that is actionable, right? And what I've shown you here today is basically just a small introduction to how evals can help you, but it's definitely not the end. I think 45 minutes for a session on evals is, in my opinion, quite [clears throat] short because it can get really deep, right?

运用这些评判器，来确保你真正拿到那种高质量、一致、而且可落地执行的输出，对吧。我今天给你们展示的，基本上只是 evals 如何帮到你的一个小小入门，但这绝对不是全部。我个人觉得，用 45 分钟来讲 evals 其实相当（清嗓子）短，因为这个话题真的可以挖得非常深，对吧？

[38:15]

Because like I started this off this session with talking about benchmarks, which are in the end just evals. And every single time, why would every single model provide or care so much about benchmarks, so much about evils, if it wasn't one of the main important things when we are building new models, right?

因为我这一场是从聊 benchmark 开始的，而 benchmark 说到底也就是 evals。每一次都这样，如果它不是我们构建新模型时最重要的事情之一，那为什么每一家模型厂商都会那么在意 benchmark、那么在意 evals 呢，对吧？

[38:36]

Exactly. We need to find the things that we are failing at. Exactly. We need to find things. What are we good at? What are we bad at? How can we make the model better in future generations? And that's the same thing when building applications that are consisting that's using AI agents, right? It's the same thing. It's just finding what works, finding what doesn't, iterating, and

没错。我们需要找到自己做得不好的地方。没错，我们要找出这些东西。我们擅长什么？我们不擅长什么？我们怎样才能在未来几代里把模型做得更好？而在构建那些用到 AI agent 的应用时，道理是一模一样的，对吧？完全是一回事。就是去找出哪些有效、哪些无效，然后不断迭代，

[38:56]

making sure that the changes that you're making that you're informed on the decisions that you are making, and making sure that the changes you make have actually positive influence on your final outputs. Okay. Thank you guys so much. This is all the time I have. Thank you guys.

并确保你所做的改动，是建立在你对决策有充分了解的基础上，确保你做的这些改动真的对最终输出产生了正面的影响。好，非常感谢大家。我的时间就到这里了。谢谢各位。