How to Build Better AI Evals with Claude Code in 5 Steps | Shreya & Hamel

频道: Peter Yang
视频: https://www.youtube.com/watch?v=bdMHQLvtVaQ
原文语言: en
统计: 共 222 轮 · Hamel 50 · Peter 62 · Shreya 59

[0:00] Hamel

You want to externalize your taste and your judgment before you start writing evals.

你得先把自己的品味和判断标准外化出来，然后再动手写 evals。

[0:04] Peter

No matter how many stuff I add and how many evals I add, it cannot get it perfect in one shot.

不管我往里加多少东西、加多少条 eval，它就是没法一次就做到位。

[0:09] Peter

Do you have the bottom up evals? I think this is where my question mark comes up for me. [music] Bottom up is like data driven. Top down is not about the data. Claude is very very bad at coming up with bottom up evals. That's all you. The interface that it comes up with is going to be so much better than me looking at my data in say Google spreadsheets [music] or something. We joke that writing is the final boss for LLMs. To like have it right like you in a way that you are satisfied with is extremely difficult. Hey everyone, my guests today are Hambo and Shrea, the instructors behind the most popular AI eval course online. I think they've taught over 4,500 students now. So, they're here to share a really practical example about how to use AI to automate evals the right way without doing a whole bunch of slop. And uh yeah, really excited to have you both.

自下而上的那半套 evals 你有没有？我的问号就出在这儿。（音乐）自下而上是数据驱动的，自上而下跟数据无关。Claude 特别不擅长想出自下而上的 evals，那部分只能靠你自己。它给你搭出来的那个界面，会比我自己在 Google 表格里翻数据强太多。（音乐）我们老开玩笑说，写作是 LLM 的最终 boss——要让它写得像你、写到你满意，难度极高。大家好，今天的嘉宾是 Hamel 和 Shreya，他们是全网最火的 AI eval 课程的两位讲师，我记得到现在已经教了 4500 多名学员。今天他们来分享一个特别实操的例子：怎么用 AI 把 evals 自动化，而且是用对的方法，不产出一堆垃圾。嗯，非常高兴请到你们俩。

[0:56]

Really excited. Likewise. [laughter]

我们也很期待，同感。（笑）

[0:58] Peter

Great. Okay. So, so Haml, last time you came on the podcast, I feel like I learned a lot from you. I think some of the lessons were manly review customer traces and conversations to figure out where things are going wrong. Use like yes, no, past fail evals instead of making up random scores. But maybe before we get to the demo, can you guys talk about at a high level about how eval like last year?

太好了。那咱们开始。Hamel，上次你来播客，我觉得自己学到特别多。印象比较深的几条是：主要靠人工去看用户的 trace 和对话，找出哪里出了问题；还有别乱编一堆分数，直接用是/否、通过/不通过这种 eval。不过在进 demo 之前，你们俩能不能先从大方向聊聊，evals 这块跟去年比有什么变化？

[1:19] Hamel

So, the fundamentals still apply. You still want to start everything by looking at data and you want to do error analysis and you want to have a structured approach to looking at data so that you can externalize your taste and your judgment before you start writing eval. The main thing that has changed is getting agents to help you look at the data in a very thoughtful way. So agents have become a lot more powerful over time and we have figured out a workflow that allows you to kind of have agents running in the background while you look at data supporting you so that you can have a lot more leverage in that first stage of looking at data. And that's what Treya is going to show today is an example of that.

基本功还是那些。一切还是从看数据开始，还是要做错误分析，还是要有一套结构化的方法去看数据——目的是在动手写 eval 之前，先把你的品味和判断外化出来。真正变了的，是现在可以让 agent 帮你更有章法地看数据。agent 这一年多变强了很多，我们也摸出了一套工作流：你在前台看数据，agent 在后台跑着给你打配合，这样在「看数据」这第一步上，你的杠杆率能高很多。今天 Shreya 要演示的就是这么个例子。

[1:57] Shreya

The other thing I'll add to that is that um we we teach LLM judges a lot in our course. LLM judges are a way to have an LLM look at a trace and make a judgment about a very specific failure mode. So for example, uh is is this output too long or is this output too short? Does it follow a particular structure? Um LLM judges have gotten very good at evaluating these very well- definfined criteria more accurately. So we are much bigger fans I would say of LLM judges.

我再补一点。我们课上花了不少篇幅讲 LLM judge。LLM judge 就是让一个 LLM 去看一条 trace，然后针对某一个非常具体的失败模式做判断。比如说，这段输出是不是太长了？是不是太短了？有没有按规定的结构来？现在的 LLM judge 在评这类定义得非常清楚的标准时，准确率已经高多了。所以我们现在对 LLM judge 的态度，比以前正面得多。

[2:29] Peter

Got it. Since I have two ex experts here, I'd love for you guys to take a look at my pretty pretty dumb evals for a skill that I built and maybe give me some feedback. Basically, I built a skill for podcast post-production and part of the skill is it writes a bunch of it if tried to find the top 10 takeaways from the podcast and then it writes an initial draft as a newsletter post and I built a bunch of evals for for this or I as in claude a bunch of evals for this. So uh so for example if you go down here there's some evals around like are the takeaways the right length between 240 and 330 characters and then there's also some like judgment things is each takeaway understandable on its own without watch the episode that does it give plain useful advice so so I guess there's kind of two types of take two types of eos right one is just like checking if the character lens is correct and the other one is more like judgment based but yeah do you guys have any thoughts about this or like how how can I make this better? Yeah, this is awesome. I love that you have a real demo for us to take a look at. There's some things that I want to back up a little bit on, which is what's in the skill, right? If we're going to start doing evals, it's very hard to do evalu.

明白。既然请到了两位专家，我想让你们看看我给自己做的一个 skill 配的那套挺笨的 evals，给我点反馈。事情是这样：我做了一个播客后期制作的 skill，它的一部分工作是从播客里挑出十条最值得记的要点（takeaway），然后写一版 newsletter 初稿。我还给它配了一堆 eval——准确说，是让 Claude 给它写了一堆 eval。比如你往下翻，有些 eval 是管长度的，比如每条 takeaway 是不是落在 240 到 330 个字符之间；另外还有一些偏判断的，比如每条 takeaway 是不是不看那期节目也能读懂、是不是给出了朴素又有用的建议。所以我这套 eval 大概分两类：一类就是查字符数对不对，另一类更偏主观判断。你们怎么看？我该怎么把它做得更好？挺好的，我特别喜欢你带了个真实的 demo 过来。不过有几件事我想先往回退一步问：这个 skill 里到底写了什么？因为要是直接开始做 eval，光凭这些是很难评的。

[3:40] Shreya

And uh what was your I love that you tried to make evals. So then kind of what was your thought process on making the evals? Was it like I'm just going to ask Claude to make some emails for me or I don't know you saw some outputs or you you saw some things that you noticed were like off. Um and then you wanted to create emails around that or like tell us a little bit more about that.

还有就是——我很欣赏你真的动手写了 eval。那你当时是怎么想的？是那种「我就让 Claude 帮我编几条 eval」，还是说你看了一些输出、发现某些地方不对劲，然后才想着围绕这些问题去建 eval？多跟我们讲讲这个过程。

[4:01] Peter

That's a good question. Yeah. So um let me pull up my little blog post. Um so basically I do these interviews and uh there's a video and then the people who want to watch video I have a bunch of takeaways down here. So that that's basically what the skill does. it it produces these draft takeaways and uh you know just for people watching I I don't just paste the stuff in and call it a day. I also manually review and go back and forth with it. That that's basically what it does. So let let me show you a real example. So so like it's producing these takeaways for interview I did with Tharic from Anthropic and then I I I just give like a bunch of feedback like hey go change all all this stuff [laughter] and then did it again and then here's some more feedback right and and this is after running the skill. This is after running a skill and the email. So it no matter how many uh stuff I add and how many emails I add, it it cannot get it perfect in one shot because like I guess each interview is different and it kind of goes back and forth. Yeah. But basically the output is like this thing.

好问题。我把我那个博客页面调出来。基本上我会做这些访谈，会有一期视频，然后为那些不想看视频的人，我在下面放一堆要点。这就是这个 skill 干的活——它产出这些 takeaway 初稿。跟大家说明一下，我不是把它生成的东西一贴就完事，我还会人工过一遍、来回改。它大概就干这些。我给你们看个真实例子。比如这是我跟 Anthropic 的 Thariq 那期访谈，它生成了这些 takeaway，然后我一顿反馈：喂，这些地方全给我改掉（笑），它改了一版；然后我又给了一轮反馈。注意这已经是跑完 skill、跑完 eval 之后的状态了。所以说，不管我往里加多少东西、加多少条 eval，它就是没法一次到位——我猜是因为每期访谈都不一样，总得来回磨几轮。但最后的产出大概就是这个东西。

[4:54] Shreya

Yeah. So I think you hit the nail on the head with one takeaway which is you know eval are iterative. It's a work in progress and by seeing more and more examples you're going to see more failure modes give more feedback that you kind of want externalized into those eval. So that makes sense. Um so let me start with kind of there are two halves of evals that you definitely should have. One half is like the top down evals and the idea here is if you think very critically about the nature of the task for example podcast takeaway generation um at a very high level what makes for good takeaways. So these are absolutely things like word length. These are absolutely things like make sure you're using action verbs correctly or make sure these are actionable takeaways for humans. Um, so think of top down as what if you're in a vacuum just given the task description, what would you come up with? And I think Claude does a very good example with help or does a very good job of helping you with these top down um, evals. Now bottom up evals are the other half. Bottom up evals means when you look in lots and lots and lots of sample outputs,

对。我觉得你说到点子上了：eval 本来就是迭代出来的，永远是个半成品；你看的样本越多，就会发现越多失败模式、给出越多反馈，而这些反馈正是你想外化进 eval 里的东西。这个逻辑是通的。那我先给你一个框架：eval 有两半，两半你都必须有。一半是自上而下（top-down）的 eval。思路是你先非常认真地想清楚这个任务的本质，比如「播客要点生成」，站在很高的层面看，什么才算好的 takeaway。字数长度绝对算一条；「动词要用得对」「必须是人真能落地的建议」这些也绝对算。所以自上而下就是：假设你处在真空里，手上只有一份任务描述，你能想出哪些标准。我觉得 Claude 在帮你做这半套自上而下的 eval 上，表现非常好。另一半是自下而上（bottom-up）的 eval，意思是当你把大量大量的样本输出翻一遍之后——

[6:01] Shreya

what is your gut vibes and feedback that you also want externalized into evals? So these are the things that you're talking about when you're iterating with Claude on uh the takeaways itself. These are things that you come up with that you want externalized into the eval. Now Claude is very very bad at coming up with bottomup evals. That's all you um and that's also why it it occurs over time. So when I look at your skill here, what I'm mentally thinking is do you have the top down evals? It seems like you really have it. Um and do you have the bottom up evals? I think this is where my question I the question mark comes up for me because I would love to know you know which which of these bullets are things that are bottom up. Uh maybe are how do you do you feel confident that it is exhaustive of all the feedbacks that you've had over several different podcasts um and so forth.

——你的直觉、你的体感反馈是什么？这些同样要外化进 eval 里。也就是你刚才说的、你跟 Claude 来回磨 takeaway 时冒出来的那些东西。那是你自己长出来的判断，得写进 eval。而 Claude 特别特别不擅长想自下而上的 eval，那部分完全靠你自己，这也是为什么它是随着时间一点点积累出来的。所以我看你这个 skill 的时候，脑子里在想的是：自上而下那半套你有没有？看起来你确实有。那自下而上那半套呢？我的问号就在这儿——我很想知道这些条目里哪些是自下而上来的；还有，你有没有把握说，它已经穷尽了你做过好几期播客攒下来的所有反馈？

[6:51] Peter

That's actually a really good point. So basically my my my loop is like you know I run this skill for like a raw transcript and it produces a bunch of output and then I go back and forth with it and and I you know finally we get to a result that I'm happy with right then I tell you like hey okay now now reflect on our entire conversation and and like suggest what kind of updates you'll make to the skill and the eval so that we don't have to go through this whole process again

这点说得真好。我的循环大概是这样：我拿一份原始转录去跑这个 skill，它吐出一堆内容，然后我跟它来回改，最后改到我满意为止。到那个时候我会跟它说：好，现在你回顾一下我们整段对话，看看该怎么更新 skill 和 eval，让我们下次不用再走一遍这么长的流程。

[7:14] Peter

right

对吧。

[7:14] Shreya

exact that's perfect that's amazing I think it's really great that you do that

没错，这个做法太对了，特别棒。我觉得你能坚持这么做真的很难得。

[7:18] Peter

but but I guess one thing I'm worried about is it it'll just kind of overfit things so so for example we we'll do one podcast and then let's say the takeaways are not mei like mei is like this consulting thing right it's like they're not mutually exclusive and they're not comprehensive and then it adds this eval but then the next time I come and do this then actually the me thing is maybe not that important for for this interview and then it just like overfits to

但我有个担心，就是它会过拟合。比如说我们做完一期播客，假设这次的 takeaway 不够 MECE——MECE 是咨询圈那套说法嘛，就是「相互独立、完全穷尽」——然后它就加了一条这样的 eval。可下次我再做另一期，MECE 这件事对那期访谈可能根本没那么重要，它就这么过拟合到——

[7:40] Peter

you know this episode is brought to you by whisperflow whisperflow saves me at least 3 hours a week and is one of my favorite AI apps by far it's just so much faster to dictate to AI using your voice than to type. You just talk naturally and it outputs clean, ready to send text. Whisper Flow even removes filler words and formats your sentences for you. I use Whisper Flow for everything, including drafting newsletter posts, writing product specs, replying on Slack, and more. It works on Mac, Windows, iPhone, and Android across all of your favorite apps. Try a free at whisperflow.com and use my code peterwisperflow to get six months free. That's peterwisperflow. Now, back to our episode.

本期节目由 Wispr Flow 赞助。Wispr Flow 每周至少帮我省下 3 个小时，是我最喜欢的 AI 应用之一。用语音跟 AI 说话，比打字快太多了。你只要自然地讲，它就输出干净、可以直接发出去的文字。Wispr Flow 还会自动帮你去掉口头禅、把句子格式化好。我什么都用它：起草 newsletter、写产品 spec、回 Slack 消息等等。它支持 Mac、Windows、iPhone 和 Android，在你常用的各种 App 里都能用。去 wisprflow.com 免费试用，用我的优惠码 peterwisprflow 可以免费用六个月。就是 peterwisprflow。好，我们回到节目。

[8:19] Shreya

Yeah. So one advice that I always give is actually in the skill itself separate your evals into top down and bottom up. Then the other thing in the skill is if you have lots of eval criteria use sub agent to then go and evaluate each criteria or each group of criteria. So that way you can be a little bit more confident that all of the criteria is being looked at. The third thing I'll say here is once the sub agent is complete, you can detail this in the skill, have the AI write a spreadsheet or as Haml will probably say pivot table of basically each of these criteria and the comps or like an indicator yes or no pass or fail. That way you could go look and then you yourself right know the hierarchy of what is important and for example this MECI criteria might not be as important in other things. So you can kind of really quickly make a judgment call yourself.

对。我经常给的一条建议是：干脆在 skill 文件里就把你的 eval 分成自上而下和自下而上两组。第二件事，如果你的评判标准很多，就用 sub agent 分头去评——每个 sub agent 负责一条标准，或者一组标准。这样你能更有把握：每一条标准都真的被看过。第三件事，等 sub agent 跑完之后（这一步你也可以写进 skill 里），让 AI 生成一张表格——用 Hamel 的话说大概会叫透视表——每一行是一条标准，后面跟着对应的结果，或者干脆就是一个是/否、通过/不通过的标记。这样你自己扫一眼，就能按你心里那套重要性排序去判断，比如 MECE 这条在别的场景下可能就没那么要紧。你自己很快就能拍板。

[9:16] Peter

Oh, I see. Okay. So maybe like some criteria is more important than other ones and like I can I can try to like add some weights or something.

哦我懂了。也就是说有些标准比另一些更重要，我可以试着给它们加个权重之类的。

[9:21] Shreya

Yeah, exactly. But you don't even need to do that. It's as simple as just having the AI as part of the skill generate a spreadsheet for you.

对，就是这个意思。不过其实你都不用做到加权那一步，简单到只要在 skill 里让 AI 帮你生成一张表格，就够了。

[9:28] Peter

Got it. And and and when you say bottom up, Eva, just so we're on the same page, you mean like stuff that I discover through the process, right? Is that what

明白。还有你说的自下而上的 eval——为了确认咱们说的是一回事——你指的是我在实际使用过程中发现的那些东西，对吧？是这个意——

[9:35] Shreya

bottom? Yeah. Bottom up is like data driven.

自下而上？对，自下而上就是数据驱动的。

[9:37] Shreya

Top down is it's not about the data. It's just like basics of a domain expert understanding the task. What would they look for?

自上而下则跟数据无关，就是一个懂行的领域专家理解了这个任务之后的基本判断：他会去看哪些点。

[9:45] Peter

And and I guess another thing that I haven't done is just like run this skill on let's say because I have all the raw transcripts and the final output for like the past 10 or 20 interviews that I did, right? So maybe I can just like run the scale on like 10 different raw transcripts and get its output and then maybe compare with my final, you know, human edited stuff and be like, hey, okay, so looking at all these 10 things at once, what kind of patterns can you find? Or that kind of stuff.

还有一件我一直没做的事：我手上其实存着过去十期、二十期访谈的原始转录和最终成稿。所以我完全可以拿十份不同的原始转录去跑这个 skill，拿到它的输出，再跟我人工编辑过的最终版本做对比，然后问它：把这十组一起看，你能找出什么规律？大概是这种玩法。

[10:08] Shreya

I love that. That's that's a great example of bottom up directly derived from the data. Um and because you have the actual labels the the actual takeaways, right? Then you can have AI agents reflect on the difference between the takeaways and then you know whatever would come out of the skill by Elyle.

我太喜欢这个了。这就是典型的、直接从数据里长出来的自下而上。而且因为你手上有真实的标注——也就是最终那版真正的 takeaway——你可以让 AI agent 去反思：你的成稿，和 skill 自动跑出来的那版之间，到底差在哪。

[10:24] Hamel

Awesome. Yeah. I I want to start with example because like I I feel like this kind of stuff anyone can build because people are building skills all over the place and anyone can build this eval thing for their skills. Um I think this is really interesting like you opened up a a very interesting rabbit hole which is writing and the thing that and I discuss all the time we joke that writing is the final boss for LLMs to like have it right like you in a way that you are satisfied with is extremely difficult. So I just want to yeah I do want to emphasize when you have this many criteria and you want to

太好了。对，我想先从这个例子讲起，因为我觉得这类事谁都能做——现在到处都有人在写 skill，谁都可以给自己的 skill 配一套 eval。我觉得这个话题特别有意思，你其实捅开了一个很有意思的兔子洞，就是「写作」。我跟 Shreya 老聊这事，我们开玩笑说写作是 LLM 的最终 boss——要让它写得像你、写到你满意，难度极高。所以我想强调一点：当你有这么多条评判标准、而且你想——

[10:55] Hamel

check this criteria post talk it is good to fan out and like give you know sub aents because if you give it the this entire list of criteria and there's so many bullet points here

——事后一条条去核这些标准的时候，最好是把任务扇出去（fan out），交给一堆 sub agent。因为如果你把这一整份标准清单一次性丢给它，这里的条目实在太多了——

[11:06] Hamel

it's kind of get washed out like a model will tend to ignore something and get lazy. we have a focus on one piece of criteria, it's going to focus on that piece of criteria.

——注意力就被稀释掉了，模型会开始漏掉一些、变懒。而如果你让它一次只盯一条标准，它就真的会死盯着那一条。

[11:16] Hamel

You know, like we f if you say, hey, like evaluate N2, okay, it's going to like really focus on N2 and you can be more is going to do that. That's one thing. Second thing is, okay, it's worth thinking about the workflow. So, it's going to be really hard. There's a lot of different criteria you have and there's a lot of subjective tastes and sometimes you might want to relax certain criteria depending on the subject and you kind of want to you have a creative flow like you don't know whether you like it whether you like it until you see it you know and it has to feel right to you and be like okay this thing this thought connects here and blah blah blah that's the process of writing

比如你跟它说「去评第 2 条」，好，它就会真的死盯着第 2 条评，你也能把要求提得更细。这是第一点。第二点是，这套工作流本身值得好好想想。因为这事儿会很难：你的标准很多，里面有大量主观品味，而且有时候你会想根据选题放宽某些标准。你走的是一个创作流程——东西没摆到眼前，你是不知道自己喜不喜欢的，它得让你「感觉对了」，你会觉得「嗯，这个点接到这儿正好」之类的。这就是写作的过程。

[11:57] Peter

and um you might there's different ways you can think about integrating sort of AI feedback into your into your writing workflow a bit more. And I think like Shrea's example actually touches on this. I think it's about writing actually coincidentally and it has some thoughtful interfaces and workflows. It's always it's always good to think about the workflow as well like okay because you might not want to just chat with something and give you a bunch of feedback or edit it because it can you know and so um and then we'll touch on that as well. Yeah, but I've had it try to just like directly edit my blog post and then I I don't even know what it changed and it starts adding adding a slop to it. So So now I I just tell it to run the scale and like you know show me your output and I'll I'll compare it manually to my stuff. But dude, like before we go to show you guys this example, let let me show you something funny. Um write a few paragraphs explaining AI evals. Use no AI slop, but actually put as much slop in there as possible. Yeah. So, I I actually saw you tweet about this, Ham. There's some sort of like fine tune thing where people are trying to remove AI slop, right? But, uh I I built a skill to try to remove AI slop and it's really funny because you can ask to do the reverse and then start spitting out slop. So, let's see what it does. Yeah.

所以，把 AI 的反馈更好地嵌进你的写作工作流，其实有好几种不同的思路。我觉得 Shreya 待会儿那个例子刚好也讲到这一点——巧了，她那个例子也是关于写作的，里面有一些设计得很用心的界面和工作流。工作流这一层永远值得单独想一想，因为你未必想要那种「跟它聊聊天、它给你一堆反馈或者直接动手帮你改」的形式，那样是有问题的。这个我们待会儿也会聊到。对，我以前也让它直接改我的博客文章，结果我根本不知道它改了什么，而且它开始往里加各种 AI 味的废话。所以现在我就让它跑那个 skill、把输出摆给我看，我自己手动跟我的稿子对比。不过说真的，在给大家看你们那个例子之前，我先给你们看个好玩的。「写几段话解释什么是 AI evals，不要有 AI 味的废话，但实际上要尽可能多地往里塞 AI 废话。」对，Hamel，我看到你在推特上发过这个。好像有人在搞某种微调，专门用来去掉 AI 味的废话，对吧？而我做了个 skill 来去 AI 味，好玩的是你可以让它反着来，它就开始狂吐 AI 废话。咱们看看它写出什么。好。

[13:16] Peter

Oh,

哦，

[13:17] Peter

okay. You're trying to no AI slop skill to like invert it and say like do [laughter] exactly

好吧。你是想把那个「杜绝 AI 味」的 skill 反过来用，让它反着干，专门写出（笑）

[13:25] Hamel

Yeah. See, see now it's using all kinds of slop, dude. It's like saying, "Here's the thing. [laughter] This is a paradigm shift. There's quite a few antagi slop skill."

对。你看，你看，它现在满篇都是 AI 味，兄弟。张口就是「事情是这样的。（笑）这是一次范式转移。」这类反 AI 味的 skill 其实挺多的。

[13:35] Peter

Oh, no. AI slop skill.

哦，「No AI Slop」这个 skill。

[13:36] Hamel

Yeah.

对。

[13:37] Peter

Um, show me

嗯，给我看看

[13:39] Hamel

it. It It works. Yeah.

它。它、它是真的管用。嗯。

[13:41] Peter

Wow.

哇。

[13:42] Hamel

Uh, well, sometimes it kind of go a little bit overboard and starts removing my personality from the writing, but uh, it's kind of like a thing that I built over time to just remove this because like dude, these models like Fable or whatever GBT 5.6 just keep producing slot, man. I got sick sick of it. I got sick of it. So So I just added a whole ton of stuff to cut

呃，有时候它会有点用力过猛，把我文字里的个人风格也一并抹掉了。不过这东西是我一点一点攒出来的，就为了把这股味儿去掉——因为兄弟，不管是 Fable 还是什么 GPT 5.6，这些模型就是不停地在产出 AI 味的东西，我真是受够了。受够了。所以我往里面加了一大堆规则去砍掉这些东西，

[14:02] Hamel

and I and I do this pass for everything that I write. Yeah, there's a lot of stuff.

而且我写的每一样东西都会过一遍这道工序。对，里面塞了很多条。

[14:06] Peter

Oh, that's so good. That's so good.

哦，太棒了。太棒了。

[14:08] Hamel

I I'll probably open source it at some point.

我大概过段时间会把它开源。

[14:10] Peter

That would be amazing,

那就太好了，

[14:12] Peter

by the way. And it's open. It's called plain writing skills or something like that.

顺便说一句，我那个是开源的，叫 plain writing skill 之类的名字。

[14:16] Hamel

Oh, yeah.

哦，对。

[14:17] Peter

Yeah, plain writing, I think, is the name of my skill. U but it's very similar to yours. I just I feel like a lot of people have such skills and I just want to take all the things that you put in your skill and put it in my skills.

对，plain writing，我记得我那个 skill 就叫这个。呃，不过跟你那个很像。我就是觉得，好多人手里都有这类 skill，我想把你 skill 里的那些东西全都搬到我的 skill 里来。

[14:29] Hamel

Yeah, we should merge the two skills.

对，我们应该把这两个 skill 合并了。

[14:31] Peter

Yeah, merge two skills.

对，合并成一个。

[14:32] Shreya

I know. All right. So, um what I'm doing is kind of giving you a tour of what it's like to use by scale for AI automated error analysis. You're going to see like an empty browser. That's fine. You're going to see my terminal. And I'll just show you what's in my folder pretty briefly. So here is basically I have um an AI assistant to help me with writing and I've asked the AI to generate if some AI to basically write some articles, I want to show you what it's like for me to externalize my taste on writing into LLM judges. Um and rather than look at all the data myself, I'm going to use Claude to help me look at the data. Okay, so I'm going to start out by just invoking a Claude session and I'm going to dangerously skip permissions. Um, and I'm not going to use Fable because I feel like that's a bit much. Um,

我懂。好，那个——我接下来要做的，算是带你们逛一圈，看看做 AI 自动化错误分析是什么体验。你们会看到一个空白的浏览器，没关系。你们还会看到我的终端。我先快速给你们看看我这个文件夹里有什么。基本上，我这里有一个 AI 助手帮我写作，我让这个 AI 生成……让它写了一些文章。我想给你们展示的是：我怎么把自己对写作的品味外化成 LLM judge。呃，与其我自己把所有数据都过一遍，我打算用 Claude 来帮我看数据。好，那我先起一个 Claude 会话，用 dangerously skip permissions。呃，我不打算用 Fable，因为我觉得那有点过了。呃——

[15:23]

is that Fable left?

你 Fable 还有余量吗？

[15:25] Shreya

I have a little bit of Fable left, but I've definitely I'm sorry. Don't want to use it on uh what? I don't want to use it on definitely looking at the data. It's going to just exhaust everything. All right. So, I've I've set the model to Opus 48. Um, and then what I am going to say is use my error discovery skill to build me an interface to um help me look at my AI outputs and give feedback. Okay. So, what it's going to do is going to go through my skill, which I'll show you in a little bit, and build this interface for me to review my data. this I'm going to stress that the interface that it comes up with is going to be so much better than me looking at my data in say Google spreadsheets or something it's just very very hard on the human eyes to read okay so this is running uh it's telling me what it's doing it's found the data let me un examine them to understand the structure of the data so these this data here is just AI generated writing samples it could also be endto-end traces like whatever you really want to do error analysis on is fine now let me show you the skill while this is running um error for discovery skill. Here we are. Um, wow, people are really using it. So, that's nice to see. So, what does the skill do? There's a lot going on. So, here are the five steps that I first want to mention. First, it's going to read the data set and figure out, you know, what is the type of data, like the semantic type really. So, not just like a string type or whatever. That's kind of silly, but is it an article? Is it code? Is it a traces? Then what it's going to do is figure out how to render this data so that it's very easy for humans to read. So that's what's called visual encoding. So based on what varies in the data such as you know if it's agent traces maybe it's like the number of turns in a trace. I don't know I'm making that up. The AI agent is going to reflect on that and when I say something

我 Fable 还剩一点，不过我肯定……不好意思。我不想把它花在，呃，什么来着？我不想把它花在看数据上，那样肯定会一下子烧光。好。所以我把模型设成了 Opus 4.8。呃，然后我要输入的指令是：用我的 error discovery skill 给我搭一个界面，帮我查看 AI 的输出并给出反馈。好。它接下来要做的，就是去读我这个 skill——待会儿我会给你们看——然后搭出这个界面来给我审阅数据。我要强调一点：它做出来的界面，会比我在 Google 表格之类的东西里看数据好用太多，那种真的非常费眼睛。好，它现在在跑了，呃，它一边跑一边告诉我它在干什么：它找到了数据，说「让我检查一下，理解数据的结构」。这里的数据就是一些 AI 生成的写作样本，也可以是端到端的 trace——你真正想做错误分析的任何东西都行。趁它还在跑，我先给你们看看这个 skill，呃，error discovery skill。就是它。呃，哇，还真有不少人在用，看到这个挺开心的。那这个 skill 到底做什么？里面东西挺多的。我先讲讲这五个步骤。第一步，它会读取数据集，搞清楚这是什么类型的数据——是语义层面的类型，不是那种「字符串类型」之类的，那太没意思了；而是：这是一篇文章？是代码？还是一堆 trace？第二步，它会想清楚怎么把这些数据渲染出来，让人读起来非常轻松，这就是所谓的 visual encoding（视觉编码）。它会根据数据里变化的维度来定，比如说如果是 agent trace，变化的可能就是一条 trace 里的轮次数——这个是我随口编的。AI agent 会去琢磨这件事。而当我说

[17:22] Shreya

use just stall principles here. All that means is carefully use things like color, spacing, opacity uh to show me the variance in the data. Um so again this is something that like kind of speaks to our visual cortex or visual processing skills as a human being. Third step is then to actually build an interface build it in HTML. I call it a review app here. Um it's a Python backend. It's a JavaScript or HTML with some JavaScript front end. Really doesn't matter. Then the fourth thing is it's going to figure out what samples that I should look at. You might have hundreds or even thousands of traces you want to do error analysis on. What sample do you look at? Um we leave it to an AI agent to help us with this. And then finally, last but not least, it runs an interactive loop. So that is cloud code or codeex whatever I'm going to use basically looks at the activi the interactions I make in the user interface compiles those interactions into open-ended feedback um and then reads that feedback in real time and proposes either new samples for me to label or some um criteria for my rubric you'll see what I'm doing as it's going

「这里用格式塔原理（Gestalt principles）」，意思无非就是：小心地运用颜色、间距、透明度这些手段，把数据里的差异呈现给我看。呃，说到底这是在调动我们身为人的视觉皮层、视觉处理能力。第三步，就是真正把界面搭出来，用 HTML 搭。我这里管它叫 review app（审阅应用）。呃，后端是 Python，前端是 HTML 加一点 JavaScript，其实用什么都无所谓。第四步，它会判断我该看哪些样本。你可能有几百条甚至几千条 trace 要做错误分析，那到底该看哪一条？呃，这件事我们交给 AI agent 来帮我们决定。最后一点同样重要：它会跑一个交互循环。也就是说，Claude Code 或者 Codex——随便我用哪个——它会盯着我在界面上的交互动作，把这些动作汇总成开放式的反馈，再实时读取这些反馈，然后要么给我推荐新的待标注样本，要么给我的 rubric 提出新的评判标准。等它跑起来，你们就看明白我在干什么了。

[18:37] Peter

and and the data is uh you said is cloud generated writing samples based on your stuff or

那这些数据，呃，你刚说是 Claude 生成的写作样本，是基于你自己的东西吗，还是——

[18:42] Shreya

yes well not bas bas on my stuff just I um based on our evals course bas I generated some synthetic data or synthetic inputs that people might presumably want AI to help them write an article with I'm just showing you this because I've a lot of people who build AI products within their org, right? Like have to do something more than just like their own personal workflow.

对，呃，其实不是基于我自己的东西。是这样，呃，为了我们那门 evals 课程，我生成了一些合成数据、合成的输入，模拟人们可能会想让 AI 帮忙写文章的那些场景。我之所以拿这个来演示，是因为我认识很多在公司内部做 AI 产品的人，对吧，他们要做的事情，不只是自己那点个人工作流那么简单。

[19:05] Peter

Yeah. Yeah.

嗯。嗯。

[19:06] Shreya

Yeah. Um so the unfortunate thing is it takes a little bit of time like takes like five minutes or so. I just wanted to show it to you from scratch so you know that I'm like I didn't you know make something up for you like this is real it's happening in real time. Um and it's following the steps exactly as that's in here right now. I think it's on step four of clustering the data and picking a diverse initial sample for us to label

嗯。呃，麻烦的地方是这要花点时间，差不多五分钟吧。我特意从零开始跑给你们看，好让你们知道我没有事先准备好来糊弄你们——这是真的，实时发生的。呃，而且它完全是照着这里写的步骤在走。我看它现在应该到第四步了：对数据做聚类，挑出一批多样性足够的初始样本给我们标注。

[19:29] Peter

and and the goal of the eval is to basically improve someone's writing

那这个 eval 的目标，基本上就是改进某个人的写作水平——

[19:34] Peter

or

还是说——

[19:34] Shreya

in this case what it's going to do is going to come up with a rubric of criteria that's important that defines like good and bad outputs good and bad writing in this case. So think of this like as exactly what you would want in your scale essenti like what are the rubric criteria.

在这个例子里，它要做的是给出一套 rubric、一套评判标准，用来定义什么是好的输出、什么是差的输出，在这里就是好文章和差文章。所以你可以把它理解成：这本质上就是你希望在自己那套评分量表里拿到的东西——也就是 rubric 的各条标准。

[19:50] Peter

Okay.

好。

[19:51]

It's going to help you discover more of the bottoms up stuff.

它能帮你挖出更多自下而上冒出来的东西。

[19:54]

Yes.

对。

[19:54]

That we were talking.

就是我们刚才说的那种。

[19:56] Peter

Are you guys still mostly using cloud or you switch over to GPT or both?

你们俩现在还是主要用 Claude，还是已经转去用 GPT 了，或者两个都用？

[20:00] Shreya

Um I can answer first. So I feel like Fable was very exciting for me because I feel like you can do like more meetings meaty software engineering tasks with it. Um, but then I've also been quite impressed by Codeex's um, I don't know if it's Codeex or GPT, but it's the new Soul. I feel like Soul is quite good at working on hard tasks and then having a very good like human readable or digestible output. Often when I work with cloud models these days, like what it's writing, like for example, let me quickly smoke test the server before it feel it just uses jargon that I would never use in my day-to-day life. And often I'm just so lost. So

呃，我可以先说。我觉得 Fable 让我挺兴奋的，因为感觉用它能干更有分量的软件工程活儿。不过 Codex 也挺让我意外的，我不太确定该算是 Codex 还是 GPT，反正就是新出的那个 Soul。我感觉 Soul 啃硬任务挺行的，而且输出特别好读、好消化。现在我用 Claude 系模型的时候，它写出来的东西——比如「让我先快速给服务器做个 smoke test」这种——净是些我日常根本不会用的黑话，我常常看得一脸懵。所以

[20:39] Peter

yeah, how about how about you Hava?

嗯，那你呢 Hamel？

[20:40] Hamel

Yeah, I've been using codeex a lot more and it's mainly because of the codeex app itself is so incredible in its functionality. So there's two things that are really important. One is mobile support is at a completely different level. You can see all the sessions across all your devices on your mobile phone.

对，我最近用 Codex 明显多了很多，主要是因为 Codex 这个 App 本身的功能太强了。有两点特别关键。第一，移动端的支持完全是另一个级别，在手机上就能看到你所有设备上的全部会话。

[21:00]

Yeah.

嗯。

[21:00] Hamel

You can sort of like have a command center that like shows you everything with no loss in functionality. like you you can access all the same things. You can cue messages, steer, switch models, you can see artifacts, so on so forth.

就相当于有了一个指挥中心，什么都能看到，而且功能一点不缩水，该有的都能用：可以排队发消息、可以中途纠偏、可以切模型、可以看生成的 artifacts，等等。

[21:14] Hamel

The second thing that's really cool, and not many people talk about this,

第二点特别酷，而且很少有人聊到，

[21:18] Hamel

is you can let codecs steer codecs. So you can have codecs create threads. You can have it you can have threads talk to each other. You can actually tag another thread that's running and say, "Hey, like help this thread out or front this thread. This thread has like 20 tasks. do some front run it, do some research. You can and you can do all kinds of you know interesting orchestration um you know without trying to go full code factory like you know sometimes this is like a really helpful

就是你可以让 Codex 去指挥 Codex。你可以让 Codex 自己开线程，让线程之间互相对话。你甚至可以 @ 另一个正在跑的线程，跟它说「帮这个线程一把」或者「给这个线程打前站」。这个线程排了 20 个任务，那就让它先抢跑一部分、先做点调研。你能搞出各种有意思的编排，又不用真把自己整成一个「代码工厂」，有时候这真的特别顺手。

[21:50] Hamel

um and you can have like codecs like operate on codecs like hey rename all these threads

而且你还能让 Codex 直接操作 Codex，比如「把这些线程全部重命名」，

[21:54] Hamel

do something and it just does that so it's like really useful um and that makes me gravitate towards that and then like at least like the last part is the economics they have a lot less limits in codecs when it like the the Frontier models is like fully within the subscription. It's on like this 50% thing. Then they keep resetting it constantly.

让它干点什么，它就真给你干了，所以特别好用。这也是让我越来越倒向它的原因。最后一点，至少还有经济账：Codex 的额度限制少得多，前沿模型完全包在订阅里。现在还搞着那种打五折的活动，然后额度还老是给你重置。

[22:16]

Yeah. It's almost like a joke. They keep resetting. [laughter]

对，都快成段子了，一直重置。（笑）

[22:19]

It's so nice. I mean, I'm not going to complain. I saw this tweet recently that made me laugh. Like we're in the Uber Lift era of like AI models. Like remember 10 years ago where Uber was like $2?

这也太爽了。我是不会抱怨的。我最近看到一条推把我笑到了：我们现在正处在 AI 模型的「Uber、Lyft 时代」。还记得十年前 Uber 只要两美元的时候吗？

[22:32]

We're in that right now.

我们现在就在那个阶段。

[22:34]

Yeah. The subsidization is going to stop at some point.

对，补贴总有一天会停的。

[22:37]

I know. Yeah.

就是说啊。

[22:38]

Yeah. I don't know what's going on with Anthropic. I I feel like Dario hates Twitter or something. So like this is completely botched all their communication strate and and everything. I feel like the people who actually want to communicate like probably cannot because of some of internal stuff. So [laughter] I don't know what's going on.

嗯。我也搞不懂 Anthropic 到底在干嘛。我感觉 Dario 是不是讨厌 Twitter 之类的，结果把他们整个对外沟通的策略全搞砸了。我觉得真正想出来说点什么的人，可能因为某些内部原因反而说不了。（笑）不知道怎么回事。

[22:53]

What about you, Peter? What do you

那你呢 Peter？你都

[22:55] Peter

uh Yeah, I I pretty much use Codex for everything except for Fable that I used to do like planning and try to find bugs and stuff, but pretty much I use Codex and GBT for everything.

呃，是这样，我基本上什么都用 Codex，只有 Fable 我以前会拿来做规划、找 bug 之类的，其他基本上就是 Codex 和 GPT 全包了。

[23:04] Peter

Did you always use Open AI? No, I I I I was like total cloud fanboy like even like a few few months ago.

你一直都用 OpenAI 吗？没有，我以前是彻头彻尾的 Claude 粉，几个月前都还是。

[23:11]

Wow.

哇。

[23:11] Peter

But I but I discovered Codex. I I think I think the stuff Haml mentioned, but also like the browser use is just incredible on CODEX. Like half the stuff I use don't even have any APIs and I just get to click around and [laughter] figure things out.

但后来我发现了 Codex。Hamel 说的那些点我都同意，另外 Codex 的浏览器操作能力简直不可思议。我用的东西里有一半压根就没有 API，它就直接替我点来点去、自己摸索出来。（笑）

[23:25]

Wow.

哇。

[23:25] Peter

It's pretty incredible.

真的挺离谱的。

[23:26]

One of my most used skills is this reverse engineering skill. There's just like any website without a API. It just it instructs the agent to like listen to all the network traffic and document the API and like like memorialize that

我最常用的 skill 之一是一个逆向工程 skill。随便一个没有 API 的网站，它会让 agent 去监听所有网络请求，把 API 摸清楚、记录下来，

[23:42]

like scripts and whatnot so that I can Yeah, I just make any website API now. [laughter]

写成脚本之类的存起来，这样我就……对，我现在能把任何网站都变成 API。（笑）

[23:49]

That's smart. Yeah, that's smart. Just check out this thing called printing press uh CLI or something. It does something pretty similar.

这招聪明。对，真聪明。你可以去看看一个叫 printing press CLI 之类的东西，做的事情挺像的。

[23:55]

Oh, cool. Okay,

哦，酷。好，

[23:57] Shreya

so finally after Wow, it's didn't it cooked for 15 minutes. Okay, so finally after 15 minutes a built an interface. Um, so let me show show you a little bit what's going on in the interface. So there's three different tabs. There's a article by article pane. So this would either be article by article or trace by trace or whatever granularity you want to look at. So this is looking at one thing at a time. Um, there's a map view which kind of shows you the clustering that the agent did into different semantically meaningful categories. Um and then there's the progress view which is kind of empty here but this view tells us um based on our reviewing and our annotations on the data what are the common failure modes that are happening. Okay. And so this is automatically populated by an agent. So I want to talk a little bit about the design philosophy here which is have the human have yourself or myself read this and give open-ended feedback and then the job of the agent is to really it's not to invent new open-ended feedback but it's to kind of group that distill that into actionable rubric criteria so the human doesn't have to do a lot of that work.

所以最后……哇，它跑了足足 15 分钟。好，总之 15 分钟之后，它做出了一个界面。我来给你们看看这个界面里都有什么。这里有三个标签页。一个是「逐篇文章」的面板——你想一篇篇看文章、还是一条条看 trace，或者用任何你想要的颗粒度都行，总之这里是一次只看一个样本。还有一个 map 视图，展示 agent 把数据聚成了哪些语义上有意义的类别。然后是 progress 视图，现在还是空的，但这个视图会根据我们的审阅和标注，告诉我们有哪些反复出现的失败模式。好，这一块是 agent 自动填进去的。我想稍微讲讲这里的设计哲学：让人——让你自己、或者我自己——去读这些内容，给出开放式的反馈；而 agent 的职责不是去发明新的开放式反馈，而是把这些反馈归类、提炼成可执行的 rubric 标准，这样人就不用干那么多苦活了。

[25:07] Shreya

So let's go through this. Um so here I'm looking at this AI generated article about sleep hygiene. Um, and I'm just going to go through a little bit about um the how I kind of like feel. So, I'm going to just assume that the top down criteria Claude can take care of. Um, and then I'll give some feedback on this. Um, I don't really like AI phrasing around, you know, it's not X, it's Y, like the negative contrast. So, I'm going to say something like I don't like negative contrast in my writing. And then notice that I just gave that feedback what I'll call in sichu. Okay. So I just I this interface makes it really easy to just externalize my thoughts as I have them. And as I gave that feedback, my claude agent is invoking the monitor tool to basically look at whatever I give and react to it. So here you can see that it said my first note from you on the sleep article. I don't like negative contrast in my writing. And then it's going to do something. Um let it do this in the background, right? I just want to be in flow of reading and giving feedback um on the samples that it picked for me. So remember it also picked a diversity of samples and it will keep generating more samples for me if it wants but that's not important. Um anyways so I'm going to keep reading uh and then

我们走一遍。这里我在看一篇 AI 生成的、讲睡眠卫生的文章。我大概说说我读的时候是什么感觉。我先假设那些自上而下定好的标准 Claude 能搞定，然后我针对这篇再给点反馈。我不太喜欢 AI 那种句式，就是「这不是 X，而是 Y」那种负向对比。所以我会写一句：我不喜欢我的文章里出现负向对比。注意，我刚才这条反馈是「就地（in situ）」给出的。这个界面让我可以随时把脑子里冒出来的想法直接外化出来。而我一给出反馈，我的 Claude agent 就会调用 monitor 工具，去看我写了什么、并对它做出反应。这里你能看到它说：这是你在睡眠那篇文章上给我的第一条笔记——我不喜欢我的文章里出现负向对比。然后它就会去做点什么。让它在后台跑着就行，对吧？我只想保持在读稿子、给反馈的心流里，专注在它帮我挑出来的样本上。别忘了它挑样本时也照顾了多样性，如果它觉得有必要还会继续给我生成更多样本，不过那不重要。总之我接着往下读，然后……

[26:32] Shreya

like that kind of stuff like this one humble me is like also really annoying.

就这类东西，比如这句「humble me」，我也觉得特别烦人。

[26:37]

Oh let's do that.

哦，那就把这条写上。

[26:38] Shreya

Yeah. H. Uh, this feels annoying. Not exactly sure why. Can't give you a pippy label yet, but reflect on more feedback I gave you and reason why.

对。呃，这个读着就让人烦，但我也说不太清为什么。我现在还没法给你一个精准的标签，不过你可以结合我给过的其他反馈想一想，推理一下原因。

[27:01] Shreya

So, let's do that. Um

那我们就这么办。嗯……

[27:07] Hamel

so just so you know like the two things that are really interesting going on behind the scenes here is one you recall the clustering that Treya showed the diverse examples are being drawn from the clusters so that diversity and then the monitor tool the cloud monitor tool that's built in the quad is being used behind the scenes to to look at all the stuff that he's

顺便说一句，这背后有两件特别有意思的事。第一，还记得 Shreya 刚才演示的聚类吗？这些多样化的样本就是从那些聚类里抽出来的，多样性是这么来的。第二，Claude 里内置的那个 monitor 工具，正在后台看着她给出的所有这些反馈……

[27:30] Shreya

just I'm like giving a lot of feedback back. Um, and it's totally fine that like you know if you you might most people like are going to have to go through many samples to be able to give feedback. Um,

……我这会儿是在很密集地给反馈。而且完全没关系——你可能，其实大多数人都得先看过很多样本，才给得出反馈。嗯……

[27:42]

yeah.

嗯。

[27:43] Shreya

I've just I've done writing before so I'm able to give it now but um that's why it's very important to review lots of samples especially in the beginning.

我以前写过东西，所以现在能直接给得出来。但也正因为这样，多看样本非常重要，尤其是一开始的时候。

[27:51] Peter

So just you're still doing the manual review. You're not getting the AI to review everything. Yes.

所以你还是在做人工审阅，没有让 AI 把所有东西都审一遍。是的。

[27:56] Shreya

Well, here's the thing that's gonna happen is I'm giving manual review and in the background Claude is reflecting on it. Um, so it's about voice rather than facts, whatever. So once I give around 10 reviews, then it's going to start like thinking about a taxonomy and then it's going to apply my feedback. So for example, this was another negative contrast thing. Um, after a few more reviews, it would take all it would try to look for all instances of negative contrast. should automatically label it for me. So I don't again

其实接下来会发生的是这样：我这边在做人工审阅，Claude 在后台反思我给的反馈。比如它会判断这条是关于文风的还是关于事实的，诸如此类。等我给到大概 10 条审阅之后，它就会开始琢磨一套分类体系，然后把我的反馈应用出去。举个例子，这里又是一处负向对比。再多给几条反馈之后，它就会去把所有出现负向对比的地方都找出来，自动帮我打好标签。这样我就不用再……

[28:27] Shreya

keep keep going down. I'll let it see that. Yeah.

……接着往下翻。我让它看看这条。对。

[28:30] Shreya

Back here. Okay. Um All right. So that's fine. I've given four feedbacks. Uh and then I'm just going to also look at the right. See, so if you look at the progress like it doesn't like negative contra, it knows that negative contrast is a thing. And then this uh triadic parallel enumeration thing. I don't know about you, but I really don't like this list of threes. I feel like AI does it all the time. So,

回到这边。好，嗯，行，这样就可以。我已经给了四条反馈了。然后我也看一下右边——你看 progress 那里，它已经知道「不喜欢负向对比」是一类问题了。还有这个「三段式排比枚举」。不知道你怎么想，但我真的特别烦这种「三点式清单」，感觉 AI 老这么写。所以……

[28:55]

yeah.

对。

[28:56] Shreya

All right. So, probably running something. I don't know exactly, but let's just keep going.

好。它应该在跑什么东西吧，我也说不准，反正我们接着往下走。

[29:14]

It's like a treasure hunt for a sloth.

简直像树懒的寻宝游戏。

[29:16]

I know. Everywhere.

我知道，到处都是。

[29:18] Shreya

For some reason, this also I feel like I see AI do this a lot. Like a number, it makes like two or three three more things to know or like keep in mind.

不知道为什么，这个我也觉得 AI 特别爱干——就是给你一个数字，动不动来「另外两三件需要知道的事」，或者「有几点要记住」。

[29:29]

Yeah.

对。

[29:33]

Paragraphs start topic sentence with small things. And and uh this whole annotation thing is uh part of the skill that you built.

段落要用主题句开头，别一上来就讲细枝末节。……还有，这一整套标注的东西，是你做的那个 skill 的一部分吧？

[29:53] Shreya

Yeah.

对。

[29:53]

Or

还是说……

[29:54] Shreya

everything I'm showing you is like the skill and it's this the skill was invoked to do this.

我给你们看的这些全都是那个 skill，就是调用这个 skill 做出来的。

[30:01] Shreya

Um here's I want complete sentences. I'm just giving giving feedback so that way it can start doing the automatic pass

嗯，这里我想要完整的句子。我一直这么给反馈，就是为了让它能开始跑那一轮自动标注。

[30:14] Shreya

because this suggestions number is right now zero but this will become more um as it's doing its automatic pass.

因为现在 suggestions 这个数字还是 0，等它跑完自动标注，这个数就会涨上去。

[30:21] Peter

Okay. I I was going to ask you what's the advantage of having it reflect live versus like you know you give all the feedback and it reflects at the end.

好。我本来想问你：让它实时反思，跟你把反馈全给完、最后它一次性反思，这两种做法各有什么好处？

[30:28] Shreya

Oh, it's a good question. Um I think that you can have it reflect at the end. That works. Um, but I I also feel like trying to interle like human think time with AI think time is great. Like otherwise it's going to take a lot of time to reflect on the feedback at the end and you're waiting on it. Like I like the idea of like it reflecting in the background and then being able to suggest things for me once I hit to a certain number. Um, and and in practice I don't like always look at the AI side by side like this. I'm just showing it to you for the demo so that you can like see that the AI is doing something

哦，好问题。我觉得最后再反思也行，那样也能work。但我也觉得，把人思考的时间和 AI 思考的时间交错起来是件很棒的事。不然的话，最后让它把你那一堆反馈反思一遍会花很久，你只能干等着。我更喜欢它在后台一边反思，等我攒够一定数量之后，它就能给我提建议。而且实际用的时候，我并不会像现在这样一直盯着 AI 那一侧看。我现在摆出来只是为了演示，好让你们看见 AI 确实在干活。

[31:04] Shreya

and giving these doing these annotations is usually really it iterative meaning you refine your taste over time after you see more examples you get new ideas on what you want and a lot of times it could be helpful to see what the AI thinks as well and just to help ground you so it's like happening more iteratively even if

而且做这些标注通常是很迭代的——意思是随着你看的例子变多，你的品味会被慢慢打磨，你会对自己到底想要什么冒出新的想法。很多时候，看看 AI 是怎么想的也挺有帮助，能帮你找到一个参照。所以这个过程更像是不断迭代地在发生，哪怕……

[31:24] Peter

that makes sense that makes sense. It's like having a thinking partner.

有道理，有道理。就像有个思考搭子。

[31:28]

Yeah. Okay. Well, I maybe I need to go three more before it starts suggesting cuz I can't remember the exact number. [snorts] Start suggesting things already. I'm just going to tell it. You might be thinking, hey, like is there a SAS that does this? But look at like you don't need that. And look at the degree of flexibility Shrey is able to have by just integrating this process directly with claw. She can do whatever she wants.

是啊。好。嗯，我可能还得再标三条它才会开始给建议，我记不清具体是多少条了。（笑）赶紧开始给建议吧。我直接告诉它算了。你可能会想，欸，有没有哪个 SaaS 就是干这个的？但你看，你根本不需要那玩意儿。而且你看，Shreya 把这套流程直接接进 Claude 之后，灵活度有多高。她想怎么弄就怎么弄。

[32:01] Peter

Yeah, that that's why I this is a tangent, but that's why I feel like SAS is is actually in trouble, dude. Like [laughter] if I can just v code this kind of stuff, why would I pay for SAS? Like just build my own thing.

对，这有点跑题，但这就是为什么我觉得 SaaS 真的有麻烦了，兄弟。（笑）这种东西我自己 vibe code 一下就能做出来，那我干嘛还要花钱买 SaaS？直接自己搭一个不就完了。

[32:12] Hamel

Now the other side of the coin is you have to have a really good idea of the workflow. Like a lot of people that may not be watching this video would not know where to begin, right? Really good idea of the process, then you can probably And the other thing I that hopefully comes to mind is like it's like really tedious to give feedback. Like

不过硬币的另一面是，你得对这个工作流有非常清楚的认识。很多没看过这期视频的人，根本不知道该从哪儿下手，对吧？对流程足够清楚，你才有可能——还有一点，希望大家也能想到，就是给反馈这件事真的特别磨人。就像

[32:42]

yeah,

对，

[32:43] Shreya

I've not reviewed that many. Um I've only given eight notes. So just like yeah, know that it's it's really hard. Um, and also AI just can't read your mind for you. I've never like all of these notes that I'm having like some some of them maybe AI can come up with, but I don't know like not all of them.

我其实没看多少条。嗯，我才写了八条批注。所以就是，你得明白这活儿真的挺难的。嗯，而且 AI 也没法替你读心。我写的这些批注，有一部分 AI 也许自己能想出来，但我觉得不可能全部都想到。

[33:07] Hamel

Uh, yeah, you can probably just speed up with like voice dictation or something,

呃，对，你大概可以用语音输入之类的办法提提速，

[33:10] Shreya

but but yeah, you just have to read through everything.

但说到底，你还是得把所有东西都通读一遍。

[33:13]

Yeah.

对。

[33:13]

Yeah.

嗯。

[33:14] Shreya

Or [snorts] read through some sample at least. Um, okay. So, I'll start pushing suggestions to your queue right now instead of waiting. Yeah, probably I had to get to 10 notes, but

或者至少读一部分样本。嗯，好。那我现在就开始往你的队列里推建议，不等了。对，本来可能得凑够十条批注才行，不过

[33:24]

okay.

好。

[33:25] Shreya

I like having the 10 notes because it's like a carrot. It like makes you read some things like

我挺喜欢「十条批注」这个设定的，它像根吊在前面的胡萝卜，能逼着你真的去读点东西，就像

[33:31] Hamel

makes you sweat just a little bit which is good for you.

让你稍微出点汗，这对你有好处。

[33:35] Shreya

There we go.

来了来了。

[33:35] Shreya

Thinking about stuff. Oh, nice. Okay.

它开始想了。哦，不错。好。

[33:38]

Yes.

对。

[33:41] Peter

Okay. So, it's going to uh annotate annotate every single doc.

好。所以它接下来会给每一篇文档都做一遍标注。

[33:46]

Mhm.

嗯哼。

[33:47] Shreya

Yep. With the criteria I've already found. Okay.

对。用我前面已经找出来的那些标准。好。

[33:52] Hamel

Thing that's really great about this is like okay this shre is showing it's it's fit perfectly to this domain of writing. Like if you notice the writing is being shown like in situ as she said in the way that you would read it as a user. You're not looking at like a log file or a trace viewer at what it actually looks like

这里特别棒的一点是，Shreya 展示的这套东西，跟「写作」这个领域贴合得非常好。你注意看，文稿是像她说的那样「原位」呈现的，就是用户平时读到它的样子。你看的不是日志文件，也不是 trace 查看器里那种形态，

[34:12] Hamel

and you're just annotating it in place.

而是就地直接标注。

[34:16] Shreya

Right. So it's like it's um generated 361 suggestions. Yeah. So it said the less than four words staccato rule produced 249. Um

对。所以你看，它这会儿生成了 361 条建议。嗯。它说「少于四个词的短促句」这一条就产出了 249 条。嗯

[34:29] Peter

um but but what happens so what happens next after you accept all this? Like does it actually suggest how how do you fix this stuff or

嗯，那接下来呢？你把这些全都接受之后会发生什么？它会不会真的告诉你这些问题该怎么改？还是说

[34:37] Hamel

so the idea here as I mentioned before is to like then you can turn this failure modes all the things that you found into a rubric like this becomes your evals. I see.

所以这里的思路，我前面也提到过，就是你可以把这些失败模式、把你找出来的所有问题变成一份 rubric，这就成了你的 evals。原来如此。

[34:47] Hamel

Um, and then so you can apply that as a skill as you do. Um, you can have an LLM judge for each. You can turn that into a dashboard. You can have that monitor agent traces or outputs in real time. Like the whole the the hardest part of eval is error analysis which is coming up with this rubric.

嗯，然后你就可以像你现在这样，把它做成一个 skill 来用。你也可以给每一条配一个 LLM-as-Judge。你可以把它做成一块 dashboard。你还可以让它实时监控 agent 的 trace 或者输出。做 eval 最难的部分其实就是 error analysis，也就是把这份 rubric 弄出来。

[35:05] Hamel

Um, so this process just makes it really easy

嗯，所以这个流程把这件事变得特别轻松，

[35:07] Hamel

because of this annotation. You also have a data set. So

而且因为做过这一轮标注，你顺带还有了一份数据集。所以

[35:11]

yeah,

对，

[35:12] Shreya

you can after your evals get created now you can check those evals against the data set

等你的 evals 建好之后，你就可以拿这份数据集去检验这些 evals

[35:18]

that I mean what I'm saying is part of eval but

那个——我的意思是，这本来也属于 eval 的一部分，不过

[35:21] Hamel

eval is not just a rubric you want to be able to measure the rubric and say like is it good and so because of these been annotated you can you can you have this thing to benchmark it against

eval 不只是一份 rubric——你还得能去衡量这份 rubric，判断它到底好不好。正因为这些数据都被标注过了，你才有东西可以拿来做基准比对。

[35:33] Peter

right

对。

[35:35] Shreya

okay so it found okay so the most frequent example is um the staccato fragments Right.

好，它跑出来了。出现频率最高的一类是 staccato fragments，就是那种一顿一顿的碎句。对吧。

[35:42] Shreya

And there was also colons and stuff,

另外还有冒号那一类。

[35:44] Hamel

but it makes sense, right? It's like, so that's another thing like as somebody I didn't know that that's the most frequent example, right? Like

但这挺合理的，对吧？这就是另一个点——作为一个……我自己都不知道原来出现最多的居然是这一类。

[35:52] Hamel

I would have thought it's the not X but Y, but I I guess I guess it is the stacato fragments. I believe it. So I so again goes back to the philosophy of like human is the one driving the taste and the AI is the one like scaling up or giving superpowers to the human to apply that at scale.

我本来以为会是那种「不是 X，而是 Y」的句式，但看来还真是 staccato 碎句。我信了。所以这又回到那个理念：品味由人来把握，AI 负责放大、给人超能力，让这份品味能规模化地用出去。

[36:14] Peter

Okay. Got it. Wait. So so so just to understand like can you go back to the progress tab?

明白了。等一下，我想确认一下——能切回 progress 那个标签页吗？

[36:19] Peter

So now we have these these themes and these rubrics, right? So so what what do I do with this now? Let's I say I'm building a writing skill. Do I just like uh point AI to this thing and be like hey now create a scale for me? I mean, I would be right here. Write a skill that evaluates AI assisted writing based on the failure modes found. I like it would just ask for a skill and there's the skill. Um, and I want to emphasize that like you should spend a couple more hours doing the error discovery. Like don't don't just look at five things. like actually go through, you know, review all 18 like feel confident that you've like kind of saturated. Um, but yeah, you can just ask it to write a skill and it will write a rubric.

现在我们有了这些主题、这些 rubric，那接下来我拿它们干什么？比如说我要做一个写作 skill，我是不是直接把 AI 指到这个东西上，跟它说「来，帮我生成一个 skill」就行？——我就会直接在这儿写：根据发现的失败模式，写一个评估「AI 辅助写作」的 skill。就这么让它写一个 skill，skill 就出来了。另外我想强调一下，错误发现这一步你应该多花几个小时，别只看五条就完事，真的把 18 条全过一遍，感觉自己确实挖饱和了再说。但没错，你直接让它写 skill，它就会给你写出一份 rubric。

[37:06] Peter

I I see. Okay. Then then basically every time you have a writing sample, you just run the scale, right?

懂了。那基本上以后每拿到一份写作样本，直接跑一遍这个 skill 就行，对吧？

[37:12] Shreya

Yeah. And the rubric will output like uh this writing sample has a ton of staccato fragments or something.

对。rubric 会输出类似「这份样本里有一大堆 staccato 碎句」这样的结果。

[37:18] Hamel

Um the rubric will have staccato fragments as one of the criteria. Like if you look at if you think about your podcast evaluation skill, right? That's a rubric of many criteria. So stat will be one of the like five criteria that it comes up with or that this came up with.

rubric 里会把 staccato fragments 作为其中一条评判标准。你想想你那个播客评估 skill，那本身就是一份包含很多条标准的 rubric。staccato 就是它总结出来的五条标准之一——或者说，是这次跑出来的这几条里的一条。

[37:33] Hamel

So one thing that's really interesting is if you notice the way there is a okay the way you eval something is often should inform the way you might want to design the interface for the user to begin with. So, um, if you notice like in the annotation workflow, she had the AI, you know, suggest different annotations. You can imagine like if you were writing for real, you would want an IDE that would make the same suggestions to you and show you why. Like, hey, like here's a staccato thing, here's a negative contrast, here's this, here's that.

有个特别有意思的点：你怎么 eval 一个东西，往往应该反过来决定你一开始该怎么给用户设计界面。比如你注意到刚才那个标注流程里，她让 AI 主动给出各种标注建议。你可以想象一下，如果你是真的在写东西，你会希望有个 IDE 也这样给你提示，并且告诉你为什么——比如「这里是 staccato 碎句」「这里是负面对比」「这里是这个」「这里是那个」。

[38:07] Hamel

Um, so that you you don't have to just have AI just overwrite things. You have to keep rereading things. You want to accept and reject things. And so yeah, it's actually forces some amount of good design thinking to go through this exercise because each interface is going to be different like Sha showed you one for writing. Okay, a customer service support bot is going to look totally different in the interface or you know another

这样你就不用让 AI 直接把内容整段覆盖掉，也不用一遍遍回头重读。你想要的是能一条条接受或拒绝。所以走一遍这个流程，其实会逼着你做一些像样的设计思考，因为每种场景的界面都不一样——Shreya 给你看的是写作场景的；换成客服机器人，界面会完全不同，其他场景也是。

[38:32] Shreya

even in the error discovery interface, right? My skill asks the agent to reflect on the nature of the data and design a visual encoding for it. It's not going to look exactly like this.

就连错误发现的那个界面本身也是。我那个 skill 会让 agent 先去琢磨数据本身的性质，再为它设计一套视觉呈现方式。所以做出来不会跟这个长得一模一样。

[38:43] Peter

Okay. And I guess the the main the main benefit so uh in the previous process I still have to review traces and come over the errors but now the AI is actively helping me find the themes right so it kind of skips that

明白。我感觉主要的好处是——之前那套流程里，我还是得自己去看 trace、把错误一条条梳出来；现在 AI 会主动帮我找主题，等于把那

[38:59] Peter

host step.

一整步给省了。

[39:01]

Yep.

嗯。

[39:01] Peter

Yeah. Got it. [snorts]

对，懂了。

[39:03] Hamel

Well doesn't skip the step I would say. You still have to look at the data. There's there is no world in the future. Even if you have AGI, if you're building a product, you have to look at your data. Otherwise, you're there's no way, right? Like you cannot you have to be able to inject your taste somehow into the development of your product.

我倒不会说是「省掉」这一步。你还是得看数据。未来不存在那样的世界——就算有了 AGI，只要你在做产品，你就必须看你自己的数据。不然根本行不通，对吧？你总得有办法把自己的品味注入到产品的开发里。

[39:20] Hamel

This is we would think is our is our attempt to show you like how here's a way you can do it effectively with with all the benefit and might of AI to help you accelerate.

我们觉得这套东西是在向你展示：有这么一条路子可以把这件事做得很高效，同时把 AI 的能力和威力全用上，帮你提速。

[39:30] Peter

Got it. What what what is the point of the suggestions to re review? it just gives more like gives more human input to some of the stuff.

明白。那「待复核建议」这块的意义是什么？只是让人在里面再多给一点人工输入吗？

[39:37] Shreya

Um, so this is the agent scanning all records for the known failure modes.

这是 agent 拿已知的那些失败模式，去把全部记录扫一遍。

[39:42] Shreya

Um, and these are records that were not labeled by me with the human. So, okay, just to like I can accept all in practice I always accept all of them after like scanning briefly just to make sure like I use this to make sure that the AI understands what I meant by for example negative contrast.

而且这些记录都是我本人没标注过的。所以——我可以一键全部接受。实际操作中我一般都会先快速扫一眼再全部接受，我拿它来确认 AI 是不是真的理解了我说的比如「负面对比」是什么意思。

[40:02] Shreya

Okay. So, it's like, yeah, totally. It understands what I meant by negative contrast. Good job. Accept.

嗯，完全没错，它确实理解了我说的负面对比。干得好，接受。

[40:09] Hamel

The thing I really love about this is if there's something wrong with this interface or she can't give the feedback she wants. So, she's like, let's say she has a theme that the AI is missing out on, she can just go directly to Claude on the left hand side and fix it. You don't have you're not constrained to this UI.

我特别喜欢这一点：万一这个界面哪里不对劲，或者她想给的反馈这里给不了——比如她发现有个主题 AI 漏掉了，她可以直接在左边跟 Claude 说一声就改掉。你不会被这个 UI 框死。

[40:26]

Yeah. Yeah.

对，对。

[40:26]

Exactly.

没错。

[40:27] Peter

Yeah. This is why like uh your own product is like way more flexible than some sort of SAS SAS product. It just you have all these use cases that no common SAS will think think about. Yeah. Um so if I wanted to use you okay so now this is getting me pretty excited. So let's say I want to use my use your error analysis your error discovery skill for my previous example right for the podcast stuff. So basically I would uh give it so you got AI to generate a bunch of writing samples. So, instead of that, maybe I'll give it um like maybe before and after examples or like maybe I'll give it examples of like takeaways that it made for previous episodes.

对。所以说，自己做的产品比那种 SaaS 产品灵活太多了。你会有一堆用例，是通用 SaaS 根本想不到的。嗯……那如果我想用你的……好，现在我有点兴奋了。假设我想把你这套错误分析、错误发现的 skill 用到我前面讲的那个例子上，就是播客那件事。那基本上我得给它……你刚才是让 AI 生成了一堆写作样本，那我可能换成给它一些改写前后的对比例子，或者给它一些我之前几期做的 takeaway。

[41:04] Shreya

Yeah. What I would do is just create a folder with your current skill for generating takeaways as is and then examples of podcast transcripts and the takeaways that you said were good. And then I would just invoke the skill and say um help me do error analysis on my podcast takeaway skill. So apply the podcast takeaway skill to generate a bunch of takeaways and then build me an interface to review those takeaways especially compared against the outputs

对。我会这么做：建一个文件夹，把你现在生成 takeaway 用的那个 skill 原样放进去，再放上播客转写稿的例子，以及你认为好的那些 takeaway。然后直接调用这个 skill，跟它说：帮我对我的播客 takeaway skill 做一次错误分析。也就是先用这个播客 takeaway skill 生成一批 takeaway，再给我搭一个界面来审阅这些 takeaway，尤其是拿它们跟那些输出对照着看。

[41:36] Peter

the the that's perfect. Yeah, that's that's perfect. So yeah.

这太完美了。对，这就对了。就这样。

[41:41] Shreya

Yeah. and then just go through I mean it'll take like 15 minutes for it to give you the interface but you'll be able to go through it fast and like the whole point is confidence right like at the end of the day I want confidence that that my skill is do like I know what behavior the LLM is exhibiting so that's the whole point of error analysis

对，然后你就一条条过就行了。生成这个界面大概要花 15 分钟，但你过起来会很快。说到底，这件事的意义就是「有底气」——我最后要的是对自己这个 skill 有信心，是我清楚地知道 LLM 到底表现出了哪些行为。这就是 error analysis 的全部意义。

[41:58] Peter

so so I should have like maybe I should modify your interface to have my the AI's output on one side and the ground truth on the other so I can look

所以我是不是应该……也许我该改一下你这个界面，让 AI 的输出在一边、ground truth 在另一边，这样我好对着看。

[42:06] Shreya

so um the interface is going to be different when you invoke the skill on your data. It's not going to come up with this interface. The AI agent will read your data. It will read those takeaways and come up with a different interface to show you and render that to you as in step two here, designing the visual encoding and building the HTML.

其实你把这个 skill 用在自己的数据上时，界面是不一样的，不会长成现在这样。AI agent 会去读你的数据、读那些 takeaway，然后设计出一个不同的界面渲染给你看——也就是这里第二步做的事：设计视觉编码、把 HTML 写出来。

[42:26] Peter

Okay, that that's amazing. Thanks so much. Thanks so much for showing us. Okay, and I just I just want to point out to people who watching this that this is free as far as I know. And uh

好，这太棒了。非常感谢，谢谢你给我们演示。对了，我想跟看这期的朋友说一句：据我所知这个东西是免费的。另外——

[42:35] Shreya

yep. Yep.

对，没错。

[42:35] Peter

Just go to error discovery skill on GitHub and and you'll find or you know what I'll link in the description

去 GitHub 上搜 error discovery skill 就能找到——算了，我干脆放个链接在这期的

[42:41] Peter

of this episode. Cool. And I love how you're honest about plot helping you come up with this on the right.

描述里。挺好。还有我很喜欢你很坦诚地说，右边这部分是 Claude 帮你想出来的。

[42:47] Shreya

Oh yeah, of course.

哦，那当然。

[42:49] Peter

Yeah. I usually try to remove that to pretend I built the whole myself.

对。我一般都会把那段删掉，好假装整个东西都是我自己做的。

[42:54]

No, no, no.

不不不。

[42:55]

Yeah. Yeah.

对，对。

[42:56] Shreya

Claude is very good at like writing skills for Claude. Like I don't know how to talk to AI. So that's that's great. Um, but I mean like I wrote the readme, right? Like I I I'm the one coming up with this process, so I feel pretty confident in it.

Claude 特别擅长给 Claude 自己写 skill。我本来也不知道该怎么跟 AI 讲话，所以这挺好的。不过 readme 是我自己写的嘛——这套流程是我想出来的，所以我对它还是挺有底气的。

[43:11] Peter

Cool. All right. Well, this is awesome demo. I I think we have one more topic, which is uh Haml, you you wrote a article about do AI automations actually work or like where do they work, right? Do you want talk about that?

酷。好，这个 demo 太赞了。我想我们还剩最后一个话题：Hamel，你写过一篇文章，讲 AI 自动化到底管不管用、或者说在哪些地方才管用，对吧？要不你讲讲？

[43:23] Hamel

Yeah, I can quickly talk about that. Okay. Um, so I have a blog post, do automated evals work? And y'all, you we can put the link in the description, but basically what we did is, let me just back up. Um, so I wrote a blog post about that's titled, do automated evals work? And the reason for writing this blog post is there is a bunch of different vendors that in the last few months have created tools that will that promise to just do evals for you. And the way they work is you upload your traces into those tools and you chat with an AI. So here's an example. Um here's one in Brain Trust where you know is a trace viewer on the left. Um, and then you can chat with your AI, you can ask it some questions and basically say do my evals for me. And this particular product, this feature is called loop. Um, and uh, Arise has really the same thing, a very similar thing. It's called Alex. You can chat with it. You can say, "Hey, look through my traces, do my evals." Langmith has something similar. And um you know we were curious like okay if we take a data set that we have annotated as humans how good is the AI at catching those same errors and how good are these automated eval systems and it turns out that um and here's a fun meme. It's like okay you're faced with this choice right as a developer or as a product manager like do you just press this auto eval button sounds really convenient or do you actually look at the traces? It's actually a pretty hard decision because like it's very tempting to press the button and um and so I won't get too much into the details but um we did kind of a benchmark and we found that okay like the tools are able to recover a lot of the errors that a human would.

好，我可以快速讲一下。我有一篇博客叫《Do Automated Evals Work?》（自动化 evals 到底管不管用？），链接我们可以放在描述里。我们做的事情基本上是——我往回倒一点讲。我写了一篇博客，标题就叫「自动化 evals 管用吗」。写它的起因是：过去几个月冒出来一堆厂商，做了各种工具，号称能帮你把 evals 全包了。它们的用法是：你把自己的 trace 传上去，然后跟一个 AI 聊天。举个例子，这是 Braintrust 里的样子，左边是 trace viewer，然后你可以跟 AI 对话、问它一些问题，说白了就是跟它讲「帮我把 evals 做了」。他们这个功能叫 Loop。Arize 也有几乎一模一样的东西，叫 Alyx，你同样可以跟它聊：「嘿，把我的 trace 都看一遍，帮我做 evals」。LangSmith 也有类似的。我们就很好奇：如果拿一份我们人工标注过的数据集，AI 抓同样这些错误能抓得多准？这些自动化 eval 系统又有多强？结果发现——这里还有个好玩的梗图——你作为开发者或者产品经理，面对的就是这么个选择：是直接按下这个「自动 eval」按钮（听着可太省事了），还是老老实实自己去看 trace？这其实是个挺难的决定，因为按按钮的诱惑真的很大。细节我就不展开了，我们做了个类似 benchmark 的东西，结论是：这些工具确实能把人能找出来的错误召回相当一部分。

[45:21] Hamel

But u on the flip side it these automated tools all sort of miss the same thing which is uh finding errors that are not obvious things that require product judgment and kind of taste. So like anywhere in the trace where something obviously went wrong like a tool call failed or somebody's like expressing dissatisfaction or something is just like really obvious like you don't need product expertise to know something went wrong. AI is really good at catching that. and it's not good at catching like oh okay this uh sa this support bot didn't handle sales objections correctly you know so like what we uploaded is a customer service uh it's it was a real estate or rental sales agent that is you know supposed to interface with people looking to buy apartments or rent apartments and like yeah for example one of the items that were always missed were sales objections like hey we don't it would just say hey we don't have that available have a nice day instead of saying hey like there's alternatives for you blah blah but another kind of very interesting fact is we also benchmarked this against coding agents so clawude codeex so on and so forth and we found that those pretty much work the same so at the end of the day what you know what's going on here is the harness is somewhat thin it's just someone else's prompt some you know so when you're chatting with uh the AI these tools, you're really just invoking one of those models and they have a prompt behind the scenes that's trying to do the same thing. But what you really want to do is you want to steer the agent more actively. So the way to make this better across the board for either you know your coding agent or the eval tools is to give it an idea of what's good and bad. The only thing is you don't know what's good and bad unless you do what Shrea just did. It's really hard for humans to upfront think of like from top down like what are all the things that

但反过来看，这些自动化工具无一例外都漏掉了同一类东西：那些不明显、需要产品判断力和品味才能看出来的问题。也就是说，trace 里凡是「一眼就知道错了」的地方——比如某个 tool call 失败了、用户明显在表达不满，这种根本不需要产品经验就能判断出问题的——AI 抓得非常好。但它抓不出来的是：哦，这个客服 bot 没有正确处理销售异议。我们传上去的是一个客服场景，是个房产／租房销售 agent，要跟想买房或租房的人打交道。举个例子，被一致漏掉的一类就是销售异议：客户问了，它就回一句「不好意思我们没有这个房源，祝你愉快」，而不是说「这边还有别的选择可以给你」等等。另外还有个很有意思的事实：我们还拿 coding agent 做了对照——Claude Code、Codex 之类的——发现效果基本一样。所以说到底，这里的情况是：这层 harness 其实挺薄的，无非就是别人写好的一段 prompt。你在这些工具里跟 AI 聊天的时候，调用的其实就是那几个模型，背后有一段 prompt 在做同一件事。但你真正该做的，是更主动地去牵引这个 agent。所以不管是你自己的 coding agent 还是这些 eval 工具，要让它整体变好，办法都是告诉它什么叫好、什么叫坏。唯一的问题是：除非你做一遍 Shreya 刚才演示的那套，否则你根本不知道什么算好、什么算坏。让人一上来就自顶向下地想清楚所有

[47:25] Hamel

are good and bad. It's like almost impossible.

好与坏的情况，几乎是不可能的。

[47:28] Hamel

Um and so that's really what we found here is like okay they're all missing the same thing. So but it's useful. It's not that it's not useful. It can catch a lot of things. It just won't catch some of the important things especially those that are likely to make your product work well. And so you should think like okay all your competitors are going to be using this. This is like a baseline of like you can of course you can point claude at your product and say like find all the errors and your competitor is going to do that too. So the thing that's going to matter at the end is

所以我们发现的核心就是：它们漏掉的都是同一批东西。但它有用，我不是说它没用，它能抓到很多问题，只是抓不到其中一些最要紧的——尤其是那些真正决定你产品好不好用的问题。所以你得这么想：你的竞争对手也都会用这套东西。这只是一条基线——你当然可以把 Claude 指向你的产品说「把所有错误都找出来」，你的竞争对手同样会这么干。所以最后真正拉开差距的，是

[48:03] Hamel

how much taste can you infuse into your product beyond that uh to to like define like what is good. And so that's why you need to look at your your um you need to look at your data and I spelled it out here um you know I spell out like okay these are the kinds of things missed for example you know sales objections in this example this example it was multi-channel so customers of this particular agent can talk on the web and text make a phone call they the eval didn't know that so the eval agent didn't catch that like hey like we shouldn't be putting markdown in t text text messages it doesn't work.

在这之上你还能往产品里注入多少「品味」，去定义什么才算好。这就是为什么你必须去看你自己的数据。我在这里都写清楚了：比如这些是被漏掉的问题类型，像刚才说的销售异议。而且这个例子还是多渠道的——这个 agent 的客户可以在网页上聊、可以发短信、也可以打电话，而 eval 并不知道这件事，所以 eval agent 就没抓到「我们不该在短信里放 markdown，那东西在短信里根本不生效」这种问题。

[48:45] Hamel

Um yeah so things like that like of course you can give this context to the LLM but this is not something you might think about giving as criteria up front. You would have to see that failure.

对，就是这类事。你当然可以把这些上下文喂给 LLM，但这不是你一开始就能想到要写进评判标准里的东西——你得先亲眼看见那次失败才行。

[48:55] Hamel

Um so so yeah that's that's the gist of it. Um we also uh have this recording. Well you should actually cut that off. So um that's the just blog post. Um

嗯，大概就是这么回事。我们还录了个……这段你等会儿剪掉吧。总之这就是那篇博客。

[49:07]

yeah.

嗯。

[49:08]

Okay.

好。

[49:09] Hamel

Yeah, just sharing that in case that's useful.

对，就是分享一下，说不定对大家有用。

[49:11] Peter

Okay. So the takeaway is like this stuff will get you to a good baseline, but if you give a demo of your product, you just do the manual reviews. That's the take away, right? The manual

好。所以结论是：这类工具能把你带到一个不错的基线，但如果你要拿自己的产品去做 demo，你还是得自己做人工 review。结论就是这个吧？人工

[49:20] Hamel

review. Use use the automated tools and also look at data yourself. Like it's a competition, right? You you got to be better than your competitors.

review。自动化工具照用，同时你自己也要看数据。这是一场竞争嘛，你得比你的竞争对手做得更好。

[49:29] Hamel

Yeah. So one takeaway is okay the coding agents are just as you know are at par with these auto eval tools. The only benefit of auto eval tools is they're integrated into the the rest of the stack. Like

对。所以有一条结论是：coding agent 跟这些自动 eval 工具其实是一个水平的。自动 eval 工具唯一的好处，是它跟你整套技术栈是打通的。比如说

[49:44] Hamel

yeah they're in the ecosystem. So if you store your dasis traces in in Langmith you should totally use lang like it's a no-brainer.

对，它们本来就在生态里面。所以如果你的数据和 trace 本来就存在 LangSmith，那你完全应该用它，这根本不用犹豫。

[49:52] Shreya

I see. I think it was pleasantly surprising for me because I wasn't involved in writing this blog post, right? It was Hamill. Um I I was so s I was very happy to find that oh these these off-the-shelf tools work so well. So it's great like you should absolutely use them and and find all the errors in your data but also you know look at your data for more errors and also look at the errors that for example linksmith found just to make sure you agree with them right because the precision on these is is like 80% to 90% in the best case. Um so that means 10 to 20% of the errors found by the automated eval tool are actually not errors. they're like red herrings and might distort your product if you just listen to that automatically. Right? So, it's very important to look at both recall and precision.

明白。这个结果对我来说其实是个惊喜，因为这篇博客我没参与写，是 Hamel 写的。我发现这些现成工具居然这么好用的时候特别开心。所以这些工具你绝对该用，用它们把你数据里的错误找出来；但同时你自己也得去看数据、找更多它们没找到的错误，而且还要回头核对一下——比如 LangSmith 报出来的那些错误，你得确认自己同意它的判断。因为这些工具的 precision 最好的情况下也就 80% 到 90%，也就是说自动 eval 工具报出来的错误里，有 10% 到 20% 其实根本不是错误，纯属干扰项；你要是无脑照单全收，反而会把产品带偏。所以 recall 和 precision 这两头都得看，这非常重要。

[50:39] Peter

Okay. So, basically actually looking at stuff is is the edge. [laughter] Actually reading and looking at stuff

好。所以说到底，真的动手去「看」东西才是那个优势所在。（笑）真的去读、去看

[50:45]

always.

永远都是。

[50:47] Peter

Yeah. Cuz because like you know with all this AI generated stuff you can just like let let it go. But like actually reading stuff this is what I learned from another guest that I had to just actually read what what it produces.

对。因为你也知道，面对这些 AI 生成的东西，你很容易就撒手不管、随它去了。但真正去读它产出来的内容——这是我从另一位嘉宾那儿学到的，就是你真的得去读 AI 到底产出了什么。

[50:58]

Oh yeah. Yeah. And even like sometimes with code, that's an edge.

哦，对，对。而且有时候连代码也一样，这就是优势所在。

[51:03]

Yeah.

对。

[51:03]

You need selectively code, it's definitely an edge.

你得有选择地去读代码，这绝对是一种优势。

[51:06] Peter

Cool. All right. So, why don't we kind of summarize what we covered in this episode? I think we covered a lot. Um, so I I think my number one takeaway is to actually use SH's scale because uh I think it's pretty awesome and um give a bunch of examples and then have it help you identify like the common themes of errors, right? That's kind of number one. And number two from from your example is like use all the coding harnesses and like the eval stuff, but also just read the stuff. That's [laughter] that's kind of my two takeaways. So uh thank you for sharing all this information for free. Can you guys talk about what is covered in your eval course that's that we didn't cover here or like you know what was the more advanced stuff?

好。那我们把这一期聊过的东西小结一下吧？我觉得我们讲了特别多。嗯，我最大的一个收获是：真的去用 Shreya 的那个 skill，因为我觉得它相当厉害——喂给它一堆例子，然后让它帮你归纳出错误的共性主题，对吧？这是第一点。第二点来自你举的例子：各种编码 harness、eval 工具都用起来，但同时也要自己去读那些东西。[笑] 这大概就是我的两个收获。那么，谢谢你们免费分享了这么多。你们能讲讲你们那门 eval 课程里都覆盖了哪些今天没聊到的内容吗？或者说，更进阶的部分是什么？

[51:46] Shreya

Yeah, so here hopefully you got a taste of our philosophy of like how you're really going to differentiate your AI generated product. Um, so there's some our our course has five modules. It's totally revamped for this era of like agentic AI. So beyond just teaching you the error analysis philosophy, we also teach things like okay, how do you really productionize this work and scale it up to work within your company? For example, like if you have multiple people working on the same product, I need to align on criteria. How do you integrate this stuff into, you know, CI/CD for example? um how do you monitor over time like drift over many months of your deployment. We also have a module on safety and kind of adversarial evaluation. So it's really important you know how do you make sure you're not leaking tenant data? How do you make sure that you can evaluate that over time especially when you want to deploy to your actual customers. Um and then finally the last module is all about improvement. So it's cost improvement and accuracy improvement. really those both of those go hand in hand, right? If you switch to a cheaper model, how do you improve the accuracy of your cheaper model? So, it's both cost and accuracy improvement. Um, and really no one else talks about this, right? If the the benefit of having evals is that you can then put put it in an auto research kind of loop. Um, what are the kinds of strategies to invoke? How do you make sure that you know you we found that in in our um consulting cases or even for some of our students we're able to cut cost by 100x truly compared to um like using Opus or probably Fable will be even more um so we're really excited to kind of teach people the techniques to do that.

好。希望大家从今天的内容里已经尝到了我们这套理念的味道——就是你到底要靠什么，让自己的 AI 产品真正做出差异化。嗯，我们的课程有五个模块，为 agentic AI 这个时代做了彻底重做。所以除了教你错误分析（error analysis）这套理念之外，我们还会讲：好，你怎么把这套做法真正产品化、放大到能在你们公司内部跑起来。比如说，如果有好几个人在做同一个产品，那就得先在评判标准上对齐；再比如，怎么把这套东西接进 CI/CD；还有怎么做长期监控，比如上线好几个月之后出现的 drift（漂移）。我们还有一个模块讲安全和对抗性评测。这个真的很重要——你怎么确保自己没有泄露租户数据？你怎么确保这件事能被长期地评估，尤其是当你要真正部署给客户的时候。嗯，最后一个模块全是关于「改进」的：成本的改进和准确率的改进。其实这两件事是一体两面，对吧？如果你换成一个更便宜的模型，那你怎么把这个便宜模型的准确率提上去？所以是成本和准确率一起优化。嗯，而且这块基本没别人在讲，对吧？有了 eval 的好处就是，你可以把它丢进一个自动化的「研究循环」里去跑。那具体该动用哪些策略？你怎么确保……我们发现，在我们做的咨询项目里，甚至在我们一些学员身上，成本是真的能砍到原来的百分之一——这是跟用 Opus 比；换成 Fable 的话，省下来的可能还更多。所以我们特别期待把这些技巧教给大家。

[53:22] Peter

Awesome. And uh yeah, I'll I'll put the link to the course in description. And I I think uh from from now on I'm gonna interview Straas and Haml. Never Ham by himself because I learned a lot more. [laughter]

太棒了。嗯，我会把课程链接放在视频简介里。而且我觉得，从今往后我要采访就得把 Shreya 和 Hamel 一起请来，绝不再单独请 Hamel 一个人了，因为这样我学到的东西多太多了。[笑]

[53:34]

Yeah. Even just [laughter]

对。哪怕只是……[笑]

[53:39]

yeah. Yeah.

嗯，对。

[53:39]

I'm a full-time educator now. So

我现在可是全职做教育的人了。所以——

[53:42] Peter

yeah. Yeah. That that that's amazing. Thanks for sharing all this knowledge and um yeah, I'm gonna go off and use your skill right now. So we'll see how it goes.

对，对。这太厉害了。谢谢你们分享这么多干货。嗯，我这就下播去用你那个 skill，看看效果怎么样。

[53:50]

Cool. Thanks for having us, Peter.

好嘞。谢谢你请我们来，Peter。