Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind
频道: AI Engineer
视频: https://www.youtube.com/watch?v=0vphxNt4wyk
原文语言: en
统计: 共 12 轮 · Philipp 12
[0:01] Philipp
[music]
[音乐]
[0:12] Philipp
Yes, so hi everyone. My name is Philip. I'm based out of Germany. I'm part of the Google DeepMind team mostly working on Gemini API and agents. And we are going to talk about why you should not ship skills without eval. And maybe before we start, I need a little bit of your help. So if you could raise your hands if you use coding agents to write code. So yeah, hopefully every every hand goes up, right? And do you use skills with it? Okay. Do you have evals for those skills? Okay, yeah, that's um not a lot of hands. Everyone uses skills, no one has evals. Hopefully we can fix that today. And like very important is like why checks fail in productions. And um Skill Bench is a very popular and nice like eval or benchmark which um index like over like 50,000 skills from like it up and like try to look into him and almost none of those skills had evals. Most of them were AI written um not really tested and it's very hard to know if your skill is good or bad because like agents are really non-deterministic. So you might not know if your uh task fails because your skill is bad or if your task fails because it's way too challenging for the model. So um very important um before we go into it there's I want to like really make sure that we know the difference between the agents we use and the agents we build. Um most of us use agents for writing code, doing productivity work. That's the agents we use. It's like anti-gravity, cursor, Claude code. And there are you are the engineer and you have context about skills, right? If you write some prompt to I don't know, like help me build a new Gemini API feature. And if your agent does not invoke the skill on the first time, you will notice it very quickly. You stop your your task and reprompt it or like use slash commands for for triggering those skills. When you build an agent inside your application for consumer or customers, they have no idea about what a skill is. They don't start their prompt with use customer support skill to like help me refund or use refund skill to help me solve my problem.
大家好，我叫 Philipp，来自德国，是 Google DeepMind 团队的一员，主要负责 Gemini API 和 agent 方向的工作。今天我们要聊的是：为什么你不该在没有评测（eval）的情况下就把 skill 发布出去。开始之前，先请大家帮个忙——用 coding agent 写代码的请举手。好，基本上所有人都举手了，对吧？那有在用 skill 的呢？好。那给这些 skill 写过 eval 的呢？嗯……举手的就没几个了。人人都在用 skill，却没人做 eval，希望今天能把这个问题解决掉。很关键的一点是要搞清楚东西为什么会在生产环境里失效。Skills Bench 是一个很流行、很不错的评测基准，它从 GitHub 上索引了超过 5 万个 skill，逐一分析后发现，这些 skill 几乎都没有 eval。其中大部分是 AI 写的，没经过真正的测试。而 skill 到底好不好其实非常难判断，因为 agent 本身就是高度非确定性的——任务失败了，你分不清是 skill 写得差，还是任务本身对模型来说太难了。在深入之前，我想先明确一个重要区分：我们「使用的 agent」和我们「构建的 agent」是两回事。我们大多数人用 agent 来写代码、做效率工作，这是我们使用的 agent，比如 Antigravity、Cursor、Claude Code。在这些场景里，你自己就是工程师，你清楚 skill 的存在——比如你写个 prompt 说「帮我做一个新的 Gemini API 功能」，如果 agent 第一次没有调用 skill，你马上就会发现，然后停掉任务重新 prompt，或者用斜杠命令手动触发那个 skill。但当你在自己的应用里为消费者或客户构建 agent 时，用户根本不知道 skill 是什么东西。他们不会在 prompt 开头写「请使用客服 skill 帮我退款」或者「用退款 skill 解决我的问题」。
[2:22] Philipp
So, there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills in like the context there. And what is a skill? I mean, every one of us knows hopefully in by now what a skill is. It's like basically really a folder with a skills.md file in it and then some additional assets to make that skill really work. And the big difference with skills is that they work on progressive disclosure. So, most of the skills start very small. So, you have the title and a description. The description is normally part of the model's context. So, the model knows when to use the skill. Second layer is we have a skills body with more instructions, more details, and hopefully more references to external files. And then you can really go deep in those reference files where there's all of the context the model needs to discover to to solve the task. And I like to differentiate between two kinds of skills. So, they are capability skills and preference skills. Capability skills teach models something they cannot do consistently at the moment. Maybe it's like, I don't know, like tracing some logs, creating a new React app. And those capability skills are temporary. So, the better our model gets, the more likely it is that we can remove those skills. And Evals will tell us when we can retire skill and when not. And then we have preference skills. Those are more durable, mostly encode some references. So, if you have a specific workflow in your team or a specific style language or other preferences which are very specific to your company, Um will have or create preference skills and those uh preference skills are then protected with e-walls where because most of like the foundation models might not uh integrate the knowledge which is very specific to your use case or your domain. And preference skills are very valuable, so we really want to make sure that those are working and we don't like update our agents to uh degrade performance. So, do skills work?
所以，「我们使用的 agent 里怎么用 skill」和「我们构建的 agent 里客户会怎么用 skill」，这两者的语境差别很大。那 skill 到底是什么？相信在座各位现在都知道了——它本质上就是一个文件夹，里面有一个 SKILL.md 文件，再加上一些让这个 skill 真正能跑起来的附加资源。skill 最大的特点是「渐进式披露」（progressive disclosure）。大多数 skill 的起点都很小：第一层是标题和描述（description），描述通常会常驻在模型的上下文里，让模型知道什么时候该用这个 skill；第二层是 skill 的正文，包含更多指令、更多细节，最好还有指向外部文件的引用；第三层就是那些引用文件，模型可以深入进去，找到解决任务所需的全部上下文。我喜欢把 skill 分成两类：能力型 skill（capability skills）和偏好型 skill（preference skills）。能力型 skill 教模型做它目前还做不稳定的事，比如追查日志、创建一个新的 React 应用。这类 skill 是临时性的——模型越强，这些 skill 就越可能被移除，而 eval 会告诉我们什么时候可以让一个 skill 退役。偏好型 skill 则更持久，主要编码的是一些偏好和约定：比如你们团队特有的工作流、特定的文风语言，或者其他非常公司特有的偏好。这类 skill 要靠 eval 来保护，因为基础模型大概率不会内置这种只属于你的用例、你的领域的知识。偏好型 skill 非常有价值，所以我们要确保它们真的在起作用，不要在升级 agent 的时候把性能搞退化了。那么，skill 到底管不管用？
[4:26] Philipp
Yes, they do work and I going back to a skills bench which has an update of 1.1 which has evaluated all kinds of open and closed models in different harnesses showing that skills on average improve the performance by roughly 15%. Skills bench covers around 100 different tasks uh based on like coding and also productivity across different languages. It's uh openly available and they have a very nice website, a very nice leaderboard, are also very open for uh community contributions. And then they did a second analysis on self-generated or AI-generated skills, right? It's very easy if you are in a coding agent and you work on something, tell the model create a skill and then it writes a skill.md file. We maybe look at it very closely. It roughly covers what we want to do and then we just accept it and start using it. And what I found out is that human-written skills are the best we can provide. Uh AI-generated skills can uh impact performance negatively. And that skills or skills.md files should be below 500 lines of words. So, if you have your laptop open and have a skill available, if you open that and if it's above 500 lines, you should definitely look at the skill after our session. And um the last topic about what is a skill and how a skill works, uh we have different ways of triggering our skill, right? We can have a model-triggered skill meaning uh based on the context and the description, the model decides to use or read a skill to, uh uh, get more context to solve a task. And then there are user-invoked skills. And I think people underestimate how powerful user-invoked skills are. Um, and they most of the time just accept, uh, the overhead by pro- like adding it into the context. I have like many user-invoked skills for more workflow type of tasks like creating a pull request, uh, staging documentation, and all like of the very uh, normal dev work which could be run in a script should most likely be a user-invoked, uh, skill. And when you build agents for customer, you don't have those user-invoked skills. We are only working in the model-invoked skills, and that's where we are like focusing on for the small eval section we are going to look in a second.
答案是管用的。回到 Skills Bench——它更新到了 1.1 版本，在不同的 harness 里评测了各种开源和闭源模型，结果显示 skill 平均能把性能提升大约 15%。Skills Bench 覆盖了约 100 个不同任务，涉及编程和效率类工作，横跨多种编程语言。它是完全开放的，网站和排行榜做得都很不错，也非常欢迎社区贡献。他们还做了第二项分析，针对自动生成、也就是 AI 生成的 skill。这事太容易了，对吧？你在 coding agent 里干着活，顺口让模型「创建一个 skill」，它就写出一个 SKILL.md 文件，我们也许粗略瞄一眼，觉得大致覆盖了想做的事，就直接接受开始用了。而分析发现：人写的 skill 才是质量最好的，AI 生成的 skill 反而可能对性能产生负面影响。另外，skill 也就是 SKILL.md 文件应该控制在 500 行以内。所以如果你现在开着电脑、手头有个 skill，打开一看超过 500 行，那今天这场分享结束后你真的该好好审一下它。关于 skill 是什么、怎么工作的最后一个话题：skill 有不同的触发方式。一种是模型触发（model-triggered）——模型根据上下文和描述，自己决定要不要读取某个 skill 来获取更多上下文、解决任务。另一种是用户主动调用（user-invoked）。我觉得大家低估了用户主动调用型 skill 的威力，多数人只是默默接受了把它塞进上下文带来的开销。我自己有很多用户主动调用的 skill，用于偏工作流类的任务，比如创建 pull request、发布文档——凡是那些可以用脚本跑的常规开发工作，大概率都应该做成用户主动调用的 skill。但当你为客户构建 agent 时，是没有用户主动调用这条路的，只能依赖模型触发的 skill。这也正是我们接下来要看的这个小型 eval 环节的重点。
[6:34] Philipp
So, writing skills, um, is an important topic. Uh, we're going to look at eight, um, examples on how or tips on how you can write good skills. And most importantly, if you work with model-invoked skills is the description because the description are most of the time two sentences we provide to the system instruction to help the model know when it should use a skill or not. And it's bad if your description is too weak because then it might trigger too often or the it might not be triggered if you need it. So, very important is the the why and the how for the model. So, why it should use that skill and then how it should use that skill. Um, very common is like use that skill if you are working on a React application, for example. And then, of course, the when. And we should write directives instead of essays. So, we should not say something like, "Hey, the Interactions API is recommended for multi-chat, um, multi-chat because it handles like session state and it's like where you should be way more directive like use the Interactions API if you're working on like a chat application. So, you need to give the model like clear instructions and directives on when it should use the skill and how it should use the skill. And similar to what we have seen in the skills bench results, we should keep the skill lean and layer information. So, the description is the cost you always pay on every model invocation. So, on every model call, the description is part of the model context. So, you always pay that 100 200 tokens cost and you don't want to have a super long description because then you always have to pay that. When you have a very long skill MD file, it will be always read into context when the model decides to read the the skill or to use the skill. Uh which can be expensive as well. That's why we want to keep it as concise as possible, but still include all of the reference and details for the model to solve the task. And then of course, the layer three is like we can have those reference files where the model needs to like um go really deep into a very specific task.
接下来说写 skill，这是个很重要的话题。我们会看八条关于如何写好 skill 的建议。首先，如果你做的是模型触发型 skill，最重要的就是描述（description），因为描述通常就是我们放进系统指令里的那两句话，用来帮模型判断该不该用这个 skill。描述写得太弱是很糟糕的：要么触发得太频繁，要么该触发的时候不触发。所以关键是给模型讲清楚「为什么」和「怎么用」——为什么该用这个 skill，以及该怎么用，再加上「什么时候用」，比如很常见的写法是「当你在开发 React 应用时使用此 skill」。第二，要写指令，不要写散文。不要说「Interactions API 适合多轮对话场景，因为它能处理会话状态」这种话，而要更加直接：「如果你在做聊天类应用，就用 Interactions API」。你得给模型明确的指令：什么时候用这个 skill、怎么用。第三，和 Skills Bench 的结论一致：让 skill 保持精简，信息分层。描述是每次模型调用都要付的成本——每一次模型调用，描述都在上下文里，那一两百个 token 你是永远要付的，所以描述不能写得太长。而 SKILL.md 文件很长的话，模型一旦决定读取这个 skill，整个文件都会被读进上下文，这同样很贵。所以我们要尽量精炼，但又要保留模型解决任务所需的全部引用和细节。当然还有第三层，也就是引用文件，供模型在需要深入某个非常具体的任务时去查。
[8:41] Philipp
And a good example for this is like if you are working in like maybe a multi-cloud environment and you have a skill to deploy your application, you might need instruction to deploying to AWS and deploying to Google Cloud. Those should not be part of your skill MD file. Those should be references. That you have a reference for AWS, reference for Google Cloud, maybe a reference for Azure so that the model can basically explore based on the context where it should go to get all of that information. Then we should set the right level of freedom. Um I see many people very clearly describing the exact workflow in a skill. Step one, go there. Step two, do this. Step three, do this. If you have those type of use cases, you should not use skills. You maybe you should write a script because if the the process or the workflow is always the same, you don't need to waste models and tokens for that exercise. You can create a script. You can tell the model use that script to run a specific workflow. So, rather define goals and constraints. So, if you need to like deploy to your update or stage your documentation, describe how the model can do that. Or like for your database updating a config, you should not say like read the config, update the port, and then like deploy again. The model knows what to do. Just like hey, if we need to change the config, here's the file, make the change. Then uh don't skip negative cases. So, we always look at the when we want to use this skill, but most of the time we don't look at when we don't want to use this skill. So, if we have a description for our skill which says use it for web development tasks, it might over trigger. Maybe you work with React, maybe you also work with Angular, and the model always loads the skill if you are working in like a web development environment, but if you are very specific for like hey, only use that skill for React components or for Tailwind CSS, then the model knows hey, that's very specific for one to use. And with Evals, we can also identify those. Um And then test early. So, that's what we are going to look at. We should really try to test when you create a new skill. Always try to create 10 of 20 prompts.
举个很好的例子：假设你在多云环境下工作，有一个部署应用的 skill，可能既要有部署到 AWS 的说明，也要有部署到 Google Cloud 的说明。这些不应该写进 SKILL.md 主文件，而应该做成引用文件——一份 AWS 的、一份 Google Cloud 的，可能再加一份 Azure 的，让模型根据上下文自己去探索该读哪份来获取信息。第四，设定合适的自由度。我看到很多人在 skill 里把工作流写得死死的：第一步去哪里，第二步做什么，第三步做什么。如果你的用例是这样的，那你根本不该用 skill，你该写个脚本——流程每次都一模一样的话，何必浪费模型和 token 呢？写个脚本，然后告诉模型「用这个脚本跑这个工作流」就行。所以正确做法是定义目标和约束。比如要部署更新或发布文档，就描述模型可以怎么做到；比如要改数据库配置，不要写「先读配置，再改端口，然后重新部署」——模型知道该怎么做，你只要说「需要改配置的话，文件在这里，改就是了」。第五，别漏掉反例。我们总在写什么时候该用这个 skill，却很少写什么时候不该用。如果 skill 的描述写的是「用于 Web 开发任务」，那就可能过度触发——你可能在写 React，也可能在写 Angular，只要是 Web 开发环境模型就把 skill 加载进来。但如果你写得很具体，比如「只在写 React 组件或用 Tailwind CSS 时使用」，模型就清楚这是专门给某种场景用的。而且通过 eval，我们也能把这类过度触发问题找出来。第六，尽早测试，这正是我们接下来要看的。创建新 skill 时一定要测，每次都写上 10 到 20 条 prompt。
[10:40] Philipp
I like to create five for like the happy path. So, when do I want to use that skill? Five when I don't want to use that skill just to make sure the model is not over triggering the skill and confusing itself. And then if you have already some customer or production traces, try to include those as well because nothing is better than than real-world data. And then tip seven which is quite new and I have to give all credits to Matt. So, if you don't know Matt, he's a great AI educator and you should definitely follow him. He published a tweet and also a skill on like killing all of the no-ops. And what he found is that AI generated skills tend to include a lot of no-ops. And no-ops basically is an instruction which does nothing to change the agent's behavior. It's like before making an implementation easy to read. Like the model knows how it when it should make something easy to read or write clear high-quality code. I mean, like that's what we expect from the model to do without telling it really. So, um definitely look at those no-ops. He have has published a very good skill in in his like skills repository. Uh and then last but not least, uh know when you should retire skill. Um skills are not there to live forever. Models get better, behaviors change, expectation change, um the environment changes. So, um always try to run evals with and without the skill enabled. And if the model achieves the performance without even like triggering the skill, you know you can retire that skill, save the cost uh for your tokens, and then also um don't keep like it redundant. So, save cost at the end and maintenance also as well. And to look at a little bit of a practical example and also how you can create your own small eval or eval harness for skills. Um earlier this year we wanted to create a new skill for the Gemini Interactions API. So, the Gemini Interactions API is our new interface for working with Gemini models and with agents. And the Interactions API was released after the last training of Gemini. So, the model or Gemini 3 and like 3.1 or even 3.5 has no context about what is the the Gemini Interactions API.
我喜欢写 5 条正向路径（happy path）的，也就是我希望触发这个 skill 的场景；再写 5 条不该触发的，确保模型不会过度触发、把自己搞乱。如果你已经有客户或生产环境的真实 trace，一定也要加进去，因为没有什么比真实数据更好。第七条建议比较新，功劳要全部归给 Matt——如果你还不认识 Matt，他是一位很棒的 AI 教育者，强烈建议关注他。他发过一条推文，还发布了一个专门清除「无效指令」（no-ops）的 skill。他发现 AI 生成的 skill 里往往塞满了 no-ops——也就是那些对 agent 行为毫无改变作用的指令，比如「实现之前先确保代码易读」。模型本来就知道什么时候该让代码易读、该写清晰高质量的代码，这是我们对模型的默认期待，根本不用专门交代。所以一定要检查这些 no-ops，他在自己的 skills 仓库里发布了一个很好用的 skill。最后一条：知道什么时候该让 skill 退役。skill 不是用来永远活着的——模型会变强，行为会变化，期望会变化，环境也会变化。所以要经常跑两组 eval：开启 skill 一组、关闭 skill 一组。如果模型不触发 skill 也能达到同样的表现，你就知道这个 skill 可以退役了，既省 token 成本，也免得留着冗余的东西增加维护负担。接下来看一个实际的例子，也顺便讲讲你可以怎么为 skill 搭一套小型 eval 或者说 eval harness。今年早些时候，我们想为 Gemini Interactions API 创建一个新 skill。Gemini Interactions API 是我们用于调用 Gemini 模型和 agent 的新接口，它是在 Gemini 上一次训练之后才发布的，所以不管是 Gemini 3、3.1 还是 3.5，模型对 Gemini Interactions API 是什么完全没有概念。
[12:57] Philipp
So, we decided, "Okay, let's look uh at creating a skill to help the model create good code for the Interactions API, to use the latest models." And to do that, we created 117 test cases. Those are uh based on like data we see uh from real users trying to generate uh Gemini code from um synthetic generated uh test cases, and also from like feedback we see people like, "Hey, the model is like using Gemini 2.0 even if we are already on 3.0." And the the end result was that we improved the the performance up to like almost 90% for generating valid interactions API code with the latest Gemini models. And to do this, we basically only needed like two very simple uh assets. So, one of that was a JSON file with all of our test cases. And it's very like no clear structure. It's like, "Hey, we have a prompt." That's basically what we expect the user to provide. We have a language because we wanted to test the skill against TypeScript and Python. Uh we have a should trigger. That's basically there to tell us if the agent should read the skill or not read the skill. And then we have different expected checks. Uh we look at them in a little bit. Those are basically uh very simple asserts uh for that prompt if it should trigger or not. And then we have a very basic Python script which runs a coding agent. In this case, it was the Gemini CLI which uh passes the output and returns runs it so we can like take a look at the the outcome whether we have valid code for the interactions API or not. And uh most of the tests or evals for skills can be regex. It's like very amazing how good of regex you can write using coding agents. And it's really for us it was all about, "Okay, do we use the correct SDK? Do we use the correct um model? Do we use the correct methods? Do we use any old patterns?" And we created um very basic asserts for all of those cases, which are very cheap to run. So, we can run our skill against the evals many times. So, if a new model releases, we have a very easy way to update those asserts to the latest model IDs. And it's very cheap to run for it well because we don't need to use like LLM as a judge.
于是我们决定：好，那就做一个 skill，帮模型写出正确的 Interactions API 代码、用上最新的模型。为此我们创建了 117 个测试用例，来源有三：真实用户尝试生成 Gemini 代码的数据、合成生成的测试用例，还有用户反馈——比如「都已经是 3.0 了，模型还在用 Gemini 2.0」这种。最终结果是，生成有效的 Interactions API 代码并使用最新 Gemini 模型的成功率提升到了接近 90%。而做到这一切，我们其实只需要两个非常简单的东西。第一个是一份 JSON 文件，装着全部测试用例，结构非常直白：有一个 prompt，就是我们预期用户会输入的内容；有一个 language 字段，因为我们要在 TypeScript 和 Python 两种语言上测试这个 skill；有一个 should_trigger，用来标注 agent 应不应该读取这个 skill；然后是各种 expected checks（预期检查），一会儿我们会细看——本质上就是针对这条 prompt 该不该触发的一些非常简单的断言。第二个是一个很基础的 Python 脚本，它负责跑一个 coding agent——我们用的是 Gemini CLI——解析输出并运行，这样我们就能检查产出的 Interactions API 代码到底有没有效。而且针对 skill 的测试或者说 eval，大部分用正则就够了——用 coding agent 写正则的水平高得惊人。对我们来说，核心就是几个问题：用的 SDK 对不对？模型对不对？方法对不对？有没有混进旧的写法？我们为所有这些情况写了非常基础的断言，跑起来成本极低，所以可以反复拿 skill 去跑 eval。新模型一发布，我们也能很轻松地把断言更新到最新的模型 ID。整套东西跑起来非常便宜，因为完全不需要用 LLM 当裁判（LLM as a judge）。
[15:16] Philipp
But of course, you can use LLM as a judge if you have like more complex skills which need to look at the whole traces or the whole steps taken. And a very easy case is like you just create LLM as a judge with a rubric on like what you want to look at, and then like take the output, put it through the LLM as a judge, try to get a pass or a fail, and then if it fails, look at the data, and then like try to um improve your skill based on that. And that's also how we now uh eval skills at Google DeepMind. So, the we don't use YAML, but it's like just as like uh an example. Uh we have um tests or evals alongside every skill we have internally at Google DeepMind. Um every test has multiple cases with like a prompt. Uh we all run them in like clear workspaces, so you can define your workspace or environment if it should include additional files like your application environments. You have uh startup commands, which basically preloads or installs libraries into the environment. And then you have script evals or data. Those are those regex where we look at all of the traces to see what the skill triggered, was a certain command run, was a certain CLI run. And then we also have LLM as a judge where we have some expectations, which are basically matched against like hey, did it trigger the skill, did it run a certain bash command to like also evaluate it. And we run them on every change to the skill. So, if a change happens to or like a diff to the skill file, the eval will be run, and there will also be a result, and the change will not be merged if it is not improving the test cases. So, we always have those regression tests for every change to the skill, and you can only change the skill if it improves the eval or add new evals. And um yeah, that that's how we we we basically manage it. And then last but not least, uh 10 examples for best practices for skills. You don't need to take photos. They are in the blog post I can share later. So, um the I mean, we had it many, many times. The the skill skill description is very important.
当然，如果你的 skill 比较复杂，需要审视完整的 trace 或全部执行步骤，你也可以用 LLM as a judge。最简单的做法是：给 LLM as a judge 配一个 rubric（评分标准），写清楚你要看什么，然后把输出丢给它判定 pass 还是 fail；如果 fail 了，就去看数据，据此改进你的 skill。这也是我们现在在 Google DeepMind 内部评估 skill 的方式。我们内部不用 YAML——这里只是举个例子——但我们内部的每一个 skill 旁边都配有 tests 或 evals。每个测试包含多个 case，每个 case 有一条 prompt。我们全部跑在干净的 workspace 里，你可以定义自己的 workspace 或环境，比如要不要包含额外文件、你的应用环境之类。还有 startup commands，用来预先往环境里加载或安装依赖库。然后有脚本类 eval，也就是用 regex 去扫全部 trace，看 skill 有没有被触发、某条命令有没有执行、某个 CLI 有没有被调用。另外还有 LLM as a judge，我们写好一组预期（expectations），去比对：它触发 skill 了吗？它跑了某条 bash 命令吗？以此来做评估。而且 skill 每次变更我们都会跑这些 eval——只要 skill 文件有 diff，eval 就会执行并出结果，如果这次改动没有让测试用例变好，就不允许合并。所以每次改 skill 都有回归测试兜底：你只有在改动能提升 eval、或者补充了新 eval 的情况下才能改 skill。我们基本上就是这么管理的。最后，给大家 10 条 skill 的最佳实践。不用拍照，博客文章里都有，我稍后可以分享。第一条我们已经说过很多很多遍了：skill 的 description 非常重要。
[17:31] Philipp
Uh we have seen 50% of the failures uh because the skill was not triggered correctly because the prompt of the user was not uh detailed enough for the model to understand, "Hey, I need to use that skill to solve that task." And especially if you build agents for others, they are not aware of the skill descriptions you have for your model and for your skill. So, they might write something very um shallow and then the model needs to know, "Okay, I need to trigger that skill." Um we should write directors over passive information. So, we should always think about it. You should tell the agent what to do or not what to do and not just like, "Hey, if you feel happy today, please use the skill." Um include negative tests. Uh we always forget negative tests. Start small. Even like 10 to 20 skill eval samples are better than nothing. You will be surprised on how much you will find even from like five to 10 examples. And then like definitely create outcomes, not paths. Uh we don't want to test if the model loads the skill on like the first turn. We really want to test if it can achieve the task based on the prompt. And if it loads the skill, it loads the skill. If not, then not. If it um loads the skill after five turns, that's also okay. Then we want to have isolated runs because coding agents are very good at finding or cheating. So, if you run inside uh your existing environment, it might look up previous chats or it might look up some other executions and then like try to cheat it and get the context from the skill without even using the skill. Then definitely run more than one trial when running evals. Like agents, our models are non-deterministic. Maybe the first one works, the second one doesn't. So, always run the to six uh trials per case and to measure reliability Uh um test across different harnesses if you work with like or if you have employees or um people working with like different harnesses, not only just evaluate against Claude or anti-gravity. If you have people working with cursor, try to include them as well because agent harnesses behave differently and of course model behaves differently.
我们观察到 50% 的失败，都是因为 skill 没有被正确触发——用户的 prompt 写得不够详细，模型没法意识到“我需要用这个 skill 来解决这个任务”。尤其当你是给别人构建 agent 时，他们根本不知道你给模型和 skill 写了什么 description，可能只随手写一句很笼统的话，模型就得自己判断“好，我该触发这个 skill”。第二，写指令而不是被动陈述信息：要明确告诉 agent 该做什么、不该做什么，而不是“嘿，如果你今天心情好，请用这个 skill”这种。第三，要包含负面测试——我们总是忘掉负面测试。第四，从小做起：哪怕只有 10 到 20 条 skill eval 样本也比没有强，你会惊讶地发现，哪怕只跑 5 到 10 个例子就能暴露很多问题。第五，一定要考核结果而不是路径：我们不想测模型是不是在第一轮就加载了 skill，我们真正想测的是它能不能基于这条 prompt 完成任务。加载了 skill 就加载了，没加载就没加载，第五轮才加载也没关系。第六，要隔离运行，因为 coding agent 非常擅长“找捷径”或者说作弊：如果你在现有环境里跑，它可能会翻出之前的对话或其他执行记录，不用 skill 也能从中偷到上下文，把 eval 糊弄过去。第七，跑 eval 时一定要多跑几个 trial：agent 和模型都是非确定性的，可能第一次成功、第二次就失败，所以每个 case 跑两到六次 trial 来衡量可靠性。第八，跨不同 harness 测试：如果你的同事或用户在用不同的 harness，不要只针对 Claude 或 Antigravity 做评估，有人用 Cursor 就把 Cursor 也纳进来，因为 agent harness 的行为各不相同，模型的行为当然也不一样。
[19:35] Philipp
So, maybe your skill is very good with a Gemini but very bad with Codex and then you have uh customers, consumers using your harness with Codex and then it fails. And then um create your evals. So, if your um model is good enough that it doesn't need the skill anymore, keep that eval. You don't need to throw that eval away because you throw the skill away. You can keep that eval to make sure that the model or the agent keeps the performance and as soon as you start seeing some degradation, you can reintroduce the skill. You can maybe tweak some other tools or pieces to keep like the the performance up and then really detect when you can retire skill and you will be very surprised with all of the model updates how fast you can retire skill which you might need it like six months ago but not today anymore. And I have some homework for you. So, if you are back from holiday on Monday, um pick uh the most used skill and write five test prompts. Uh you can also use your coding agent and ask it to see look at your trajectories which are my most used skills and then try to create some some skills. It's you have seen it's like very easy to write your eval harness. It's like a JSON or YAML file and then like some Python script which runs your coding agent or your agent harness and then like look at the outcome. Definitely uh try to look at the removing no-ops. Maybe it does not change the eval performance but it helps you save cost because all of the tokens which are not helpful or not changing the agent behavior are money you will like spend. So, look at um writing great skills uh from Matt. It's you can find it on on GitHub and then also run ablation test. So, run always evals with your skill loaded and without your skill loaded. Only that way you will know when you can retire skill or if a skill is really helpful for your performance. So, don't ship skills without evals. Thank you.
也许你的 skill 配 Gemini 效果很好，配 Codex 就很差，而你的客户、你的用户恰好在用 Codex 跑你的 harness，那它就挂了。第九，保留你的 eval：如果模型已经强到不再需要某个 skill 了，把 eval 留着——skill 可以扔，eval 不用跟着扔。留着这个 eval，你就能确保模型或 agent 的表现不掉线；一旦发现性能开始退化，你可以把 skill 重新引入，或者调调其他工具和环节把性能撑住。第十，真正学会判断什么时候可以让一个 skill 退役——随着模型不断更新，你会非常惊讶：六个月前还必需的 skill，今天可能就不需要了。最后给大家留个作业：如果你周一度假回来，挑一个你用得最多的 skill，写 5 条测试 prompt。你也可以直接让 coding agent 去翻你的 trajectory，看看哪些 skill 用得最多，然后据此创建一些测试。你们也看到了，写一套 eval harness 其实非常简单：一个 JSON 或 YAML 文件，加一段 Python 脚本去跑你的 coding agent 或 agent harness，然后看结果就行。另外，一定试试删掉那些无效内容（no-ops）：也许删了之后 eval 表现不变，但能帮你省钱——所有对 agent 行为没有帮助、不产生改变的 token，都是你在白花钱。推荐去看 Matt 写的 writing great skills，GitHub 上能找到。还有，一定要跑消融测试（ablation test）：每次都分别在加载 skill 和不加载 skill 的情况下跑 eval，只有这样你才知道什么时候可以让 skill 退役、一个 skill 是否真的对性能有帮助。所以——别在没有 eval 的情况下发布 skill。谢谢大家。
[21:25] Philipp
[applause]
（掌声）