Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

频道: AI Engineer
视频: https://www.youtube.com/watch?v=jWq-aZIU0kM
原文语言: en
统计: 共 9 轮 · Ali Khial 8

[0:12] Ali Khial

Hello everyone. Um this is the last talk of this session. So hopefully it's going to be short. I know that you guys had to go through a long day. So try to keep it short and light for you all. Um I'm going to present myself. Um I'm Ali. I'm the director of AI and ML at G2I. Um I have zero experience in ML. So I don't know why they put the ML in my title. I'm a software engineer uh at heart. And to prove that I have more than 50 abandoned side projects in my machine. So uh you can know. So uh I'm going to make a disclaimer. The the title of the the presentation is a little bit misleading. Uh as I was working on it, I realized that it would be better if I presented my journey uh into benchmarks and what I learned instead of trying to find a dichotomy of the the bad, the ugly, and and the good. So um let's start with um I want to grab your attention. And I invite you to look at this. These beautiful three screenshots are a single prompt on one of the benchmark tasks. As I was looking at it, I was like how can an engineer write a task like this? So I said, "Nah, it's impossible. No one writes prompts like these ever." But I wanted to double-check with my engineers. So I took three of our best engineers. I showed them the prompt and I said, "Would you ever write a prompt like this?" And the answer was no.

大家好。这是本场最后一个 talk 了，所以我尽量讲短一点。我知道大家熬了一整天，那就轻松点、快点讲完。先自我介绍一下：我叫 Ali，是 G2i 的 AI 与 ML 总监。不过我在 ML 上的经验是零，所以我也不知道他们为什么要把 ML 塞进我的头衔里。我骨子里是个软件工程师——证据就是我电脑里躺着 50 多个烂尾的 side project，你们懂的。然后我得先声明一句：这个演讲的标题其实有点误导。我在准备的过程中意识到，与其硬把这些东西分成「好的、坏的、丑的」三类，不如直接讲我自己一头扎进 benchmark（基准测试）之后的经历，以及我从中学到了什么。那我们就从一个能抓住你们注意力的东西开始。请大家看这里——屏幕上这三张漂亮的截图，其实是同一个 benchmark 任务里的一条 prompt。我当时看着它就在想：一个工程师怎么可能写出这样的任务描述？我心说，不可能，这世上没人会这么写 prompt。但我还是想确认一下，于是我找来我们最好的三位工程师，把这条 prompt 摆到他们面前，问：你们会这么写 prompt 吗？答案是：不会。

[1:39] Ali Khial

And they're right. They shouldn't. And so at that point I'm I was like what is a what are benchmarks anyway? Uh I needed to take a step back. I needed to look more. I needed to understand. And so as I was researching, I faced a wall of keywords. Um graders, long horizon, verifiers, bench benchmarks, and and a lot of jargon. So, I was like, either this is too complicated or um there's a lot of jargon and a lot of um words to to to work through here. So, um I worked through it, worked with my team. I have a lot of good researchers in the team, and we uh kind of like nailed like simplified it to the most basics. Um and so, the way I see it is that it starts as a prompt or an instruction. That prompt is fed to models and agents. Agents provide solutions. Those solutions are verified uh and graded through verifiers and rubrics. All of that is wrapped in a harness that's that's preventing it from um from the external factors. And if it all goes good, uh we have um trajectories, scores, and um metadata that we can use um to to to verif- to basically uh rank um models. And so, the equation is simple. If prompts and instructions are great and verifiers and rubrics are doing their job while the harness is preventing um or creating an environment that is good for a benchmark, we should have amazing results.

而且他们是对的，他们本来也不该这么写。到这一步我就开始想：那 benchmark 到底是个什么东西？我得往回退一步，多看看、真正搞懂它。结果我一开始查资料，就撞上了一堵术语墙——grader（评分器）、long horizon（长程任务）、verifier（校验器）、benchmark，一大堆黑话。我当时的判断是：要么这事儿本身太复杂，要么就是行话和名词太多，得一层层扒开。于是我就跟团队一起扒——我们团队里有不少很不错的研究员——最后把它简化到了最基本的形态。在我看来，一切始于一条 prompt 或者说一份 spec（规格/指令）。这条 prompt 被喂给模型和 agent，agent 给出解法；这些解法再由 verifier 和 rubric（评分细则）去验证、去打分。所有这一切都被包在一个 harness（执行框架/沙箱）里，由它把外部干扰因素挡在外面。如果一切顺利，我们最后拿到的就是 trajectory（轨迹）、分数和一堆 metadata，用这些来给模型排名。所以这个等式其实很简单：只要 prompt 和 spec 写得好，verifier 和 rubric 各司其职，harness 又确实营造出了一个适合做 benchmark 的环境，那我们就该得到非常漂亮的结果。

[3:31] Ali Khial

Um but, that's not the reality. So, what what what went wrong? So, the first thing is when looking deeper in benchmarks, uh most of the instructions are unrealistic. Um I did a quick research on SweetBench Pro, and um there's 481 words per instruction in average. That's a two-pager per task. That is not how people write prompts. And to illustrate more of that, um I took a couple examples here. The first one I looked at I I call the leaky prompt. It's a go um task that's basically um that's trying to match in some rejects and doing test it's on on some rejects. So, in the first screenshot here, um the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that. The second one is is even worse. Um it's basically providing a complete interface of the implementation. Basically locking the LLM from any kind of uh creativity and it's forcing it to do it that way. So, that's the leaky prompt. The second example, it's the the not economically valuable prompt. Uh this is from Sweet Marathon. And this prompt is well-formed. It's It's It's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compiler in Rust. So, I don't know if any of you ever tried to do that, but I don't think it's a good idea. We should not do that.

但现实并不是这样。所以，到底哪里出了问题？第一件事是：往 benchmark 深处看，你会发现大多数 spec 都极不真实。我拿 SWE-bench Pro 快速统计了一下，平均每条 spec 有 481 个词——那是每个任务配一份两页纸的文档。真实世界里没人这么写 prompt。为了把这一点讲透，我挑了几个例子。第一个我管它叫「泄题式 prompt」（leaky prompt）。这是一个 Go 语言的任务，大意是要匹配一些正则表达式，并针对这些正则跑测试。看第一张截图：这条指令直接把测试文件的路径指了出来。这意味着什么？意味着 LLM 手里已经拿到了全部食材，它可以直接顺着这条线去把那个 test file 找出来，然后照着测试去写实现。第二张更离谱：它干脆把实现的完整 interface（接口定义）原封不动给出来了。等于把 LLM 的任何创造空间都锁死，逼着它只能按这一种方式实现。这就是泄题式 prompt。第二个例子，我叫它「没有经济价值的 prompt」，来自 SWE-Marathon。这条 prompt 本身写得很规范，抽象程度也刚好，给 LLM 留出了发挥空间——但它要求的是：用 Rust 写一个 C 编译器。我不知道你们有没有人真干过这事，反正我认为这不是个好主意，我们不该这么出题。

[5:16] Ali Khial

All right, moving on. The second problem, weak verifiers. Um so, the screenshot here is is a uh is the work that Deep Sweet um did uh to compare their uh their bench against Sweet Bench Pro. And um let me just fix here so I can see the numbers. In Sweet Bench Pro, 8.5 of 8.5% of all the tasks uh accepted wrong implementation in one hand and more than 20 24% of the tasks uh rejected um correct implementations. And so, I kind of went again, dug a little bit, and I extracted one of the tasks, and I started looking at it. Um and and here's here's what's happening in the example of uh re- rejecting um possibly rejecting good good answers. So, in this example, the test is is basically expecting a variable to exist. But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the variable name this way? So, this test is cornering the LLM and basically uh causing uh those false negatives. In the other example, it's base the test is basically checking functions that are unexported. So, if that was a PR in any of our projects, and exposed these type of tests, we would not accept it. So, this is what a weak verifier looks like. All right, moving on.

好，往下走。第二个问题：verifier 太弱。这张截图是 DeepSWE 团队做的工作，他们拿自家的 bench 去跟 SWE-bench Pro 做对比。我把数字调一下好让我看清楚——在 SWE-bench Pro 里，8.5% 的任务把错误的实现判成了通过，另一头，超过 24% 的任务把正确的实现判成了失败。于是我又往下挖了挖，捞出其中一个任务，一行行去看它到底怎么回事。先说「把好答案错判为失败」这一类。在这个例子里，测试的要求本质上是「必须存在某个变量」。问题在于：第一，这个变量在 spec 里压根没提；第二，凭什么指望 LLM 会恰好用这个名字去命名这个变量？所以这个 test case 等于是在给 LLM 下套，false negative（假阴性）就是这么来的。另一个例子是：那个测试去检查的居然是没有导出（unexported）的函数。换句话说，如果这是我们自己项目里的一个 PR，里面写着这种测试，我们是不会合的。这就是弱 verifier 长的样子。好，继续往下。

[7:00] Ali Khial

Re- reward hacking. So, what's happening is models are becoming increasingly increasingly able to optimize and figure out solutions to hard problems by going around the problem. So, instead of actually trying to fix the to to apply a patch to a task, they try to go and find dot git folders, or they look up the internet for any kind of traces that would allow them to um to do the task. And this first graph here shows like shows that as models evolve, they are now more smarter and smarter in being able to do reward hacking, but that's what we want. We want LLMs to be smart. The benchmarks are lacking behind and they're not preventing from from that to happen. Um More in detail, as you can see here, the more you go in time and the more you have new versions, the delta of um of um reward hacking is increasing. So, the conclusion here is there's a quality gap and it's causing a trust gap. I have not met an engineer in the last 6 months that would choose a model or choose um an LLM based on the leaderboards. Um they look at them. There's a lot of hype, but then they move on and they test things by themselves and they apply that. So, how do we close the gap? Um in the last 2 months, we've been working with our team at G2i to basically try to define a framework, uh a set of principles that would allow us to build tasks for benchmarks that are um better than what we have today.

第三个问题：reward hacking（刷奖励/钻分数漏洞）。现在的情况是，模型越来越擅长优化、越来越会「绕开问题本身」把难题解掉。也就是说，它不是老老实实去给任务打一个 patch（补丁），而是跑去翻 .git 目录，或者上网去搜有没有留下什么痕迹能帮它把任务糊弄过去。第一张图说明的就是：随着模型迭代，它们在 reward hacking 上越来越聪明。但这本来就是我们想要的啊——我们就是希望 LLM 变聪明。真正掉队的是 benchmark，它没能挡住这种事发生。再看细一点，这里能清楚看到：时间越往后、版本越新，reward hacking 的增量就越大。所以结论是：存在一条质量的 gap，而这条 gap 又制造出了一条信任的 gap。过去半年里，我没有遇到过任何一个工程师是照着排行榜去选模型的。他们会看排行榜，热度也确实很高，但看完就翻篇了，回头还是自己动手测、自己去验，然后照自己测出来的结果做决定。那我们怎么把这条 gap 补上？过去两个月，我在 G2i 和团队一直在做一件事：定义一套框架、一组原则，让我们能造出比今天这些更好的 benchmark 任务。

[8:54] Ali Khial

The first one, human instructions. Authored by humans, reviewed by humans. This is basically the entry point for any great tasks. The instructions given to an agent or an LLM should lean towards expressing desired behaviors, objectives, and hard constraints, not implement details or try to guarantee self-containment when the task itself is is expressing too much uh too much details. The second principle is holistic graders. Behavioral tests in one hand and then precision what were needed. This is very similar to how we approach um tests in engineering. We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed. So, for security issues or business logic, we want to have the whole stack units unit test integration tests and then end-to-end tests. But, for the rest of the the rest of the the software, we don't want to have 100% coverage because that's um not efficient. The third principle, production grade. The tasks have to be tasks have to have value um and they have to be economically valuable. Um it is one thing to have a test a task that is failing the LLM proven that the LLM is not there yet. It is another for it's another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that."

第一条：人写的 spec——由人撰写，由人评审。这是所有好任务的入口。给到 agent 或 LLM 的 spec，应该偏向于表达「期望的行为、目标、硬性约束」，而不是实现细节；也不要为了追求所谓的自包含，就把任务描述堆进过多细节。第二条原则：holistic grader（整体性评分器）——一头是行为层面的测试，另一头是在真正需要的地方做到精确。这跟我们做工程测试的思路非常像：我们希望覆盖面尽可能大，同时又不要写得太死板；但在该精确的地方必须精确。比如涉及安全问题或核心业务逻辑，那就全套都上——unit test、集成测试、再到端到端测试；但对代码库其余部分，我们并不追求 100% 覆盖率，因为那不划算。第三条原则：production grade（生产级）。任务必须有价值，而且必须是有经济价值的。一个能把 LLM 难住的任务，只能证明「LLM 还没到那个水平」，这是一回事；而让一个工程师看着这个任务说出「如果 LLM 能修好这个，那我就信得过它去修那个」——这是完全另一回事。

[10:33] Ali Khial

Currently, we don't have that. So, production grade. The fourth principle, contamination free by design. We want to do novel tasks only and we want to make sure that we keep private holdout sets. This is a principle that is very important as currently the tasks that are existing in benchmarks are all put from GitHub repos or from um from from public repos. So, our approach here is that it should always be novel. This way, it's contamination free by design. And the fifth and last principle here is information about leaderboards. The benchmark needs to tell a story and needs to help people make decisions. Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't to you why. And so, we want to basically put the x-axis back on um on on the first page. Uh the idea here is that there's um there's a lot of um data that we can extract from those these runs, and unfortunately, they're not being put in the forefront. And people have to dig uh a lot and do their own experiments to get to those data points. And so, finally, uh initially, I wanted to have a kind of a a lofty like ending to this, but I think I I I pivoted to something more interesting. Uh this is a call to action to software engineers.

而今天，我们并没有后面这种任务。所以：production grade。第四条原则：从设计上就杜绝 contamination（数据污染）。我们只做全新的任务，并且一定要留出私有的 holdout set（保留集，不公开）。这条原则特别重要，因为现在 benchmark 里的任务，基本全是从 GitHub 仓库、从各种公开仓库里扒出来的。我们的做法是：任务永远是新造的。这样一来，它在设计层面就天然免疫 contamination。第五条也是最后一条：给信息，而不是只给排行榜。benchmark 应该讲出一个完整的故事，应该真正帮人做决策。我们今天在 benchmark 里看到的就是排行榜——它告诉你谁赢了，却不告诉你为什么赢。所以我们想做的，是把「横轴」重新摆回首页。意思是：这些跑测过程里其实能提取出大量数据，但很遗憾，它们从来没被摆到台面上，人们必须自己去刨、自己做实验，才能拿到那些数据点。最后——我本来想用一个比较高远、比较升华的结尾，但后来我改主意了，换成了一个我觉得更有意思的：这是一封写给软件工程师的行动倡议。

[12:11] Ali Khial

Um benchmarks are not hard. We need to look under the hood. And we need to understand them and join the Discord because engineers' input is valuable. And thank you.

benchmark 并不难。我们需要掀开引擎盖往里看，需要真正把它搞懂，然后去加入 Discord——因为工程师的意见是有价值的。谢谢大家。

[12:28]

[applause] [music]

[applause] [music]