I built an AI code review bot in 30 minutes - here’s how

频道: How I AI
视频: https://www.youtube.com/watch?v=cmATJGbA8bI
原文语言: en
统计: 共 15 轮

[0:00]

Everybody is drowning in AI created PRs. When anyone can code, anyone can cook, and anyone can vibe anything to GitHub, that means that a lot of us are sitting on a big queue of pull requests that then we have to review. If you are an engineering leader or a software engineer in a company that has embraced AI in any way, shape, or form, you feel this pain. And I get this question all the time. We figured out how to code with AI. Now, what do we do with all these PRs? Well, people smarter than us have figured this out. And you know what the answer is? You don't have to review all your PRs. Yep, you heard it here first. You do not actually have to review one-on-one by a human all those PRs. So, in today's episode, I'm going to show you how to build your own PR review, risk-scoring, and automatic approval bot so you can take all the low-risk PRs off your plate, get those to production, and focus on the ones that really require your human expertise. This episode is inspired by a couple folks, especially our past How I AI guest Brian from Intercom, who has led the way in auto-scoring and auto-approval PRs. But I'm going to show you exactly how it gets done, and we're going to get it done very quickly in this 30-minute How I AI mini episode.

现在所有人都快被 AI 生成的 PR 淹没了。当谁都能写代码、谁都能上手、谁都能凭感觉 vibe 出一堆东西推到 GitHub，结果就是我们很多人手里堆着一长队等着评审的 pull request。如果你是工程负责人，或者你是一家多少已经拥抱 AI 的公司里的软件工程师，这份痛你肯定懂。这个问题我被问了无数遍：我们已经搞明白怎么跟 AI 一起写代码了，那这么多 PR 到底该怎么办？其实，比我们聪明的人早就把这事想通了。答案是什么？你根本不需要把所有 PR 都评审一遍。没错，你是在这儿第一次听到的：那些 PR，并不是每一个都得由人一对一地过一遍。所以今天这期，我要手把手教你搭一个属于自己的 PR 评审 + 风险打分 + 自动批准机器人，把所有低风险 PR 从你盘子里端走、直接送上生产环境，让你把精力留给真正需要人类专业判断的那些。这期的灵感来自几个人，特别是我们 How I AI 之前的嘉宾、来自 Intercom 的 Brian，他在 PR 自动打分和自动批准这件事上走在最前面。不过接下来我会把整个过程原原本本演示一遍，而且我们要在这期 30 分钟的 How I AI 迷你节目里飞快地做完。

[1:24]

Let's get to it. This episode is brought to you by [music] WorkOS. AI has already changed how we work. Tools are helping teams write better code, analyze customer data, and even handle support tickets automatically. [music] But there's a catch. These tools only work well when they have deep [music] access to company systems. Your co-pilot needs to see your entire code base. Your chatbot needs to search across internal docs. And for enterprise buyers, that raises serious [music] security concerns. That's why these apps face intense IT scrutiny from day one. To pass, they need secure authentication, access [music] controls, audit logs, the whole suite of enterprise features. Building all that from scratch, [music] it's a massive lift. That's where WorkOS comes in. WorkOS gets you drop-in [music] APIs for enterprise features, so your app can become enterprise ready and scale [music] upmarket faster. Think of it like Stripe for enterprise features. OpenAI, Perplexity, [music] and Cursor are already using WorkOS to move faster and meet demands. Join them and hundreds of other industry leaders at workos.com. Start building today. Okay, I know what you all are thinking. You're thinking, "Claire, you are crazy.

开工吧。本期节目由 WorkOS 赞助播出。AI 已经改变了我们的工作方式。各种工具正在帮团队写出更好的代码、分析客户数据，甚至自动处理支持工单。但这里有个前提：这些工具只有拿到公司系统的深度访问权限才能真正好用。你的 co-pilot 需要看到整个代码库，你的 chatbot 需要在内部文档里做检索。而对企业客户来说，这就带来了很严肃的安全问题。所以这类应用从上线第一天起就要面对 IT 部门的严格审查。想过关，它们需要安全的身份认证、访问控制、审计日志——整套企业级能力。这些东西全部从零自己造？工程量大到吓人。WorkOS 就是来解决这个的。WorkOS 给你一套开箱即用的企业级功能 API，让你的应用快速具备企业级能力、更快打进高端市场。可以把它理解成「企业功能界的 Stripe」。OpenAI、Perplexity 和 Cursor 都已经在用 WorkOS 提速、满足客户要求。上 workos.com，和另外几百家行业头部一起用起来，今天就可以开始搭。好，我知道你们在想什么。你们在想：Claire，你疯了吧。

[2:39]

Of course I have to review every PR that comes out of AI. That's just how it's done. We review all of our human PRs, so I have to review all of our AI PRs." And I am telling you, in 2026, the answer is you can safely not review PRs by putting AI in the loop. And I want to call attention to two blog posts that have really inspired me in this process. One is from Intercom about how they made AI-approved PRs safe. If you don't know, Intercom has 2x 3x their PR throughput. They are shipping a lot of code. That means they had a lot of code to review. And so, they built a PR review agent that scores their PRs and then auto-approves them. And what I love about what Intercom has done here is they don't just worry about speed and velocity and volume. Yes, they've multiplied the PRs across their R&D organization. Yes, they want to get those PR cycle time down and those reviews out quickly. But what they really think and what they've proven with their adoption of AI is that AI-approved PRs and AI-written code in general can be even safer and even higher quality than what you're doing with just a human in the loop. And so, they have used AI as an advantage when it comes to safety and quality as opposed to a disadvantage for safety and quality. And to set the vision of what you can expect here, I love that they shared some metrics. One is that AI approved PRs are approved faster, actually five times faster than their human PRs. So, one, you are getting some speed. Two, they are seeing higher quality, so the revert rate, the amount of code that has to be fixed after it hits production, is much lower when they write that code with AI. And then they figured out how to label and trace and audit all those changes and approvals so that they are in compliance with all of their required compliance frameworks, whether those are SOC 2, HIPAA, etc. So, I often get this objection from folks, they say, "Well, I'm in a SOC 2 environment, I couldn't

「AI 出的每一个 PR 我当然得审啊，一直都是这么干的。人写的 PR 我们全都审，那 AI 写的 PR 我当然也得全审。」而我要告诉你的是，到了 2026 年，答案是——只要把 AI 放进这个环节里，你完全可以安全地不去逐个评审 PR。这里我想特别提两篇给了我很大启发的博客。一篇来自 Intercom，讲他们是怎么让 AI 批准的 PR 变得安全的。你可能不知道，Intercom 的 PR 吞吐量翻了两三倍，他们在疯狂地发代码，也就意味着要评审的代码量非常大。于是他们做了一个 PR 评审 agent，给 PR 打分，然后自动批准。我特别喜欢 Intercom 这套做法的一点是：他们关心的不只是速度、效率和吞吐量。没错，他们整个研发组织的 PR 数量翻了好几倍；没错，他们希望把 PR 的周期时间压下来、让评审快点走完。但他们真正相信、并且已经用 AI 落地验证了的，是这件事——AI 批准的 PR，以及 AI 写出来的代码，整体上可以比只有人类在环时更安全、质量更高。所以他们是把 AI 当成安全与质量上的优势来用的，而不是当成隐患。为了让你对能达到什么效果有个概念，他们分享的几个数字我很喜欢。第一，AI 批准的 PR 通过得更快——比人类评审的 PR 快 5 倍。所以第一点，你确实拿到了速度。第二，他们看到质量更高：回滚率，也就是上生产之后还得返工修补的代码比例，在用 AI 写代码之后反而低得多。第三，他们还搞定了怎么给所有这些变更和批准打标签、留痕、可审计，这样就能符合他们必须遵守的各类合规框架，不管是 SOC 2 还是 HIPAA。所以我经常听到这样一句反驳：「可我们是 SOC 2 环境，不可能

[4:56]

possibly auto approve PRs." But in fact, there are frameworks, as long as it's in your risk policies, as long as it's in your code review policies, as long as it's auditable, queryable, and defensible, you can do this within your compliance framework. I'll have to work with your security and compliance teams to figure out how this can work for you. So, I really love just this whole concept. I also took some inspiration, who took inspiration from Intercom on this, Rewind Bot, which is a little bit more of the technical way that they review PRs. And so, this blog post about Diff Vader, which I love, shows a couple components of how they've scored risk, whether it's blast radius, how correct it is, if all the actions have run, etc. And so, between these two blog posts, I had a really good starting point for building an agent. I knew that I wanted to do this for chat PRT. I knew we had a lot of low-risk PRs that were just sitting in the queue because my colleague and I were not getting to review them. It was kind of boring. Especially a lot of like Devon authored PRs. And so this was going to be a really high-impact thing that I knew I could build and also let me work with a new framework that I'm excited to demo for you all here on How I AI. So, we're going to go to this Code X vibe coded deck and we're going to walk through how I built a code review risk scoring bot with Vercel's Eve. So, if you have not heard about Eve, I'm a little bit obsessed with Eve. Why do I like Eve? Well, Eve has become the simplest way for me to deploy AI agents in the enterprise in Slack and GitHub without the torture and pain of something like Hermes agent or my beloved OpenClaw. So, when I'm working and helping other companies sort of like think about how to deploy agents, if they're using Vercel, I tell them to use Eve. Why do I tell them to use Eve?

自动批准 PR。」但事实上，是有成熟框架的——只要它写进了你的风险政策、写进了你的代码评审政策，只要整个过程可审计、可查询、说得清来龙去脉，你完全可以在合规框架内做这件事。具体怎么落地，你得和自己公司的安全与合规团队一起去敲定。总之我非常喜欢这整个思路。另一个给我启发的来源，是同样受 Intercom 启发的 Rewind Bot，他们那篇更偏技术实现，讲他们具体怎么评审 PR。那篇讲 Diff Vader 的博客——这名字我太爱了——展示了他们给风险打分的几个维度：影响半径（blast radius）、代码正确性、所有 action 有没有跑完，等等。有这两篇博客打底，我搭 agent 的起点就非常扎实了。我知道我想给 ChatPRD 做这个东西。我知道我们有一大堆低风险 PR 就那么堆在队列里，因为我和同事一直腾不出手去审——说实话审起来还挺无聊的，尤其是一堆 Devin 写的 PR。所以这件事我很确定是高价值的、我也能把它做出来，顺便还能让我用上一个新框架，正好在 How I AI 上给大家演示一下，我挺兴奋的。那我们就打开这份用 Codex vibe coding 出来的演示文稿，一起过一遍我是怎么用 Vercel 的 Eve 搭出这个代码评审风险打分机器人的。如果你还没听说过 Eve——我现在有点沉迷 Eve。为什么喜欢它？因为 Eve 已经成了我在企业环境里把 AI agent 部署到 Slack 和 GitHub 上最省事的方式，不用再经历像 Hermes agent、或者我很爱但真的很折磨人的 OpenClaw 那种痛苦。所以我在帮别的公司想「agent 该怎么部署」的时候，如果他们本来就在用 Vercel，我就直接让他们用 Eve。为什么我推荐 Eve？

[6:59]

Well, what's awesome about Eve is a lot of it is just a directory. An Eve agent is very similar to OpenClaw, just a directory of instructions, skills, code, and it can talk to you in channels that are out of the box in Vercel. So again, it's like very easy to set up the agent. It's very easy to give it skills, which we all use. It's very easy to give it tools. It can even have a sandbox in which it can execute things including your code if you're doing a code review. And then my favorite part is managing talking to the agent in specific channels and connecting it to enterprise data is super easy. So, because Vercel has shipped a Vercel connectors, which are like managed connections inside your Vercel account, you can just literally go through like a little wizard to hook up Slack to your Eve agent. Little wizard to hook up GitHub to go through your agent. And it's all set up. The refresh tokens, all that stuff. Everything that's painful about creating an agent is just taken care of. And I sound totally like a Vercel shill. Um I've just tried a bunch of stuff, and this has been the easiest framework to get up and running. It uses the chat SDK behind the scenes. And if you are creating any sort of chatbot in Slack, Teams, WhatsApp, whatever, I mean, bless. I have I was asking for this chat SDK for a long time. It got built. This is an open-source framework that I think is so useful, whether or not you use Eve or not, to stand up a chatbot. It just like takes away all the complexities of managing like multi-channel agents. It handles all like the Slack configuration. It gives you a manifest for Slack. Like I love it that Chat Perry's Slack bot and Teams bot actually runs on the chat SDK, but it's even better when it's bundled into this Eve framework. And so, I highly recommend you try Eve framework. Again, like they didn't pay me to say this. I just really like it, and I found it super easy to use. And

Eve 最爽的一点是，它本质上就是个目录。一个 Eve agent 跟 OpenClaw 很像，就是一个装着指令、skills、代码的目录，而且它能在 Vercel 里开箱即用的那些渠道跟你对话。所以 agent 搭起来非常容易，给它加 skills 非常容易，给它加工具也非常容易。它甚至能有一个 sandbox，在里面执行东西，包括你要做代码评审时执行你的代码。而我最喜欢的部分是：管理 agent 在哪些具体频道里对话、以及把它接到企业数据上，都超级简单。因为 Vercel 上线了 Vercel connectors——就是你 Vercel 账号里的托管连接——你真的只要点一个小向导，就能把 Slack 接到你的 Eve agent 上；再点一个小向导，把 GitHub 接进来。全部配好了，refresh token 之类的通通不用管。做 agent 时所有让人头疼的部分，它都替你处理掉了。我这话说得完全像 Vercel 的托儿，但我是真的试过一大圈，这是我用过上手最快的框架。它底层用的是 Chat SDK。如果你要在 Slack、Teams、WhatsApp 或者别的什么地方做 chatbot——真的，谢天谢地。这个 Chat SDK 我求了好久，终于有人做出来了。它是个开源框架，我觉得非常有用，不管你用不用 Eve，拿它来起一个 chatbot 都很值。它把管理多渠道 agent 的复杂度全都抹平了，Slack 的配置它全包了，还直接给你生成 Slack 的 manifest。我特别喜欢的一点是，ChatPRD 的 Slack bot 和 Teams bot 其实就跑在 Chat SDK 上，而当它被打包进 Eve 框架之后就更好用了。所以我强烈建议你去试试 Eve 框架。再说一次，他们没给我钱让我夸，我就是真心喜欢，用起来太省事了。

[9:06]

now every agent that I'm creating that I want to interact with in Slack for an internal use case, if I don't already have an open claw, I'm porting over to Eve. So, how did I actually build this? What does the agent do? Well, I built it in Codex. And so, I'm going to show you the amazing prompt where I started this PR review bot, which it says, "I want to make an internal GitHub bot/app that reviews PRs after all CGXs, that's checks, are green and grades low, medium, high risk and approves the low risk Oh my god, you guys I can't type. All All of the type of this. Approves the low risk PRs automatically. That's all I did and then I interrupted it. I steered it and said, we could design it as a Vercel Eve agent if you want and it said, sure that seems like a good good idea and it was off to the races. And so instead of going through kind of like the chat on Codex on how I built this is took a couple turns but really not that many. I would say a couple things that were like magical as I ran Codex is not that it did the code for the Eve agent in the repo. That was fine. That was interesting and I'll tell you a little bit about how the code actually works in a sec. But what was actually awesome is if you've ever set up a Slack bot or a GitHub app, you have to like click through a bunch of configuration screens. You have to pick permissions. You have to do all this stuff. And so what I did is my favorite hack is I had it use Chrome browser use and I let it navigate Slack bot setup. I let it navigate um GitHub setup and I just like clicked buttons and two-factor and just double-checked that it was right. And so I said, you know, is it configured yet?

现在只要是我新做的、要在 Slack 里交互的内部 agent，如果不是已经跑在 OpenClaw 上的，我一律迁到 Eve。那我到底是怎么把它搭出来的？这个 agent 到底做什么？我是在 Codex 里做的。我给你们看看我启动这个 PR 评审机器人时那句「神级」提示词，原话是：我想做一个内部 GitHub bot / app，在所有 CI（也就是各项 checks）都变绿之后去评审 PR，把风险分成低、中、高三档，然后自动批准低风险的那些。天呐我打字真是……满屏错别字。自动批准低风险 PR。我就写了这么一句，然后中途打断了它，给它拨了一下方向：如果你愿意，我们可以把它设计成一个 Vercel Eve agent。它说：好啊，这主意不错。然后就一路狂奔了。与其带你们看 Codex 里那些对话记录——其实来回了几轮，但真的没几轮——我想说几个跑 Codex 时特别神奇的点。神奇的不是它在 repo 里把 Eve agent 的代码写出来了，那部分还行、也挺有意思，代码到底怎么跑我等下会讲一点。真正让我惊艳的是：如果你配过 Slack bot 或者 GitHub app 就知道，你得点开一堆配置页面，勾各种权限，一通折腾。而我用了我最爱的一个偏方——我让它用 Chrome 的 browser use，让它自己去走 Slack bot 的配置流程，自己去走 GitHub 的配置流程，我只负责点几下按钮、过一下双因子认证，顺便核对它做得对不对。我就问它：配好了吗？

[10:51]

It said, no, it needs to configure all this stuff in the permissions on the GitHub bot. I was like, great, you're logged into Chrome and then I was like, I can press save. It was having a hard time press save. And so if you're just sort of like meta tip, if you ever have to do like complex SaaS configuration, set up an app, you're like, uh, I'm fine writing code but I don't want to do the configuration in somebody's third-party service, using Chrome browser use is such a useful hack to get that stuff done. So again, I just built this in Codex. I actually didn't give it any instructions on scoring. I didn't give it any instructions on configuration. I didn't give it any instructions on risk. I just said kind of like one-ish shot it, few shot this into production. And so, I want to talk a little bit about what the agent does and how it works. So, it's pretty simple. If you were building PR review agent, this is what I would success suggest you do, which is it reads the PR, it looks at the exact diff, um and it scores the risk and it publishes the evidence to the risk. If it's low risk, if it's assessed that it's a low risk PR, it submits an approval on the PR. If it needs a human, it escalates to human and if there's a blocking um requirement, then it stops and it it will require a a request change. So, it's like very similar to how a human would do code review. How does it work from a technical flow perspective? So, the GitHub I have a GitHub app. It keys off an event after all of the PR changes are done. Vercel's GitHub integration picks that up in the GitHub Vercel channel and it sends it a little bit of information.

它说还没有，GitHub bot 的权限那块还有一堆要配。我说太好了，你已经登录 Chrome 了。然后我说保存我来点吧——它点保存那一步有点费劲。所以给你一个通用的小心得：以后你要做那种复杂的 SaaS 配置、要在别人家的第三方服务里把一个应用配起来，心想「写代码我没问题，但我真不想去点配置」——用 Chrome 的 browser use 简直是个超好用的偏方，能把这类活儿干完。所以再强调一遍，我就是在 Codex 里把它搭出来的。我其实没给它任何关于打分的指示，没给任何关于配置的指示，也没给任何关于风险的指示。就相当于一次成型、顶多几次成型，直接推上了生产。接下来我想讲讲这个 agent 具体做什么、怎么运作。其实很简单。如果你要做一个 PR 评审 agent，我建议你就这么做：它读取 PR，看具体的 diff，给风险打分，并把打分依据一并写出来。如果判定是低风险 PR，它就直接在这个 PR 上提交批准。如果需要人来看，它就升级交给人；如果碰到有阻断性的问题，它就停下来，提出 request changes。所以逻辑跟人做 code review 非常像。从技术流程上怎么跑？我有一个 GitHub app，它监听的是 PR 所有变更完成之后的那个事件。Vercel 的 GitHub 集成在 GitHub-Vercel 这个通道里接到这个事件，然后把一小段信息传给 agent。

[12:40]

Vercel spins up its sandbox, it checks out the repo, it runs it and looks at the diff, it runs a couple skills and tools to assess the risk and quality of the PR, and then on the end it outputs like a thumbs up check uh like it needs changes and a comment. one fun thing that it does is after it does that review, it pings my colleague and I in Slack to say this PR is ready for review or it needs some help. So, if you were to set up your Eve agent to be similar to mine, what would it have? Well, it would have a couple things. It needs to have the GitHub um channel, it needs to have instructions, which I can show you a little bit of the instructions. It needs a skill to review the PR, it needs two tools. Read like all the files and information around the PR and then how to actually put a risk decision in and then it has, you know, a couple things like a Slack notifier. So, this is not that many files. It's pretty simple. I will show you our instructions just to make this like very clear how easy it is to set this up. It's a chat parity engineering agent. It reviews PRs. It calls the risk context. It scores it and then it has a couple of instructions. This is it. There's no scrolling. It is like four four or five paragraphs, a couple bullet points, and it's ready to go. You do not need to over-engineer this. And it works really, really well. So, this is again why I like this Eve agent framework because you're just working in natural text or in TypeScript, and it's super easy to implement.

Vercel 拉起它的 sandbox，把 repo checkout 下来，跑起来看 diff，再跑几个 skills 和工具去评估这个 PR 的风险和质量，最后输出一个结论——点赞通过、或者需要修改，再加一条评论。还有个挺好玩的设计：评审做完之后，它会在 Slack 里 @ 我和我同事，说这个 PR 可以看了，或者这个 PR 需要人帮忙。那如果你想照着我这套配一个自己的 Eve agent，它需要哪些东西？其实没几样。它需要 GitHub 那个 channel；需要 instructions，一会儿我给你看看我写的指令；需要一个用来评审 PR 的 skill；需要两个工具——一个用来读取 PR 相关的所有文件和信息，另一个用来把风险判定结果写回去；然后还有几个零碎的，比如一个 Slack 通知器。所以文件真没多少，非常简单。我把我们的 instructions 给你们看一下，好让你们清楚地感受到搭这东西有多容易。内容就是：这是 ChatPRD 的工程 agent，它评审 PR，它调用风险上下文，它打分，然后是几条指令。就这些。连滚动条都不用拖，大概四五段话、几个要点，就能跑了。你完全不需要过度设计，而且它跑得非常非常好。所以我才这么喜欢 Eve 这个 agent 框架——你只需要写自然语言，或者写点 TypeScript，实现起来超级简单。

[14:17]

Setting up the GitHub app, you can ignore this. I mean, I guess like point your agents to this instructions. I did not do any of this. Again, Chrome browser use did this on our behalf. And so, it just has to have access to the pull request, the contents of your files, um CI checks and action checks, and then some metadata. And I just had Vercel set all that up. It has a couple rules on when it's triggered. So, you don't want it triggered on every PR before checks are done. So, I kind of had to like reduce some noise in the PR rules. And then there's just two basically steps to this. It reads the PR. It's like a skill to read the PR. I will show it to you. It's not that thrilling, but again, just showing you how simple it is. It reads and reviews the PR. It has a couple repo specific uh risk kind of category. So, like docs are low risk, feature log it logic is medium risk, and then like authentication billing's high risk. Diff size does not set risk, and then it gives some like instructions on writing. Again, this is it. It's like not that much. It's maybe one page of text. I did not write any of this. I refined it. And so, what I'm trying to reinforce to you is like writing these agents is not that hard. It is literally just writing instructions and skills. That is what it takes to create an agent like this, and that's why I like this e framework cuz you can do a lot of it in markdown.

至于 GitHub app 的配置，这部分你可以直接跳过——非要说的话，就是把这些说明丢给你的 agent 让它去做。这些我一点都没自己动手，还是那句话，是 Chrome 的 browser use 替我们干的。它需要的权限也就是：能访问 pull request、能访问你的文件内容、能看 CI 检查和 action 检查，再加一点元数据。这些我全交给 Vercel 配好了。它还有几条触发规则。你肯定不想每来一个 PR、在 checks 还没跑完的时候就触发它，所以我在 PR 规则那儿做了一些降噪。然后整件事基本上就两步。第一步读 PR——就是一个读 PR 的 skill，我给你们看一眼，其实没什么刺激的，但正好说明它有多简单。它读 PR、评审 PR。里面有几条针对这个 repo 的风险分类，比如文档改动算低风险，功能逻辑算中风险，认证和计费相关算高风险。diff 的大小不作为风险依据。再往下就是一些关于怎么写评审意见的指令。真的，就这些，加起来大概就一页文字。这些内容我一个字都没写，我只是做了打磨。所以我想反复强调的是：写这类 agent 一点都不难，本质上就是写指令和写 skills。造一个这样的 agent，需要的就是这些。这也是我喜欢 Eve 框架的原因——很多东西你直接用 markdown 就能搞定。

[15:47]

And then once it reads the code, it scores the risk. And so, it looks at six things: how big is the change surface and blast radius, is it easily reversible, right? Like a huge data migration is maybe harder to reverse. Does it touch data security and does it cover data security? Does it change anything about our operations? And verification gap is like, are our tests complete? Did CI complete? Can we actually test a couple ways that this is true? And then it has sort of like a script that it runs to calculate a score. And then anything under 24 points, I did again, I did not like choose these thresholds. Anything below 24 points is low risk, medium risk is 25 to 64, and then 65 and above is high risk. Medium and high risk PRs require human approval. Um so, it's pretty simple. There's some logic in here to only review the latest changes, which I like. And then the final piece is it escalates us to us in Slack for final action. So, we have repo rules probably very similar to all of you that requires a review in our GitHub to like match all of our our SOC 2 compliance requirements. It just makes auditing and um management of our compliance really easy. I went through a couple ways where the bot could like {quote} act like a human to do the review. It just didn't seem worth it. So, what it does is escalate to Slack, and then it says this one's been automatically reviewed. You can punch the approve button and merge and you're good good to go. And so we sort of like built in an operational process here as well. But, let's see what this actually looks like in action. So, I have three PRs for you um just to show you kind of how this works. Now, the the PR bot called Merge Mommy, she's Merge Mommy the PR review bot. Um you know, when you make internal bots, we don't have Diff Vader, we have Merge Mommy cuz we're fun at Chat PRD.

读完代码之后，它就开始给风险打分。它一共看六件事：改动面和爆炸半径有多大；是不是容易回滚——比如一次大规模的数据迁移，撤回起来可能就麻烦得多；有没有碰到数据安全、有没有把数据安全覆盖到；有没有改动我们的运维相关的东西；还有一项叫「验证缺口」，就是测试是不是完整、CI 有没有跑完、我们能不能用几种方式真正验证这事儿是对的。然后它有一段脚本，跑一下算出总分。低于 24 分的——我再强调一次，这些阈值不是我定的——24 分以下算低风险，25 到 64 分是中风险，65 分及以上是高风险。中风险和高风险的 PR 必须人工批准。所以逻辑挺简单的。里面还有一小段逻辑，只审查最新的改动，这点我挺喜欢。最后一块，是把事情升级到 Slack 交给我们做最终动作。我们的 repo 规则大概跟你们的差不多，要求 GitHub 里必须有一次 review，这样才能对上我们所有的 SOC 2 合规要求，审计和合规管理起来也特别省事。我试过好几种让 bot「装成人类」去做 review 的路子，感觉都不太划算。所以它现在做的就是升级到 Slack，说一句：这条已经自动审过了，你按一下 approve 再 merge 就好了。等于我们顺手也把一套运作流程内建进去了。不过咱们还是看看它实际跑起来是什么样。我给大家准备了三个 PR，演示一下这套东西是怎么工作的。这个 PR bot 叫 Merge Mommy，她就是 Merge Mommy，PR 审查机器人。你懂的，自己做内部 bot 的时候就是这样——我们没有 Diff Vader，我们有 Merge Mommy，因为我们 ChatPRD 就是这么会玩。

[17:47]

And this was a docs-only update automatically done by Devin. And typically, I just like smash the approval button on these, merge them, but I found that like we were just getting queued up of a lot of docs changes that didn't really need a bunch of review. And so, what happens on um once the PR is ready to go, Merge Mommy runs, it gives a risk score six out of 10, it's very low risk because it only changes docs, but it's not auto-approved because it has merge conflicts. So, that's one of the scoring issues it has to check. And then there are some detailed notes here about why it's blocked and the approval blocker is the pull request has merge conflicts. A cleaner example here is the stock which I already merged, which again went through and said it's seven out of 10 low risk, it's auto-approved, and you can see up here Merge Mommy gave a little check that said it's approved. Now, what we do with these approved PRs, as I said, is we send them to a Slack channel. And so, the Slack channel says, "Hey, either of you can take this, approve it. The risk is low, the checks are green. You just need to smash that approve button and merge it." So, it's just like a really nice automated flow.

这条是 Devin 自动提的、纯文档更新的 PR。这种我一般就是啪一下点批准、直接 merge，但我发现文档类的改动会越攒越多，排一大队，其实根本不需要花那么多力气 review。所以现在，PR 一准备好，Merge Mommy 就跑起来，给了 6 分（满分 10 分），风险非常低，因为只动了文档——但它没有自动批准，因为这条有 merge 冲突。这也是它评分时必须检查的一项。下面还有一段详细说明，讲清楚为什么被卡住，批准的阻塞原因就是这个 pull request 有 merge 冲突。再看一个更干净的例子，这条我已经 merge 掉了：它跑完说 7 分（满分 10 分），低风险，自动批准，你能看到上面 Merge Mommy 打了个小勾，表示已批准。那这些被批准的 PR 我们怎么处理呢？刚说了，我们把它们推到一个 Slack 频道里。频道里的消息大意就是：嘿，你们俩谁来都行，批一下。风险低，检查全绿。你只要啪一下点 approve 然后 merge 就完事了。所以整个流程自动化得非常舒服。

[19:07]

Now, why do we have to have a human do it? As you can see, this check mark is gray. A bot cannot actually meet our repository rules for required approval, and so we can either skip those rules, hack around it. We just decided this check would be a signal that our humans can approve it without really looking at it and then merge it as we want. And then let's show one where Merge Mommy did not approve it. This is a deprecation PR, so we had chat V1 and chat PRD and we're moving to chat V2. We kind of like refactored a bunch of code, but we left the old stuff behind a feature flag, and it was like ready to delete. So there's like 35 changes, a big red diff that we wanted to delete, and you can see here Merge Mommy like I'm sorry, I can't take myself seriously when I say it, but I'm going to say it anyways. Merge Mommy gave it a 45 out of 100, a medium risk, and it was not approved by a human because both it had some code issues as well as the policy said it changed server API behavior. It was a large change service, and so it's medium, not low, and it can't approve the PR. Now, we can run this against every PR that goes through our process, and again, we're going to just keep getting these notifications in Slack that just let us know, "Hey, this can be approved or this requires human review."

那为什么还非得让人来点这一下呢？你看这个勾是灰色的。bot 其实满足不了我们 repo 里「必须有人批准」的规则，所以我们要么跳过这些规则，要么想办法绕过去 hack 一下。我们最后的决定是：把这个灰勾当成一个信号——我们的人可以不用真去细看就批准，然后想 merge 的时候直接 merge。接下来看一个 Merge Mommy 没批准的例子。这是个废弃清理的 PR：ChatPRD 里原来有 Chat V1，我们要迁到 Chat V2，代码重构了一大堆，但旧的东西留在了 feature flag 后面，现在可以删了。所以这条有 35 处改动，一大片红色的 diff 等着被删掉。你看这里，Merge Mommy——不好意思，我每次说这名字都绷不住，但我还是得说——Merge Mommy 给了 45 分（满分 100 分），中风险，没有自动放行，必须由人来批，因为它一方面发现了一些代码问题，另一方面按策略判定这次改动动了服务端 API 的行为，改动面又大，所以是中风险而不是低风险，它就不能批准这个 PR。这套东西可以挂到我们流程里的每一个 PR 上，然后 Slack 里就会源源不断地收到通知，告诉我们：嘿，这条可以批，或者，这条需要人来审。

[20:36]

Again, I like to say this thing where we can put AI to work for us or we can have AI put us to work, and this is an example where you do both. I put AI to work, this lovely Eve agent, I put AI to work to review and score the PR and tell me if there's any risk and like be detailed eyes on it, and then I have AI put me to work by escalating into Slack for the final action and making it like a two-click process to get this thing to production. Now, I just have to say this is something that I was really intimidated to build. I thought this was going to take me like days and days and days. I was like, "I don't want to set up the GitHub app." I actually had tried this before Codex like browser use was really good and before the Eve agents had come out and it was just like not easy to do. But now that we have these open-source frameworks, now that we have these agent-building frameworks, now that kind of like everybody's agreed that we need Slack bots that do AI, a lot of these problems have been solved and then Codex browser use was just like the cherry on top that made building this agent super super super simple. So, to just recap for you all in this lightning-fast mini episode about how to build a PR risk score and review bot, use Vercel's Eve agents, give it instructions and a skill to read a PR and calculate a score based on a couple components, hook up that Vercel agent to GitHub and to Slack, let it read from GitHub and give you a little gray check for approval and then have it Slack you in Slack for that final human review and merge, and your cycle time will be super super fast and your PR velocity will go through the roof. Now, I want to hear from you all. Is this crazy? Can you pull this off at your company and what would you put into your risk scoring that I haven't thought of?

我老爱说这么一句话：我们可以让 AI 替我们干活，也可能反过来被 AI 支使着干活——而这个例子是两头都占。我让 AI 干活：这个可爱的 Eve agent 帮我 review、给 PR 打分、告诉我有没有风险，替我当那双盯细节的眼睛；然后我又让 AI 支使我干活：它把事情升级到 Slack，让我来做最后一步，把上线变成一个点两下就完成的动作。我必须说，这东西本来是我特别怵、特别不敢动手做的。我以为得花上好几天好几天。我当时想的是：我可不想去配那个 GitHub app。其实我以前试过一次，那是在 Codex 的 browser use 变得这么好用之前，也在 Eve agents 出来之前，当时做起来就是不容易。但现在有了这些开源框架，有了这些搭 agent 的框架，加上大家现在基本都认了「我们需要会用 AI 的 Slack bot」这件事，很多问题已经被解决掉了；Codex 的 browser use 又是锦上添花的那一笔，让搭这个 agent 变得超级超级超级简单。所以给大家快速回顾一下这期闪电版迷你节目，怎么做一个 PR 风险打分加审查的 bot：用 Vercel 的 Eve agents，给它一套指令和一个 skill，让它读 PR、按几个维度算出分数；把这个 Vercel agent 接到 GitHub 和 Slack 上；让它从 GitHub 读取信息，给你打一个灰色的批准勾，然后在 Slack 里 @ 你，由人完成最后的 review 和 merge。这样你的周期时间会快得飞起，PR 的吞吐速度也会一路冲上天。现在我想听听你们的想法。这事儿是不是疯了？在你们公司能落地吗？还有哪些维度是我没想到、你会加进风险打分里的？

[22:49]

The final bonus piece off this, which I didn't show, but a lot of great folks do, especially the the folks at Intercom do as part of this process, is they run evals on this internal agent. So, every time this review is run, it gets logged into I think it's like an internal eval platform, and an engineer looks at it and says, "Did the agent get this right? Did the agent get this wrong? Are we happy with the scoring mechanism?" So, very similar to how you'd use evals to improve your customer-facing AI products, you're going to want to use evals to improve your internally-facing AI bots, especially ones that touch really critical things like code. That has been my mini episode on how to build a PR approval and review bot here at How AI. I cannot wait to hear if you think this is crazy or this is the way of the future. Thank you so much for joining, and we'll see you soon. Thanks so much for watching. If you enjoyed the show, please like and subscribe here on YouTube, or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiaipod.com.

最后还有一个加分项，我今天没演示，但很多厉害的团队都在做，尤其是 Intercom 的人，他们把这一步也放进了流程里：他们给这个内部 agent 跑 evals。每次这个 review 跑完，结果都会被记录进——我记得是一个内部的 eval 平台，然后由工程师去看：这次 agent 判对了吗？判错了吗？我们对这套打分机制满意吗？所以，跟你用 evals 去改进面向客户的 AI 产品是一个道理，你也会想用 evals 来改进面向内部的 AI bot，尤其是那些碰到代码这类要害东西的。以上就是我在 How I AI 做的这期迷你节目，讲怎么搭一个 PR 审批和审查 bot。我特别想知道你们觉得这是疯了，还是未来的方向。非常感谢收看，我们很快再见。谢谢大家的观看。如果你喜欢这个节目，请在 YouTube 上点赞订阅，或者更好——留言告诉我们你的想法。你也可以在 Apple Podcasts、Spotify 或者你常用的播客 App 上找到这档播客。也请考虑给我们打个分、写个评价，这能帮更多人发现这个节目。所有往期节目和关于节目的更多信息，都在 howiaipod.com。

[24:10]

See you next time.

下期见。