Anthropic's CCA Exam as a Field-Guide for Agentic Engineering — Frank Coyle, UC Berkeley

频道: AI Engineer
视频: https://www.youtube.com/watch?v=Z-c11pV_uvU
原文语言: en
统计: 共 15 轮

[0:01]

[music]

[音乐]

[0:12]

Okay, I'm getting rolling and uh welcome aboard. We just had a little technical issues, but uh we resolved them. So, my name is Frank Coyle. Uh I am a computer science guy. I've been teaching computer science for over 30 years, and I'm now teaching at Berkeley. And one of the problems that uh all my students, past and present, are having is AI, because computer science is no longer the magic pathway to a job. So, I've been trying to figure out ways to uh help them come up with schemes to help them get ready for this world of agentic AI. And one of the things that sort of uh dropped into my uh plate was the something called the Claude Certified Architect exam, which I will be talking about today, and it has um a number of aspects to it. And I think if you're interested in a career in agentic AI, then certainly take a look at least what the exam is about, because I feel that um Anthropic knows how people are using their system and what the issues are going to be. So, before we jump into that, I want to give a little bit of my uh my philosophy. bop bop bop bop May have to do this manually, getting stuck. So, this is a quote from uh a woman named Sister Corita Kent. Nothing is a mistake. There's no win and no fail. There's only make.

好，我这就开始，欢迎各位。刚才出了点技术故障，不过已经解决了。我叫 Frank Coyle，是搞计算机科学的，教了三十多年计算机，现在在 UC Berkeley 教书。我的学生——不管是以前的还是现在的——都撞上了同一个问题：AI。因为计算机科学已经不再是通往一份工作的那条魔法通道了。所以我一直在琢磨，怎么帮他们找到一些路子，为这个 agentic AI 的世界做好准备。恰好有件事落到了我手上，就是所谓的 Claude Certified Architect（CCA）考试，今天我要讲的就是它。这个考试涉及好几个方面。我觉得如果你想在 agentic AI 这个方向上做职业发展，至少该去看看这个考试考的是什么，因为我认为 Anthropic 最清楚大家是怎么用他们这套系统的、以及接下来会遇到哪些问题。在进入正题之前，我想先讲一点我自己的理念。（哔哔哔哔）可能得手动翻页，卡住了。这是一位叫 Corita Kent 修女说过的话：没有什么是错误，没有赢也没有输，只有去做。

[1:48]

Bottom line here is experiment, experiment, experiment. Not only should you read, but you should do. You should make stuff. Now, what happens when you make stuff? A lot of times things don't work. Thomas Edison said, "I have not failed. I've only found 10,000 ways that don't work." And what I want to emphasize here is that what this shows us are something that in the design patterns movement, which came around in the early 1990s with object-oriented programming, we had patterns for objects. We now have patterns for agents, but there's also anti-patterns. And I think anti-patterns are a key to understanding what you should not do because understanding what you should not do is the key to leading you to what you should do. So, a little bit about the Claude Certified Exam, released in March, so it's brand new. It is uh it is based on scenarios. It is timed. It is proctored. It is available to companies in the Claude ecosystem, the Anthropic ecosystem, but individuals can pay $99 and take the exam once every once every 6 months. And it's not just multiple-choice questions. It is multiple-choice, but they're they are based on uh realistic constraints and realistic scenarios. The five domains.

归根到底就一句话：动手实验，实验，再实验。你不能只是读，你得做，得做出东西来。那做东西的时候会发生什么？很多时候东西根本跑不通。Thomas Edison 说过：我没有失败，我只是找到了一万种行不通的办法。我想强调的是，这里体现出来的，正是 design patterns（设计模式）运动里的那套东西——设计模式运动出现在 1990 年代初，伴随着面向对象编程，当时我们有面向对象的模式。现在我们有了 agent 的模式，但同时也有 anti-pattern（反模式）。我认为反模式是理解「什么不该做」的关键，而搞清楚什么不该做，恰恰是通向「该怎么做」的钥匙。再说说 Claude Certified 这个考试：三月份才发布，非常新。它是基于场景的，有时间限制，有监考。Claude 生态、也就是 Anthropic 生态里的公司可以考，个人也可以花 99 美元考，每六个月能考一次。而且它不只是选择题——形式上确实是选择题，但题目都建立在真实的约束条件和真实的场景之上。五个领域。

[3:26]

There are five domains that are covered and they give you the percentages of each. So, agentic architecture, 27%. Claude code, how to configure the Claude code system and workflow, 20%. How to doing prompt engineering, structuring your output, using JSON all over the place. Tool design. Model context protocol integration. These are topics that you should understand and know whether you're going to take the exam or not. This is going to help you get ready for whatever the agentic world is going to throw at you. And then there's going to be contact management and reliability. So these are the areas of of the kind of questions you're going to run into. Then there are and they they provide you with six production scenarios and your the exam will randomly choose four and all the questions will be centered around the four that they choose. And what I'm going to do is walk you through um the production scenarios and give you some anti-patterns to be aware of because there's a number of ways you can solve the problem but one of the big things is what not to do and that often can be the key to getting these questions right. So, number one customer support resolution agent. So we have agentic loops, control, something called stop reason which is uh what Cloud Code has. Every time something happens, there's a stop reason and you need to take a look at that because that can give you a lot of information about what's going on.

考试覆盖五个领域，而且每个领域占多少分都写明了。agentic 架构占 27%。Claude Code——怎么配置 Claude Code 这套系统和工作流——占 20%。还有 prompt engineering、怎么组织输出结构、到处用 JSON。tool 设计。Model Context Protocol（MCP）集成。这些主题不管你考不考试，都应该弄懂、心里有数。它能帮你为 agentic 世界扔给你的各种状况做好准备。然后还有 context 管理和可靠性。这些就是你会碰到的题目大致覆盖的范围。另外，他们会给你六个生产场景，考试时系统随机抽四个，所有题目都围绕抽中的这四个来出。我今天要做的，就是带你过一遍这些生产场景，并且指出一些要留意的反模式——因为解决问题的方式有很多种，但很关键的一点是「什么不能做」，这往往就是答对这些题的钥匙。第一个：客服问题解决 agent。这里涉及 agentic loop、控制流，还有一个叫 stop reason 的东西——Claude Code 里就有。每次有事情发生，都会带一个 stop reason，你必须去看它，因为它能告诉你大量关于当前状况的信息。

[5:05]

Uh scenario two, code generation. Three, multi-agent research system which we'll look at. How do you How do you distribute your agents? Hub and spoke. Who's the orchestrator? How much information should they know? All these are important factors. Um scenario four, developer productivity with code. So how do you do subtask isolation? Keep your tasks in their little universes. And this hearkens back to what we learn in computer science from doing multi-threaded programming. When you have multiple threads operating and sharing memory, then you get into issues with synchronization. You You to put locks Keep the little threads independent. Keep your agents independent. Um and then some cloud code for continuous integration. And then we'll look at some patterns for structured data extraction. Okay, that's kind of where we're going to go. Now, here's something that I I I like to point out. Everybody's talking about loops, right? Every The loop is the new thing. Um uh Boris Cherney says he doesn't write code, but his job is to write loops. And Peter Steinberger master of Open Claw says, "I don't I don't uh I don't code anymore. I just design loops that prompt your agents." So, loops are the new big thing, right?

场景二，代码生成。场景三，多 agent 研究系统，等下我们会看。你怎么分配你的 agent？中心辐射式（hub and spoke）？谁来当 orchestrator？它们各自该知道多少信息？这些都是重要的考量。场景四，用代码提升开发者生产力。子任务隔离怎么做？让每个任务待在自己的小宇宙里。这其实可以追溯到我们在计算机科学里从多线程编程学到的东西：多个线程同时运行、共享内存的时候，就会遇到同步问题，你得加锁。让那些小线程保持独立，也让你的 agent 保持独立。然后还有把 Claude Code 用在持续集成上。最后我们会看几个结构化数据抽取的模式。好，这大致就是今天要走的路线。这里有件事我特别想指出来。现在人人都在谈 loop，对吧？loop 成了新的热点。Boris Cherny 说他不写代码，他的工作是写 loop。OpenClaw 的作者 Peter Steinberger 说：我已经不写代码了，我只设计那些驱动 agent 的 loop。所以，loop 是眼下最火的东西，对吧？

[6:34]

Well, no, they're not. Okay? Um back in the day uh early days of computing, we had programming languages were exploding. We had Fortran, we had COBOL, and there were big fights. My program My programming language is better than yours. It can do more. No, it can't. We can do this. Böhm and Jacopini, 1966 proved that if you want a language to be Turing complete, which means can compute anything that computers are possibly able to compute, then you need only three things. The ability to to to write statements sequentially, okay? To have if-then conditionals, and the third piece is the loop. If you add the loop, you have Turing computability. And now we are seeing this being resurrected in the agentic world with the focus on loops, cuz up to now we've had sort of sequences. You have prompts, you have maybe if-then, but now we have a loop. And now this is what's giving us the power. This is where the agentic stuff is getting very exciting. Okay. I'm start with uh with scenario one, customer support resolution. So here we have a loop operating and the I'm going to jump to the anti-pattern. What you don't want is just to let the agent go and do something and get the response back and use it, okay? What you want to do is you want to loop with something called the stop reason. So I'm going to show you a little code here.

其实并不是什么新东西。回到计算的早期，编程语言在爆炸式增长，有 Fortran，有 COBOL，当时吵得很凶：我的语言比你的强，能干的事更多。不对，你干不了，我们能做这个。1966 年 Böhm 和 Jacopini 证明了：如果你想让一门语言图灵完备——也就是能算出计算机原理上能算的一切——那你只需要三样东西：能顺序地写语句；有 if-then 条件分支；第三样就是 loop。加上 loop，你就拿到了图灵可计算性。而现在，我们看到这件事在 agentic 世界里被重新翻了出来，焦点就是 loop。因为到目前为止我们基本上只有顺序执行：有 prompt，可能有 if-then，而现在我们有了 loop。这才是力量的来源，agentic 这一块也正是因此变得非常令人兴奋。好，我从场景一开始，客服问题解决。这里有一个 loop 在跑，我直接跳到反模式。你不该做的是：直接让 agent 跑，拿到返回的响应，然后就这么用了。你真正该做的，是围绕 stop reason 来循环。下面我给你看一小段代码。

[8:19]

So here we have while loop. It's a while true, it's a loop. We're looping right here, okay? So the first little block is where we call uh we call the model, okay? And we pass it the messages. The messages are essentially the sequence of prompts that exist in the context window, okay? And we are asking the and we have a we have a prompt and we have we have the context and we have a tool. And we're asking the LLM to do something with this tool and help us out. The problem is the LLM can't do anything. It is just a probabilistic next word predictor. It can't execute tools. So what it does though is it can figure out if you point it to a tool, it can figure out how to set things up so that you or your code can execute it. So it's important to understand that the LLM is not executing these tools. It can't do anything except talk back to you, very intelligently sometimes, but all it can do is talk back to you. So when it finishes this task and has a result which is basically here is I've I know what you want. I know what the tool can do. Here's how I It sets up the parameters that can then be or that then used to actually execute the tool. So, the second block you see why did the LLM come back to us? That's our stop reason.

这里是一个 while 循环，while true，就是个 loop，我们就在这儿循环。第一个小块是我们调用模型的地方，我们把 messages 传进去。messages 本质上就是 context window 里那一串 prompt 的序列。我们有一个 prompt，有 context，还有一个 tool。我们让 LLM 用这个 tool 去做点事、帮我们把活干了。问题在于，LLM 什么都干不了。它只是一个概率性的下一个词预测器，它没法执行 tool。但它能做的是：只要你把某个 tool 指给它，它能想清楚该怎么把参数配好，好让你、或者你的代码去执行。所以务必要理解：执行这些 tool 的不是 LLM。它除了跟你说话什么都做不了——有时候说得非常聪明，但它能做的只有说话。所以当它做完这一轮、给出一个结果时，那个结果本质上是：我知道你想要什么，我知道这个 tool 能干什么，这是我给出的方案——它把参数配好，这些参数随后就能拿去真正执行那个 tool。所以第二块你看到的是：LLM 为什么返回给我们？这就是我们的 stop reason。

[9:57]

Tool use. Oh, okay. We've stopped because the LLM it wants to use the tool. So, let's just run the tool. So, that's what the second block is. Run tool, the response is what the LLM said, and it's basically the parameters that it has extracted from the data that you provided it. Okay? Then it executes that. Then it goes back. That then it continues. Continues means the LLM sees it and says, "Oh, successful run. So, okay." Come back down. We're not running a tool anymore. We're end the end of our loop. Bingo. Now, then we take the answer, and this is an opportunity for you to have a human in the loop potentially. You check the confidence. If it looks good, you keep it. If you don't, then you escalate to a human. So, now there's another reason why you need to make sure you check your stop reason. One of the stop reasons may be you have run out of tokens, and this response is based on partial when the LLM had to stop. And it's going to give you a response, but if you have run out of tokens, then you need to take action. Okay. Um Next scenario. Uh code generation with Claude. So, Claude code has this has this concept of the Claude MD file, a markdown file, where you put all the things you wanted to know.

tool use。哦，好——我们停下来，是因为 LLM 想用这个 tool。那就把这个 tool 跑一下。这就是第二块干的事：执行 tool，响应就是 LLM 给出的内容，本质上就是它从你提供的数据里抽出来的那些参数。然后执行它，执行完再回去，接着继续循环。continue 的意思是 LLM 看到结果说：哦，跑成功了，好。于是走下来，我们不再需要跑 tool 了，循环到此结束。搞定。接下来我们拿到答案，这里就是一个可以引入 human in the loop 的机会：你检查一下置信度，看着靠谱就留下，不靠谱就升级给人来处理。另外还有一个必须检查 stop reason 的理由：某个 stop reason 可能是你的 token 用完了，这个响应是在 LLM 被迫中断时基于不完整内容给出的。它照样会返回给你一个响应，但如果 token 已经耗尽，你就必须采取相应措施。好，下一个场景：用 Claude 做代码生成。Claude Code 有一个 CLAUDE.md 文件的概念，一个 markdown 文件，你把所有希望它知道的东西都写进去。

[11:27]

What Anthropic recommends is you have three levels of Claude. One that you have at the top level of your project, the other that you have in inside your sort of the project folder, and then within directories you can also specify. So, the idea is to have a hierarchical set of rules that that can then control how the system is going to respond. Okay. Moving right along, uh we have a multi-agent research system. So, here we're going to have uh the problem is how do I how do I get my agents to to go off and do stuff and bring the answers back in a reasonable way? The anti-pattern you have one agent and you load it up with tools, all right? So, I like to think about you you know, you hire somebody to come to your house, you hire a carpenter to come to the house, and the guy shows up with uh plumbing tools, carpenter tools, electrical tools. He says, "I can do anything." Well, maybe you don't want this guy, maybe you want a a professional carpenter. So, that's the kind of idea. And this kind of back takes us back to some of the the functional programming uh ideas that functions should be do one thing. And if you can get your agents to do one thing, you with maybe one or two tools available to it, then that's going to be a win, and that's going to help you with this exam. So, specialize, don't overload.

Anthropic 建议你设三个层级的 CLAUDE.md：一个放在项目的最上层，一个放在项目文件夹里面，然后在各个子目录里也可以再指定。思路就是形成一套分层的规则，用它们来控制这套系统怎么响应。好，继续往下，多 agent 研究系统。这里的问题是：我怎么让我的 agent 出去干活，再把答案以一种合理的方式带回来？反模式是：你只用一个 agent，然后往它身上塞满各种 tool。我喜欢这么打比方：你请人来家里干活，你请了个木匠，结果这人带着水管工的工具、木匠的工具、电工的工具来了，说「我什么都能干」。那你可能并不想要这个人，你想要的是一个专业的木匠。大概就是这个意思。这也让人想起函数式编程里的一些理念：一个函数应该只做一件事。如果你能让你的 agent 只做一件事，最多配一两个 tool，那就是一次胜利，考试的时候这个思路也会帮到你。所以：要专一，别塞太满。

[13:07]

The other part of this is don't let your agents context spill over into the main context because context means tokens, tokens mean money, and the more context you have, the more confused the LLM is going to be in giving you an answer. So, even though oh, a million token context window, I can put everything in there. No, no, don't put everything in there. Limit what's going to go in there because then you're going to get a much more accurate system. So, here's a Here's an example of a specialized sub agents. You're giving it So, this would be the critic. So, let's say you've run some stuff. Now, you want to get an agent to look at what's happened. What you want to do is just give it what it needs to solve that critic problem. I'm only giving it here the we're passing it the claim and the evidence. So, this is your claim is sort of how we're going to solve the problem. Here's Here's the evidence, but we're not giving it the the thought processes that went in to creating this claim. Why? When you When you get a bunch of agents together collaborating and talking to each other, there's a tendency to have group think. And all the agents seem to kind of devolve into one idea. I mean, it's it's like, you know, you're in a group, you know, you're at a party, and everybody wants pizza except you, but then people talk you into you you know, you don't want to be uh you don't want to spoil the party, so you'll go along. And it seems that agents kind of work in the same way.

另一半是：别让子 agent 的 context 溢出到主 context 里去，因为 context 就意味着 token，token 就意味着钱，而且 context 越多，LLM 给你答案时越容易犯迷糊。所以别想着「哦，一百万 token 的 context window，我什么都能往里塞」。不不不，别什么都塞。限制住进去的东西，这样你才能得到一个准确得多的系统。这里是一个专用子 agent 的例子。比如这个是 critic（评审 agent）。假设你已经跑了一些东西，现在你想让一个 agent 来看看到底发生了什么。你要做的是：只给它解决这个 critic 问题所需要的东西。我在这里只传给它 claim 和 evidence——claim 大致就是我们打算怎么解决这个问题，这里是支撑它的证据。但我们没有把形成这个 claim 的整个思考过程给它。为什么？当你把一堆 agent 凑在一起协作、互相交流的时候，就会有 group think（群体思维）的倾向，所有 agent 好像都会收敛到同一个想法上去。就像你在一个群体里、在派对上，所有人都想吃披萨，就你不想，但别人一劝，你也就跟着去了——你不想扫大家的兴，就将就了。agent 好像也是这么运作的。

[14:58]

So, you're going to return Basically, you're going to give each agent only a slice. I didn't think about the pizza analogy, but yes. Every agent gets its own slice, and and it it should come through. Okay. Fourth scenario, developer productivity. So, the anti-pattern. Let every subtask dump its full output into the primary thread, crowding out the context. Again, this is what we're I was just talking about. This is bad. Let the context grow unbounded. Bad, right? For the reasons we just talked about. You want to isolate your subtask output, and you want to compact long sessions. I'm going to take a second to talk about that. So, here's here's a an example of a pattern. Uh you want to have your agent uh look at the logs and create a summary of where the problems are in the log. So, here's your task, scan all the logs for error. Context fork. So, you're forking the agent into a like a separate thread where whatever the agent does and thinks and adds tokens to does not come back and pollute the main uh the main context. Now, you see here what happens, then you take this summation, and then you add that summation without all the other stuff into the overriding context. Now, this last little block is kind of interesting, I think. Because you can check your token count, and you can determine how big the token count is.

所以你要返回的，基本上是只给每个 agent 一「片」。披萨这个类比我事先真没想到，不过没错——每个 agent 只拿自己那一片，这个说法应该立得住。好，第四个场景：开发者生产力。反模式是：让每个子任务把它的完整输出一股脑倒进主线程，把 context 挤爆。这就是我刚才在讲的事，这样做很糟。让 context 无限制地涨，很糟，理由刚说过了。你要做的是隔离子任务的输出，并且对长会话做 compact（压缩）。我花点时间讲讲这个。这是一个模式的例子：你想让 agent 去看日志，然后总结出日志里问题都出在哪。所以这里是你的任务：扫描所有日志里的 error。context fork——你把 agent fork 到一个类似独立线程的地方，这样 agent 在里面做的、想的、加进去的那些 token，都不会回流污染主 context。你看接下来发生了什么：你把这份汇总拿出来，只把这份汇总、不带其他杂七杂八的东西，加回到上层的 context 里。最后这一小块我觉得挺有意思：你可以检查 token 数，判断当前 token 量有多大。

[16:53]

And if you can set some limit and you know, if if you have more than 150,000 tokens, then what you want to do is you can run a compact. So, Anthropic and Claude have these compaction algorithms that take this giant context and and compact it in some way, shape, or form. Not quite sure how the implementation is of that, but there is compaction. Now, a little side effect a little side channel I've been walking around when you walk outside, you see see these guys handing out these books. Okay? Anybody see these guys handing out these but take them. This is this is actually a pretty good little book. In fact, I was looking at it last night and one of the things it had in it was this is by this guy Sam Sam Bagwell. I have no connection I didn't even know Sam, but it there's a online page 32. It says uh his company provides custom logic for compression of context. So, he's got an and you can write your own. He's got a he's got he you can extend his base class and have your own compression of your data, whatever you think is important. So, I think that's kind of an interesting spin on this whole thing. Okay. Cloud code for uh uh continuous integration uh anti-pattern Always have interactive modes in a pipeline. Well, no no no cuz interactive modes mean uh Cloud will stop and ask you, "You want to do this? You want to do that? Can I have permission for that?" So, there are ways to set it up so that it'll just run straight through, okay?

然后你可以设一个上限，比如说，一旦超过 15 万 token，你就可以跑一次 compact。Anthropic 和 Claude 有这些 compaction 算法，能把这个巨大的 context 以某种方式压缩掉。具体是怎么实现的我不太确定，但 compaction 确实是有的。这里插一段题外话：你走到外面会看到有人在发这种小册子，有人看到那些发册子的人了吗？拿一本，这本其实挺不错的。我昨晚翻了翻，里面有一段是一个叫 Sam Bagwell 的人写的——我跟他没什么关系，我根本不认识 Sam——在线版第 32 页说，他的公司提供自定义的 context 压缩逻辑。他做了一套东西，你也可以自己写：继承他的基类，实现你自己的数据压缩，压什么、留什么由你判断哪些重要来定。我觉得这是这件事上一个挺有意思的玩法。好，Claude Code 用在持续集成上，反模式是：在流水线里还留着交互模式。这不行，因为交互模式意味着 Claude 会停下来问你：你要做这个吗？要做那个吗？这个能给我权限吗？其实有办法把它配置成一路跑到底。

[18:34]

The other uh the other tip that I'll give you here is there's something called the uh the batch. So, you can take your prompts, you can take your work, and you can put them in a batch and for 50% fewer token cost you will get the result they promise in at at least 24 hours. So, if you're going to go take a nap, you're going to go on vacation, you're going to go out, take a a day off, run your stuff in batch mode, and you're going to have a a less to pay. Where am I here? All right, I've only got a few few minutes left, few seconds left, but I want to conclude with this. Remember, nothing is a mistake. There's no win, there's no fail, there's no exam, only make. You do it and you make it and you're going to succeed. If you want to reach out to me, reach out to me uh coil at Berkeley, look at my websites. I got a website co-supreme AI. I'm a big jazz fan and I named this website after John Coltrane, Love Supreme, if you know that song, great. Anyway, that's my story and I'm sticking to it and I'm about to zero time. Okay,

另外我要给你的一个小贴士，是有个叫 batch（批处理）的东西。你可以把 prompt、把要干的活打包成一个 batch 提交，token 成本便宜 50%，他们承诺最长 24 小时内给你结果。所以如果你要去睡个午觉、要去度假、要休一天假，那就用 batch 模式跑，账单会轻不少。我讲到哪了？好，我只剩几分钟了——其实是只剩几秒了，但我想用这句话收尾：记住，没有什么是错误，没有赢，没有输，也没有什么考试，只有去做。你动手做，做出来，你就会成功。想找我的话，coyle at Berkeley，也可以看看我的网站，我有个网站叫 co-supreme AI。我是个爵士乐迷，这个网站是照着 John Coltrane 的《A Love Supreme》取的名，你要是知道那首曲子就再好不过了。总之，我的故事就讲到这儿，我坚持这个说法，时间也差不多归零了。好，

[19:50]

[applause]

[掌声]

[19:51]

thank you.

谢谢大家。