Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI
频道: AI Engineer
视频: https://www.youtube.com/watch?v=WP3hjUXd918
原文语言: en
统计: 共 35 轮 · Louis-François 12 · Omar 12 · Samridhi 9
[0:13] Louis-François
All right. Good afternoon everyone. Thank you for joining and not watching the game. Uh I hope it will be a bit more interesting or at least you will learn something compared to to uh hopefully Germany winning or some uh anyways. Yeah. All right. Is it fine? Okay. All right. So I'm here to talk about we are here to talk about context engineering in 2026. And more specifically, we are here because we've all lived that that situation where you try to do things with an agent and ultimately it does just exactly the thing that you don't want it to do. And it in my case it usually ends up like this where I'm super mad and I just type back hoping it it learns. And uh usually the problem here is not that the the model just got dumber and you need to switch to cloud or to codeex or or whatever the the harness that you're using, but it's more that the the context is filling up and it's getting worse and worse. The results are getting worse because of it. In our case, this is important because we build courses and trainings for AI engineers specifically. And one of the features that we provide is an AI tutor to help answer questions based on our lessons. And if the interaction is just like the one before and they are super mad at us, they might just ask for refund and it could end up like this. So that's not what we want.
好，大家下午好。感谢各位没去看球赛，而是来到这里。希望这场分享能更有意思一点，或者至少让你们学到点东西——比看德国队赢球更有价值吧，反正……嗯。好的，声音没问题吧？行。那我们开始。今天我们要聊的是 2026 年的 context engineering。更具体地说，我们之所以站在这儿，是因为我们都经历过那种场面：你想让 agent 做点什么，结果它偏偏做出你最不想要的那个东西。到我这儿通常就变成这样——我气得不行，然后疯狂打字回怼它，指望它能长点记性。而这里的问题，通常并不是模型突然变笨了、你得换去 Claude 或者 Codex 或者别的什么 harness，而是 context 被越塞越满，结果因此越来越差。对我们来说这件事特别重要，因为我们专门为 AI engineer 做课程和培训。我们提供的功能之一，就是一个 AI tutor，基于我们的课程内容回答学员的问题。如果学员的交互体验就跟刚才那种一样、气得要命，那他们可能直接就来退款了，结局就变成这样。这显然不是我们想要的。
[1:48] Louis-François
And so what we did for this workshop and just for the AI tutor in general is to run many different experiments in order to figure out how in our case we can fix context rut or at least improve the i tutor as much as possible and reduce the cost as well of running the tutor. Uh the QR code here is a link to a hugging face space where you have all these experiments that you can see and also the AI tutor is open source. who will share another code for the repo but it's also linked on the hugging face. So everything is open source you can access everything uh and even see the experiments online and use the AI tutorial online as well. In the next 80 minutes, we will I will start talking about compaction, memory retrieval and um everything that you can do in 2026 that usually works and then my colleagues will jump in with our the architecture of our AI tutor, our decisions, what we built and the evaluations that we built, the how we built them and what we decided to evaluate and then the results and what we took out of this. So of of course it's applied to our use case and our AI tutor but hopefully you can get away some interesting insights at least from from this and some best practices that we learned throughout more specifically we is towards AI. Um I founded the company with my partners in in a few years ago and we've always been focused around education. Obviously back in the day it was more about computer vision and and more basic machine learning. Now it's towards AI engineering and agents, anything that works for the industry.
所以，为了这场 workshop，也为了 AI tutor 本身，我们跑了一大堆不同的实验，想搞清楚在我们这个场景下怎么解决 context rot，或者至少把 AI tutor 尽可能做好，同时把运行成本也降下来。这里的二维码是一个 Hugging Face space 的链接，所有这些实验都在上面，你们都能看到；AI tutor 本身也是开源的，我待会儿会再给一个 repo 的地址，不过 Hugging Face 上也有链接。所以全部都是开源的，你们都能拿到，甚至可以在线看实验结果、在线用这个 AI tutor。接下来的 80 分钟里，我会先讲 compaction、memory、retrieval，以及 2026 年那些通常真的管用的做法；然后我的同事会接上来，讲我们 AI tutor 的架构、我们做过的取舍、我们搭了什么、我们建了哪些 eval、怎么建的、决定评估什么，最后是结果，以及我们从中总结出了什么。当然，这些都是针对我们自己的场景和 AI tutor 的，但希望你们至少能从中拿走一些有意思的洞见，还有我们一路踩坑总结出的最佳实践。说到「我们」，就是 Towards AI。我几年前和几位合伙人一起创立了这家公司，我们一直专注在教育上。当然，早些年做的更多是 computer vision 和比较基础的机器学习，现在则转向了 AI engineering 和 agent——凡是产业里真正用得上的东西。
[3:29] Louis-François
And I'm joined by my colleagues helping me develop this AI tutor and our courses, Omar and Samidi that we'll jump in later on. And here more specifically towards AI is quite large. But what one one of the things that we do is our academy. So the towards AI academy where we build courses technical courses for AI engineers to upskill towards AI engineering and as I said we provide an AI tutor for the students. So the the AI tutor specifically uh will will be like our baseline for all our experiments. We will use that to test all the different features based on real user interactions and to have the best results possible. We had five requirements we wanted to ensure that the chatbot follows. The first one obviously we want the answers of the chatbot to be grounded in our content not just its own knowledge. We need the the tutor to be based in the current students and current lesson to not be a because we have multiple courses. So we just need to ensure that it answers from this course content. Then it needs to hold long help sessions in case the student is debugging or or just iterating a lot with the the tutor and uh obviously handled code because it's for AI engineers. So we just code a lot and it needs to have somewhat of a low latency to not be frustrating to use. And all of this is related to context engineering. And here we'll be talking about uh what is context engineering in 2026 or at least what we figured out from this from these experiments.
跟我一起做这个 AI tutor 和课程的两位同事是 Omar 和 Samridhi，他们待会儿会上来讲。具体说，Towards AI 摊子铺得挺大，但我们做的事情之一是我们的 academy——Towards AI Academy，我们在里面做技术课程，帮 AI engineer 提升技能、转向 AI engineering。前面说过，我们给学员配了一个 AI tutor。这个 AI tutor 就是我们所有实验的 baseline，我们用它来测试各种不同的方案，基于真实的用户交互，去拿到尽可能好的结果。我们给这个 chatbot 定了五条硬要求。第一条显然是：回答必须扎根在我们自己的内容里，而不是模型自己的知识。第二，tutor 必须锚定在当前这个学员、当前这节课上，因为我们有多门课程，所以得确保它是基于这门课的内容作答。第三，它得撑得住长时间的答疑会话，比如学员在调 bug、或者跟 tutor 反复来回折腾。第四，显然要能处理代码，因为面向的是 AI engineer，我们写代码写得很多。第五，延迟得足够低，不然用起来太挫败。而这些全都跟 context engineering 有关。所以接下来我们就讲，2026 年的 context engineering 到底是什么——至少是我们从这些实验里摸出来的答案。
[5:13] Louis-François
And since everything here is in the context in the in the context of models, it creates two problems. First, the context window is finite. Uh everything the model will see from instructions to to lessons to code will lie in the same space that is limited. And the more you pile things in this space, the worse the results will be and the more expensive it will be because you pay for more tokens. So that's one of the the main problem we're trying to fix. And the second problem is that the model is stateless. So when a student reopens the AI tutorially if you don't build anything around it, the model will have no idea what's going on. It's just starting from zero. So this creates two things we have to work on. the context management which means within one session and the memory aspect of these models which means across sessions in these experiments and in this workshop we focus on the first part the context management because you cannot have multiple sessions is if one session is is shitty. So we try to really optimize for this and maybe in a future workshop we will do one about memory hopefully. So for context management what does the the tutor sees in our case? It sees uh many things from the system prompt to being for being a tutor and the context of our courses and stuff to tool definitions uh so which tools it can use how to use them when to use them. The chat history if it's an ongoing discussion any old tool outputs that was called course chunks that we retrieved to answer the students and finally the user questions. So it's just not the user question that we send obviously and typically it's the smallest part but it can also have contain a lot of code to debug and to help or error logs. So it can be large as well and all of this together just ends up costing more and more to us. So we really want to optimize this this context management aspect. And what we've seen just quickly is that the the main bottleneck or the
既然所有东西最终都落在模型的 context 里，这就带来两个问题。第一，context window 是有限的。模型能看到的一切——从指令到课程内容再到代码——都挤在同一个有限的空间里。你往这个空间里塞得越多，结果就越差，花的钱也越多，因为 token 是要付费的。这是我们要解决的主要问题之一。第二个问题是模型是无状态的。所以当一个学员重新打开 AI tutor 时，如果你什么都没在外面搭，模型完全不知道之前发生过什么，它就是从零开始。于是就派生出两件必须做的事：一是 context management，也就是单次会话内部的管理；二是模型的 memory，也就是跨会话的记忆。在这些实验和这场 workshop 里，我们聚焦第一部分——context management。因为如果单次会话本身就一塌糊涂，那多轮会话根本无从谈起。所以我们先把这块真正优化好，memory 那部分也许留到以后的 workshop 再讲。那么就 context management 而言，在我们的场景里 tutor 到底看到些什么？它看到的东西很多：system prompt（定义它是个 tutor、我们课程的背景等等）、tool 定义（有哪些工具可用、怎么用、什么时候用）、如果是进行中的对话还有 chat history、之前所有 tool 的输出，也就是我们为了回答学员而检索出来的课程 chunk，最后才是用户的问题。所以我们发过去的显然不只是用户那一句提问，而且通常提问是里面最小的一块——但它也可能包含大量待调试的代码或者错误日志，所以也可能很大。所有这些加在一起，成本就一路往上涨。所以我们真的很想把 context management 这块优化好。我们观察到的是，
[7:20] Louis-François
main problem scaling the context is the old tool outputs which contains like any old chunks retrieved all the tool calls and tool results pairs uh from from all the tools called over the the many sessions that it could have the many turns that that one session could have have and any uh file or searches that it did to in its own memory. And the main problem is not is not that it's more costly. It's also that the quality degrades as we know with a program called context route because of the way how large language models are trained to handle longer context with we just inject facts into large compass which doesn't tell them to manage the whole the whole context together or understand the global context and uh so so one of the reason is to help with quality. We want to reduce the context as much as possible. And the others are that if you reququery the model in a in a discussion, you resend every previous tokens. So you pay for them as well again, which is far from ideal. And you increase the the latency, so the time to first token or TTFT uh which makes which creates a very bad user experience. So you may want to in our case manage our context for speed and for spending not just because the quality drops as we will see in our experiments and and in the results in the near future.
context 膨胀的主要瓶颈、主要问题，是那些历史 tool 输出——包括检索出来的所有旧 chunk、所有 tool call 与 tool result 的配对（来自一次会话中可能有的很多轮、很多次工具调用），以及它在自己 memory 里做的任何文件读取或搜索。而最主要的问题还不是成本更高，而是质量会衰减，也就是我们熟知的 context rot。这跟大语言模型被训练去处理长 context 的方式有关：训练时基本是往一大堆语料里注入事实点，这并没有教会它统筹管理整个 context、或者理解全局脉络。所以我们想尽可能压缩 context，原因之一就是保住质量。另外两个原因是：在一段对话中每次重新请求模型，你都要把之前所有 token 再发一遍，等于又付一次钱，这显然远谈不上理想；同时延迟也会上升，也就是 time to first token（TTFT）变长，用户体验就很糟。所以在我们的场景里，管理 context 不只是因为质量会下滑，也是为了速度和为了省钱——这些在我们后面的实验和结果里都会看到。
[8:51] Louis-François
So how we do that to manage uh spending and and speed? How do we manage our context? We use compaction. And the idea is just very simple. Uh the idea of compaction is very simple. is just to try to have the smallest context possible that contains the information to be able to answer the question and you drop or save the rest somewhere. And to do compaction, even a good compaction, you don't necessarily have to have large language models. You can start quite cheap with trivial tools like uh if you have if you use tools in your system um like a web search or just executing code you can automatically truncate outliers. So if you have like once tool that that produce 300 lines instead of 10 usually you can just truncate almost everything except the the head and tail the the beginning and the end and just re write that it's truncated so that the model in the future can recall the tool if it's if it feels it lacks context. You can use the the simplest approach that works the best to just use a sliding window or just trim. Basically, use the last n number of of turns that the user sent which uh you need to determine by based on your your own system and your users and you can clear uh for for some specific tools depending on those that you implement you can just always clear most of the outputs.
那我们具体怎么做，才能同时管住花销和速度？我们怎么管理 context？答案是用 compaction。思路其实非常简单。compaction 的想法就是：让 context 尽可能小，小到刚好装下回答这个问题所需的信息，其余的要么丢掉、要么存到别处去。而且要做 compaction，哪怕是做得不错的 compaction，也不一定非得用大语言模型。你可以从很便宜的手段开始，用一些很土的工具就行。比如你的系统里如果用到了工具——像 web search、或者直接执行代码——你可以自动截断那些异常大的输出。假设某个工具本来输出 10 行，这次吐了 300 行，你通常可以把中间几乎全砍掉，只留头和尾，然后写一句「此处已截断」，这样模型以后如果觉得信息不够，还能再把这个工具调一次。你也可以用最简单、但效果往往最好的办法：滑动窗口，或者干脆截断——就是只保留用户最近 n 轮对话，这个 n 需要你根据自己的系统和用户去定。另外针对某些特定的工具，看你具体实现了哪些，你可以永远把它们的大部分输出直接清掉。
[10:16] Louis-François
So that's for when that's not even using language models. And then you can use language models to basically spend tokens to save even more tokens. And in this case, you don't even have to to use large language models. You can even use smaller ones or even low super small local ones that that runs in like one MacBook to do a few techniques. There are many techniques that exist for for compacting. the those that had the most impact in our experiments where selective retention where the language models will just decide based on where the discussion go is going what to keep what to discard. Then the simplest one here summarization. So just summarize continuously summarize the some previous terms depending on your on on your own application and in the end you can do here what cloud code does. So when it reaches the limit you just sum produce a summary of everything and reset completely with that summary. And there are many more techniques I highlighted here those that work the best. We will discuss them later on in this workshop and the actual results and experiment setup but those are the most successful techniques. And I want to highlight also delta summarization that cloud code uses that is very useful when you spawn sub aents when you use sub aents. It just means to keep a summary and updating the summary based on the new summary that you produce over time and then the the sub agent will just give that to the main agent. But in our case, we don't use sub agents because the tutor works really well with just one main. So we don't need to add complexity for that.
以上都还没动用语言模型。接下来你就可以用语言模型了——本质上是花一些 token 去省下更多 token。而且这里你甚至不需要用大模型，用小一点的、甚至能在一台 MacBook 上跑的超小本地模型，就能实现好几种技巧。compaction 的技巧有很多，在我们的实验里效果最明显的是 selective retention：由语言模型根据对话走向来判断哪些该留、哪些该丢。然后是最简单的 summarization，也就是持续地把之前的一些轮次做摘要，具体怎么做取决于你自己的应用。最后你还可以做 Claude Code 那一套：一旦触到上限，就把前面所有内容生成一份摘要，然后带着这份摘要彻底重置。技巧还有很多，我这里标出来的是效果最好的那几个。稍后我们会在 workshop 里详细讲它们、讲实验设置和实际结果，但这几个是最成功的。我还想特别提一下 Claude Code 用的 delta summarization，它在你 spawn sub agent、用 sub agent 的时候特别有用：思路是维护一份摘要，并根据后续产出的新摘要不断更新它，然后 sub agent 把这份东西交给主 agent。不过在我们的场景里我们没用 sub agent，因为这个 tutor 只用一个主 agent 就跑得很好，没必要为此增加复杂度。
[11:55] Louis-François
And lastly, after spending token to to save more tokens, you can also offload things. Right now it's obviously memory and skills are super popular. So you can of course offload to your memory. So just saving text in your documentations and you can use what's been there for years now. Uh retrieve augmented generation which is very powerful. And as a side note we also compared with graph rag. So everything even if I not I don't mention them we compared them and I highlight just the the best results here. So we we compared graph rag with with rag here and it's in our case it just ended up being way costlier to set up and just tie on the results because it basically was 100% based on our real user evaluations. So we don't need to use graph rag but it's it depends on your own case. If you have a very large data set with inter with relations and interconnected topics and things, it might be worth implementing. So you definitely want to still test it. And uh speaking of memory and of up offloading to files, this is uh as I said, basically just saving them locally or or in a server for for you to use, which means it's fully reversible because you don't lose anything. you don't lose any ongoing discussion. You just save them ready to be referred to in the future if the same student comes back and asks related questions. And it's basically the the Tpetes idea of the LLM wiki uh which if you link with some sort of uh chunks so a version of rag it's pretty powerful and it makes your system become quite cheap and durable and it's easy to inspect from both humans and agents. So it's really interesting. More specifically, it looks like this in our case. So we have just chunks. We save everything into chunks and we cross-link the chunks with pointers and then these chunks are linked to raw data files from where they come from. Then we have one index that will just map all these chunks. So just a link to all the chunks and some
最后，在「花 token 省 token」之外，你还可以把东西 offload 出去。现在 memory 和 skills 显然特别火，所以你当然可以 offload 到 memory 里，也就是把文本存进你的文档；你也可以用那个已经存在好多年的老办法——RAG（retrieval augmented generation），它非常强。顺带说一句，我们也跟 graph RAG 做了对比。所以就算我没提到的方案，我们其实都比过，这里只是把最好的结果列出来。我们把 graph RAG 和普通 RAG 比了一下，在我们的场景里，graph RAG 搭起来成本高得多，而效果基本打平——这个结论是 100% 基于我们真实用户的评测得出的。所以我们用不着 graph RAG，但这要看你自己的情况：如果你的数据集非常大、里面有很多关系、话题之间高度互联，那可能就值得上。所以你还是应该亲自测一测。说到 memory 和 offload 到文件，正如我刚才说的，本质上就是把它们存到本地或者服务器上供你调用，这意味着这个过程是完全可逆的——你什么都不会丢，进行中的对话一点也不会丢，你只是把它们存好，等同一个学员回来问相关问题时随时可以调出来参考。这基本上就是「LLM wiki」的思路：如果你再把它跟某种 chunk 机制、也就是某种形式的 RAG 结合起来，威力相当大，能让你的系统变得既便宜又耐用，而且人和 agent 都很容易检查它。所以真的挺有意思的。具体到我们这边，它长这样：我们有一堆 chunk，把所有东西都存成 chunk，chunk 之间用指针互相交叉链接，然后这些 chunk 又链回它们来源的原始数据文件。再往上有一个 index，把所有这些 chunk 映射起来——也就是指向所有 chunk 的链接，外加一些
[14:09] Louis-François
context of what it is about and the agent will just see this index. So it sees it's like I think it was 450 tokens. So it's very small and it just sees that index and then uh based on the user question if it seems to be related to some user specific question that it may exists in the memory it will scan the index it will go back to the chunk if it's enough it will answer based on the junk the chunk retrieved if it's not enough it can even go back to the raw data to have even more information. So basically it's just the best way to pull context accordingly to the task complexity. So if the task is complex you will pull more and if it's simple you'll p pull less. And a parenthesis on this what we've seen working with our clients and just building this in general is that right now everyone is is converging towards having more and smaller skills. So you have it's it's way better to build small very precise skills that refer to each other's skills to uh to save on context and just load skills one by one and even be able to spawn a sub agent with one dedicated skill context instead of just basically to save context. It's a well the idea of progressive disclosure. So you just load what you need right now.
说明它们讲的是什么的上下文；agent 只看到这个 index。我记得它大概只有 450 个 token，非常小。agent 只看这个 index，然后根据用户的问题——如果这个问题看起来跟 memory 里可能存在的某些用户特定信息有关——它就会扫一遍 index，顺着回到对应的 chunk；如果 chunk 里的信息够了，就直接基于取回的 chunk 作答；如果不够，它甚至可以再回到原始数据里去拿更多信息。所以本质上，这是按任务复杂度来拉取 context 的最好方式：任务复杂就多拉一点，任务简单就少拉一点。这里插一句题外话——我们在服务客户、以及自己做这些东西的过程中看到，现在大家都在往「skill 更多、更小」的方向收敛。也就是说，做一堆小而精准、彼此可以互相引用的 skill，要比做大 skill 好得多，这样可以省 context，一个一个按需加载，甚至可以只带着某一个专用 skill 的 context 去 spawn 一个 sub agent——归根到底还是为了省 context。这其实就是 progressive disclosure（渐进式披露）的思路：只加载你此刻真正需要的东西。
[15:32] Louis-François
Okay, just to go back now on on compaction. This is what the the talk is about because memory we have made some experiments but we couldn't really fit here. It was a bit too much. And uh when talking about compaction there's an important problem or solution that uh that appeared recently uh well not recently but was way more popularized is the uh the the main problem is that first the main problem is that you when you ask a follow-up question you need to recomputee all previous tokens every time. So you just end up paying twice for the same tokens or three times or four times if the conversation is going which is obviously far from ideal. So what providers do nowadays is to offer prom caching. So you will it they will save the embedding and and KV cache and I won't enter into the details but they will premputee they will have saved a lot of the the compute for some tokens and you can just reload them and what's interesting for us is that these already sent token that you reuse are much much much cheaper uh specifically it can go up to 50 times cheaper with some API like deepseek uh which we will discuss in the experiments and what that means is that If you send a very long context, you will pay just one 15th of the the price per token and just pay the full price of the qu the new user question or the new interaction. And that's a problem for compaction because when you are compacting, summarizing, doing any transformation to this context, the the provider cannot use the cache because it it's a new context. It doesn't the model is not intelligent enough to understand it's the same topic change. it it just cannot use the cache. So you will pay full price for these new transform tokens. So here what it means is that you need to to compress to for for compaction to be worthwhile. You need to compress by more than 50 times the context. So it can be quite difficult in some cases without losing
好，我们再回到 compaction 这个话题上来，这次分享讲的就是它——memory 那块我们虽然也做了一些实验，但实在塞不进来，内容太多了。说到 compaction，最近冒出来一个很重要的问题、或者说解决方案——其实也不算最近，只是最近被大规模普及了。最核心的问题是：你每问一个追问，就得把前面所有 token 重新算一遍。等于同样的 token 你付了两遍钱，对话再往下走就是三遍、四遍，这显然离理想差得很远。所以现在各家 provider 的做法是提供 prompt caching：它们会把 embedding 和 KV cache 存下来——细节我就不展开了——总之很多 token 的计算它们已经提前做好并保存了，你可以直接加载回来。对我们来说关键在于，这些被复用的、已经发过的 token 便宜非常非常多，具体来说在某些 API 上能便宜到 50 倍，比如 deepseek，我们在后面的实验里会讲到。这意味着如果你发一段很长的 context，每个 token 你只付十五分之一的价格，只有新的用户提问、新的这一轮交互才按全价算。而这恰恰是 compaction 的麻烦所在：当你在做 compaction、做摘要、对这段 context 做任何变换的时候，provider 就用不上 cache 了——因为这是一段全新的 context。模型没聪明到能理解这还是同一个话题、只是换了个写法，它就是没法用 cache。所以这些被变换过的新 token，你得按全价付。这里的含义就是：要让 compaction 划算，你必须把 context 压缩 50 倍以上。某些情况下，想做到这一点又不掉
[17:37] Louis-François
quality. So caching is truly a gamecher especially because nowadays almost all APIs offer it very easy to use and typically they they save cost on like on 90% of the cost when you use caching but in some case as I said it can go up to way more than that and not only that it helps with cost but caching cache tokens are also already computed so it's way faster to get the first the answer So ultimately it means that summarization is potentially a trap. You may not want to use it at all or you you may want to just use it very specifically which is what the most serious harnesses do nowadays. cloud code, C codeex and all of them use context caching but also use a different method of compactions and we know that because obviously of the leak and then because codeex is open source and then the APIs also provide ways to manage context directly and caching directly. So it's it becomes when you build yourself a harness like we do with the AI tutor, it becomes really interesting to understand when to use which technique and test them obviously. So where is this going? It mean it means that you don't you don't want to just compact you don't want to summarize everything any time because it may kill the cache and you won't be able to use it. So some best guidance that we found is that obvious obviously when the user seems to talk about a very different topic, you may want to refresh the session to clean it to clear it. Um when you scope or files, use what I described with progressive disclosure to just show the smallest amount of context possible.
质量，是相当困难的。所以 caching 真的是个 gamechanger，尤其是现在几乎所有 API 都提供、而且用起来非常简单。一般来说开了 caching 能省掉大约 90% 的成本，但像我刚说的，某些场景下能省得还要多得多。而且它不只是省钱——cache 住的 token 是已经算过的，所以拿到第一个回答的速度也快得多。所以归根到底，summarization 有可能是个陷阱。你可能压根就不该用它，或者只在非常特定的场合用——这正是现在那些最认真的 harness 的做法。Claude Code、Codex 这些都在用 context caching，同时也用了另一套 compaction 方法。我们之所以知道，一是因为那次泄漏，二是因为 Codex 是开源的。另外这些 API 现在也直接提供了管理 context、管理 cache 的手段。所以当你像我们做 AI tutor 这样自己搭一套 harness 的时候，搞清楚什么时候该用哪种技术、并且真的去测它们，就变得特别有意思。那这会走向哪儿？意思是：你不该一上来就 compact，不该动不动就把所有东西都 summarize，因为那可能把 cache 打掉，你就用不上了。我们摸出来的一些经验是：很明显，当用户开始聊一个完全不同的话题时，你可以刷新 session、把它清干净。当你在限定范围或者处理文件时，用我前面讲的 progressive disclosure，只把最少量的 context 露出来。
[19:33] Louis-François
You want to clear every old tool outputs that are not useful anymore. You may want to compact in some case. We will see that in the experiments in a few minutes. And you may want to optimize for cache hits. So just having the model use be able to use its cache. And regarding that providers are constantly improving their feature set to manage cache. So you just need to uh stay current and follow what what the API allows nowadays. But they all provide different methods and um you will have access to the slides but there's I put a link earlier in the in the first few slides on a very interesting article regarding regarding prompt caching that I recommend checking out. It's like on the sixth or seventh slide but you will have the link to the slides. And lastly, you may want to use a model router to optimize especially cost on various tasks. And most importantly, and what the majority of people don't do, you want to log everything. It's super easy. You just ask cloud to implement opic and track everything. It's you don't have anything to do. So it's definitely worthwhile to to implement and you can track cache it rate. You can track user frustration which we've seen that cloud does. So I keep uh telling it when I'm not happy. And uh you you may want to log for some um abnormally long outputs or any weird behavior that a small language model could detect. And so all of this together is the is context engineering which basically means to decide what the model sees every time you call it. And to us for the AI tutor, it means to decide what to keep in the current context window in order to optimize the caching what to drop or compact and when to do that. Uh and those two first things are exactly what what we studied in many experiments that we that my colleague Omar will share with you right now. And uh so you can follow along with the experiments on the the QR code link. It's the hugging face that I mentioned earlier and in there
你要把那些已经没用的旧 tool 输出全清掉。某些情况下你会想做 compaction，这个几分钟后的实验部分会讲到。你还要为 cache 命中做优化，也就是让模型真的能用上它的 cache。关于这一点，各家 provider 一直在改进管理 cache 的功能集，所以你得保持跟进，看看现在的 API 都允许你做什么，它们给的方法各不相同。另外这份 slides 会给到你们，我在前面几页里放了一篇讲 prompt caching 的文章链接，非常有意思，推荐去看看——大概在第六页还是第七页，反正 slides 的链接你们都会有。最后，你可能还想用一个 model router，针对不同任务做优化，尤其是成本上的优化。而最重要的、也是大多数人不做的一件事：把所有东西都记下来（log everything）。这太简单了，你直接让 Claude 去实现、把一切都埋点追踪起来，你自己什么都不用干，所以绝对值得做。你可以追踪 cache 命中率，可以追踪用户的挫败感——我们看到 Claude 就在这么干，所以我不满意的时候会一直跟它讲。你还可以把那些异常长的输出、或者任何小语言模型能识别出来的怪异行为记录下来。把这些加在一起，就是 context engineering，说白了就是：每次调用模型时，决定模型看到什么。对我们的 AI tutor 来说，就是决定当前 context window 里保留什么以优化 caching、丢掉什么或者 compact 什么、以及什么时候动手。前面这两件事，正是我们在大量实验里研究的内容，接下来我的同事 Omar 会跟大家分享。大家可以扫二维码链接跟着实验一起看，就是我前面提到的那个 Hugging Face，里面
[21:51] Omar
there's also a link to the repo and everything. All right. So, welcome to this second part of the of the workshop. So now that we have an overall idea of what context engineering is and what are the different techniques that we can apply to our agents uh now what we want to do here is see how they actually perform in in our case for our AI tutor. I will first start by describing a little bit about how the AI tutor works. So the system design and then I will follow up with the initial experiments that we did. All right. So, the AI tutor is actually very simple. It's one agent that it's a React type of agent that just loops over over tool calls and thinking blocks. And here we just create a very simple one using the lang chain library. So we use the create agent method with the in-memory saver because we in this case we don't save past conversations we just use the the current history and to customize this agent we use the middleware feature of lang chain where you we can add different uh features that can change the behavior of the agent at runtime. So in this case, we want to summarize for example and clear the tool outputs or also in our case have the the user be able to choose the the source uh uh the sources the like which lessons which courses to to use to to answer.
也有 repo 的链接和所有材料。好，欢迎来到这场 workshop 的第二部分。现在我们对 context engineering 是什么、有哪些技术可以用在 agent 上已经有了整体概念，接下来我想看的是：在我们自己的场景里、在我们这个 AI tutor 上，这些技术实际表现如何。我先讲一点 AI tutor 是怎么工作的，也就是系统设计，然后再讲我们最初做的那批实验。好，其实 AI tutor 非常简单，就是一个 agent，是那种 ReAct 型的 agent，在 tool call 和 thinking block 之间不断循环。这里我们用 LangChain 库搭了一个很简单的版本，用 create agent 方法配上 in-memory saver——因为我们这个场景不保存历史对话，只用当前这一轮的 history。要定制这个 agent，我们用了 LangChain 的 middleware 功能，可以往里加各种在运行时改变 agent 行为的东西。比如在我们这里，我们想做 summarize、想清掉 tool 输出，也想让用户能自己选来源，选用哪些课时、哪些课程来作答。
[23:38] Omar
To do that we also add two different tools. So we have the first one the retrieve tutor context which which uses a very classic hybrid search pipeline. So um semantic search along with keyword search and we combine both results to get the the best possible list of chunks. And then recently we also added the second one which is letting the agent actually browse the file system the knowledge base uh just like we can just like coding agents can in the browsing your codebase for example and we also borrow from the idea of karpathy by creating a wiki and helping the agent basically browse uh this knowledge base more easily. I will come back to these tools in a few in a few me moments. We also add a fast API app to wrap the whole system with an endpoint and we add a nextgs UI to let students use the DII tutor. So I will talk a little bit about the first tool. So we have a large corpus. So we have all of the lessons from all the different courses that we created over the past two years. uh and also documentation from various public open source libraries like lang chain, lama index and even documentation from uh openai. So they they make available available all of the markdown files from like how to use the open API, how to use codeex and we also have like cloud code documentation. So it's a very big corpus that has over 8 million tokens and of course this cannot fit in a single context window. So we have to store it in a way where we can retrieve the most important or most relevant information.
为此我们还加了两个 tool。第一个是 retrieve tutor context，用的是非常经典的 hybrid search 流程：语义检索加关键词检索，两边结果合并，得到一份尽可能好的 chunk 列表。第二个是我们最近才加的，让 agent 真的去浏览文件系统、浏览知识库，就像编程 agent 浏览你的代码库那样。我们还借用了 Karpathy 的想法，建了一个 wiki，帮 agent 更容易地浏览这个知识库。这两个 tool 我等下会再回来讲。我们还加了一个 FastAPI 应用，用一个 endpoint 把整套系统包起来，再加一个 Next.js 的 UI，让学生能用上这个 AI tutor。先说第一个 tool。我们的语料库很大：过去两年我们做的所有课程的全部课时都在里面，还有各种公开开源库的文档，比如 LangChain、LlamaIndex，甚至还有 OpenAI 的文档——他们把怎么用 OpenAI API、怎么用 Codex 的那些 markdown 文件都开放出来了，我们也有 Claude Code 的文档。所以这是一个非常大的语料库，超过 800 万 token，当然不可能塞进一个 context window。所以我们必须用一种能把最重要、最相关的信息检索出来的方式，把它存起来。
[25:30] Omar
So in this case the agent receives a question and the user can then choose well beforehand the user can choose specific source. So we can filter this knowledge base. It makes it better for um to improve recall. So precision to get the most relevant information. Then we do hybrid search. So this is very classic um hybrid search. We use embedding model. In this case it's a coherent coherent model with BM25 for the keyword index to get the the most relevant top 30 uh chunks. Then we merge the two results from the semantic similarity and the keyword search and we then rerank to the top five most relevant chunks and that's that's what return to the agent. We also have a limit of 100,000 tokens. So we don't want to. So let's say for example we we go over we just remove the last few chunk uh the last chunks that make it so that we don't cross that threshold. So these numbers the this configuration actually is not random. We did experiments uh to optimize this pipeline. I'm not going to talk about it but it's in one of the courses that we share. Uh basically we just want to try as many configurations as possible and improve recall. So we measure did we retrieve the correct page uh in our knowledge base and we just choose the best uh settings.
在这个流程里，agent 收到一个问题，而用户可以事先选定特定的来源，等于给知识库加了一层过滤，这样能提升 recall——其实更准确说是 precision，拿到最相关的信息。然后我们做 hybrid search，非常经典的做法：用 embedding 模型，这里用的是 Cohere 的模型，再配上 BM25 做关键词索引，各取最相关的 top 30 个 chunk。接着把语义相似度和关键词检索这两路结果合并，rerank 出最相关的 top 5 个 chunk，这就是返回给 agent 的内容。我们还设了 10 万 token 的上限，比如超了，我们就把最后几个 chunk 去掉，保证不越过这个阈值。这些数字、这套配置其实不是随便定的，我们做过实验来优化这条 pipeline。今天我不展开讲，但我们分享的课程里有。基本上就是尽可能多试各种配置、把 recall 提上去，我们衡量的是有没有在知识库里检索到正确的那一页，然后挑出最好的一组参数。
[27:07] Omar
So like I said it uh yeah in this case it's very precise very it's very good but what if the agent needs to browse the whole knowledge base to get the best possible answer. So let's say I want to learn about codeex and I want to learn about cloud code how can I best use uh those two tools and also like if we need to have v various documentation pages in context is this the best possible uh tool and there's a paper uh I I link it here in the in in the slides but it's a yeah it's a paper that shows that basically letting the agent browse the the knowledge base can be very bene beneicial uh so you can look at it if you want afterwards. Very very slow to load. Uh so that's why we ended up creating this second tool the run keep run knowledgebased command where the agent can browse u the knowledge base the file system using bash commands. So first of all what we did first is create these uh three different u folders. So we have the raw folder with all the different markdown files. So all the lessons from the different courses the documentation from the different open open source libraries. We we also have this generated uh folder with this is the generated automatically with basically just the the the titles of each of the markdown files so it's easier for the agent to to find um relevant information. And then we also have the wiki where uh in this case it was cl code that created this. It basically reads all of the raw files in the in the folder in the raw folder and it creates a very uh concise uh set of files. So topics, frameworks uh from the different source uh sources. So for example, I can have a topic related to finetuning. So in this case, the agent will be able to find all of the different raw files related to finetuning. For example, here we let the agent only read. So this is when we actually deploy it. So this is the one you can try on the on the space here. The agent can only read the
就像我说的，这套很精确、效果很好。但如果 agent 需要浏览整个知识库才能给出最好的答案呢？比如我想学 Codex，也想学 Claude Code，想知道这两个工具怎么用最好；又比如我们需要把好几页文档同时放进 context，这个 tool 还是最优解吗？有一篇论文——我在 slides 里放了链接——它表明让 agent 去浏览知识库其实可能非常有帮助，感兴趣的话回头可以看看。（页面）加载特别慢。所以这就是我们后来做第二个 tool 的原因：run knowledge base command，让 agent 能用 bash 命令去浏览知识库、浏览文件系统。首先我们建了三个不同的文件夹。一个是 raw，放所有 markdown 原文件，也就是各门课的全部课时，加上各个开源库的文档。再一个是 generated 文件夹，是自动生成的，基本上就是每个 markdown 文件的标题，方便 agent 更快找到相关信息。还有一个是 wiki，这个是 Claude Code 做的：它把 raw 文件夹里所有原始文件读一遍，然后生成一组非常精炼的文件，按主题、按框架、按不同来源整理。比如可以有一个跟 fine-tuning 相关的主题，那 agent 就能顺着找到所有跟 fine-tuning 相关的原始文件。这里我们只让 agent 读——这是真正部署的那一版，也就是你们可以在那个 space 上试的版本。agent 只能读这个
[29:30] Omar
knowledge base. It cannot modify it. Uh so we only allow these bash commands that basically cannot uh modify the the file system. We also put some limits around this. So for example, if a command lasts over 8 seconds, we can just return an error or let the agent execute something else because it's taking too much time. And we also cap the tool outputs to 40,000 uh characters. So in this case, for example, if a lesson is over this amount, what the agent can do is then okay, so I just got the first 40,000. Let me let me do a follow-up command to get the last uh piece of the of the lesson. For example, uh we also limit the number of the of commands. We actually never see the agent go over this limit of 20 commands per turn, but this is just a a fallback uh in case it takes too too much time to to answer. And we also sandbox of course the agent to only browse the knowledge base this specific folder. Like I said, we create this offline. So the the three different file folders we create this once or every time we want to add a new uh a new course for example we we we tell cloud code can you add a new uh can you add this to the raw raw folder and also create new topics around this new new content. So we do this once and then we deploy it so that the agent u can can browse it with our current system prompt. We can see that it's uh used almost every time. So for almost 90% of the terms we can tweak this to make it use it less uh or more. We didn't optimize for for this specifically.
知识库，不能改它。所以我们只放行那些不会改动文件系统的 bash 命令。我们还加了一些限制。比如一条命令跑超过 8 秒，我们就直接返回一个报错，或者让 agent 去执行别的，因为太耗时了。我们还把 tool 输出截断在 4 万个字符。比如某一课时超过这个量，agent 可以这么办：好，我先拿到了前 4 万字符，那我再发一条后续命令，把这一课剩下的部分取回来。我们也限制了命令条数。实际上我们从没见过 agent 超过每轮 20 条命令这个上限，这只是个兜底，防止它花太长时间才给出答案。当然我们也把 agent 沙箱化了，只能浏览知识库这一个特定文件夹。像我说的，这三个文件夹是我们离线建好的，只建一次；或者每次要加新课程的时候，我们就跟 Claude Code 说：能不能把这个加到 raw 文件夹里，顺便围绕这些新内容建几个新主题。所以是一次性做好，然后部署上去让 agent 能浏览。在我们现在的 system prompt 下，可以看到它几乎每次都会用这个 tool，差不多 90% 的轮次都在用。我们其实可以调，让它少用点或多用点，只是我们没专门为这个做优化。
[31:15] Omar
And I guess the most interesting aspect of doing this is that we measured the precision the recall of using this tool uh actually turning turning it off and we actually got the same amount of recall. So just using the first tool was enough to get all the relevant information and just having this second tool was just 50% slower. Basically, it's faster without this tool because it does less tool calls. Um, and yeah, we we basically didn't see any improvement on uh on re on answering with the correct uh documentation. It it was fun to add but we yeah we didn't see any any benefit and one reason for that is that we tested using real world well basically the questions we get from students and basically those questions were complex enough we guess to actually benefit from using this new new setup. Um so this is what this is how initially uh our AI tutor managed its context. Uh we we started with this because uh it looked fast, it looked good. Uh we didn't actually measure anything. It was just like oh it looks good. Okay, we will just set it set it like this. And so basically we have these three different um context engineering techniques where we clear the outputs uh after 5,000 tokens uh but we at least keep the last five ones. So this was basically a way to keep a very small context uh during a conversation. We also added the the capacity to summarize. So after 30,000 tokens uh the two well the system summarizes the uh the history but we at least keep the last 20 messages to make sure that uh those are very accurate those messages. And we also have the source preference. So that that just allows people to choose what uh sources to use uh when the thetutor answers.
而我觉得做这件事最有意思的一点是：我们量了用这个 tool 时的 precision 和 recall——其实是把它关掉再量——结果发现 recall 完全一样。也就是说，只用第一个 tool 就足以拿到全部相关信息，而多了第二个 tool 只是让速度慢了 50%。说白了，不用这个 tool 反而更快，因为 tool call 更少。而且在能不能引用正确的文档来作答这件事上，我们基本没看到任何提升。加它的过程挺好玩，但确实没看到什么收益。原因之一是我们用的是真实世界的问题来测——基本就是学生问我们的那些问题——我们猜那些问题还不够复杂，没复杂到需要靠这套新配置才能受益。这就是我们的 AI tutor 最初管理 context 的方式。我们一开始这么做，是因为它看起来快、看起来不错，其实我们什么都没量，就是觉得哦看着还行，那就这么设吧。所以基本上我们用了三种 context engineering 技术：一是清掉输出，超过 5000 token 就清，但至少保留最近 5 条，这基本上是为了在对话过程中让 context 保持很小。二是加了 summarize 的能力：超过 3 万 token 之后，系统会把 history 摘要掉，但至少保留最近 20 条消息，确保这些消息足够准确。三是 source preference，让用户自己选 tutor 回答时用哪些来源。
[33:32] Omar
But like I said, these are unproven unproven defaults and we actually want to know what actually works best. Um so I guess you you can Lou showed this at the beginning, but you can access the tutor live. I'm just going to show this very quickly. This is the huging face. So this is a separate UI just to show you. We also have the the chat bubble one on the lessons on the course themselves itself. Uh and here on the on the left you can choose the different uh sources to use, enable them or disable them and then you can uh yeah send your your query. So as you can see here I can just uh send a new request and we see Gemini in this case Gemini 3.5 flash use its reasoning and use its uh capacity to to do tool calling and to answer and I guess uh here there's a I think it's just the internet bugging But we should have a response and yeah the code is open source so you can use your favorite coding agent to explore the codebase and uh and learn about how we implemented this specifically. So we have the different activity u the the activity that the the the model did. So for tool calls it thought four times and it use 10 sources and we have the the final answer. So everything that uh so every time the student uses this chatbot we actually log everything. So we for every single turn we have uh the input tokens, the the output tokens, how many of them were cached, uh what was the cost, what time it took to to get the first token, how many tool calls it did and if we actually if the system actually did something around summarization. So this is very useful and that's uh what we are going to use when uh measuring the uh the different techniques. So why do we need to measure? Well, because the the techniques that Lou showed all sound very smart. So you you might think that they are very uh useful, but sometimes they're not. And uh as we as we discovered with our experiments, actually it might be detrimental. So
但就像我说的，这些都是没验证过的默认配置，我们真正想知道的是到底什么最有效。所以——Lou 开场时已经展示过了，不过这个 tutor 你们现在就能在线用。我快速演示一下。这是 Hugging Face 上的版本，是一个独立的 UI，专门拿来给大家看的；课程本身的每节课里也嵌了一个聊天气泡版。左边这里你可以选要用哪些数据源，逐个启用或关掉，然后就能发你的问题。你们看，我这里发一个新请求，可以看到 Gemini——这次用的是 Gemini 3.5 Flash——在做 reasoning，并调用它的 tool calling 能力来作答。这里……我猜只是网络卡了，正常应该会返回一个回答。另外代码是开源的，你们可以用自己顺手的 coding agent 去逛这个代码库，看我们具体是怎么实现的。这里能看到模型做过的动作：tool call 这一栏它思考了四次、用了 10 个来源，然后是最终答案。所以每次学生用这个 chatbot，我们其实都会把全部过程记录下来。每一轮我们都会记：input token、output token、其中有多少命中了缓存、花了多少钱、首个 token 的耗时、做了几次 tool call，以及系统有没有真的触发 summarization。这些非常有用，也正是我们衡量各种技术时要用的数据。那为什么要衡量？因为 Lou 讲的那些技术听起来都很聪明，你会以为它们很有用，但有时候并不是。而且正如我们在实验里发现的，它甚至可能起反作用。原因在于——
[36:11] Omar
because of the the way APIs cache the tokens when they are um when they are sent and it's also difficult to to know in advance what what is best to use. So before I go into the experiments, I just want to define a few words because we're I'm going to use these words throughout the uh the the presentation. So a preset is basically uh the way the AI tutor was set up in the in the experiment. So for example, it can be in this preset we did summarization at this amount of tokens or in this preset we we used sliding window for example. So that's what a preset is. we have the different tasks. So at task type uh for these initial experiments we only did two tasks single turn and multiple turns also sessions uh called sessions and one run is basically just running one preset on a task and then you get the run. Um the bundle is just the the result. So it's just a JSON file with all the different uh metrics that we save to the disk. So first task single turn. Uh so these are question and answers and we didn't generate this. It's not synthetic. Uh we actually I I had codeex scrape all of the uh questions and responses from our website where students can ask questions and get answers from uh members of the staff. Uh and that's how I got this initial we got this initial um data set. uh we cleaned the data set and only used 60 60 pairs uh because we saw that some of the questions weren't good for the type of task for example uh there were old questions about previous versions of some libraries and we we might not I mean the right now if the tutor answers it's not going to be using this old version of the library so we just uh remove some of the questions some duplicate ones and and and we got this first uh data set and what we measure is the retrieval. So did we retrieve the correct is did the total retrieve the correct lesson for example this is done automatically we can see uh just by looking at the code did we did we use
——各家 API 在 token 发过去时缓存它们的方式不一样，而且你也很难事先知道用哪种最好。在进实验之前，我先定义几个词，因为整场我都会用到。preset 指的是这个 AI tutor 在实验里的配置方式，比如这个 preset 是在多少 token 时做 summarization，那个 preset 用的是 sliding window，这就叫一个 preset。然后是不同的 task，任务类型上，这批初始实验我们只做了两种：single turn（单轮）和 multiple turn，也就是我们说的 session（多轮会话）。一次 run 就是把某个 preset 在某个 task 上跑一遍，跑完得到一次 run。bundle 就是结果本身，一个 JSON 文件，里面是我们落盘的各项指标。先说第一个任务，single turn。这些是问答对，不是我们生成的，不是合成数据。我让 Codex 把我们网站上所有的提问和回复都抓了下来——学生在网站上提问，工作人员来解答，初始数据集就是这么来的。我们清洗了一遍，只留了 60 对，因为有些问题不适合这类任务。比如有些是关于某些库旧版本的老问题，而 tutor 现在回答时肯定不会再用那个旧版本了；所以我们剔掉了一部分问题，还有一些重复的，最后得到第一版数据集。我们衡量的第一项是 retrieval：有没有检索到正确的那节课？这个可以自动判定，看代码就知道有没有用对——
[38:42] Omar
the correct uh lesson or not and we also look at did do we have the correct facts in the answer or the right kind of response in the answer. These two uh are actually graded using using an LLM. Uh you can use APIs to do it but uh right now I think the best way to do it is to just use use your uh code code subscription or your codec subscription because it's cheaper than using the APIs. We also have the second task the session. So multi-turn um conversations back and forth. uh here what we want to know is is the AI tutor able to recall facts after multiple terms. So this is a bit uh in this case we do use some generated uh content. So we do in generate facts that we put at the beginning. So we have a student like a fake student state uh fact. So for example, I want to learn about rag and then we stuff the conversation with just filler messages because we just want yeah we just want to have a lot of messages and then we have um a probe which is just the student asking a question again. So for example it can be uh what should I learn today? And since the uh I'm going to go through it in the next slide, but uh yeah, let's let's let's the example. So for example, here at turn one, we we uh we have the student the fake student implement a state of fact.
——用对那节课。另外我们还看：回答里的事实对不对、回答的类型是不是我们要的那种。这两项是用 LLM 打分的。你可以调 API 来做，但我觉得现在最划算的方式是直接用你手上的 Claude Code 订阅或者 Codex 订阅，比调 API 便宜。第二个任务是 session，也就是多轮来回的对话。这里我们想知道的是：经过很多轮之后，AI tutor 还能不能回忆起之前提到的事实。这种情况下我们确实用了一些生成的内容——会先生成一些事实放在开头，让一个假学生说出一个事实，比如「我想学 RAG」。然后我们往对话里塞一堆填充消息，因为我们就是要让消息变多。接着放一个 probe，就是学生再问一个问题，比如「我今天该学点什么？」。下一页我会细讲，不过先看这个例子：第一轮，这个假学生陈述了一个事实。
[40:25] Omar
So it the student wants to learn about rag evaluation. Um and that that's basically the fact. Then we just add a lot of messages uh filler filler messages and then at at some point we have the student say for example what topic should I learn about today and then what we expect the AI tutor to to to say is that uh the student should learn about rag evaluation. So more specifically heat rate and MR for example. We also have a gate part in the evaluation where let's say we are testing the summarization technique. Uh we actually just want to know did summarization actually happen or not. Um so this is this is this is like one example of one task in the sessions uh data set in the sessions task. Now we have the evaluation hardness. Um so the main uh harness I guess the the main function is the run the run task the run battery function that just uh runs this uh this task. Um we have the grading like I said we it can either be a code check so did we retrieve the correct lesson or not and we can also have the LM as a judge and in this case we use the subscription of code. We also have the check triggers aspect. So this is just a check to to see that if the evaluation went uh good or not. Did we actually compact or not? This is this is just to make sure that the the run is actually good and we can save it. And then we have a generated report uh to see like what was the latency, what was the time to first token and every metric that we can uh measure.
这个学生想学的是 RAG evaluation，这就是那个「事实」。然后我们塞进一大堆填充消息，到某个点上让学生说一句，比如「我今天该学哪个主题？」，我们期望 AI tutor 回答的是：这位学生该学 RAG evaluation，说得更具体点，比如 hit rate 和 MRR。评估里还有一个 gate 环节：假设我们在测 summarization 这个技术，那我们得确认 summarization 到底有没有真的发生。以上就是 sessions 数据集、也就是 sessions 任务里的一个任务样例。接下来是 evaluation harness。主函数大概就是 run task / run battery 这个函数，负责把任务跑起来。打分部分刚才说了，可以是代码校验——有没有检索到正确的那节课；也可以是 LLM as a judge，这种情况我们用的是 Claude Code 的订阅。还有 check triggers 这一块，用来检查这次评估跑得对不对：到底有没有真的做 compaction？这只是为了确认这次 run 是有效的、可以存下来。最后会生成一份报告，看 latency 多少、首个 token 的时间多少，以及所有我们能测的指标。
[42:17] Omar
So yeah, basically we can evaluate everything and then grade it afterwards. Uh we we can run it once and grade it afterwards. So what what we run so we run 11 presets uh and we change them uh for for each experiment. So we have the the full history. So these are the the main ones the full history. So this is the case where we don't don't touch the context. We we leave everything as is in the history. And then we also have the production that I showed at the beginning the defaults that we have. Uh so these are the reference points and then we have these six techniques that I want to compare. So sliding window prompt compression uh selective retention and the other ones and I what I want to see is just what uh memory recall recall do I get if I keep everything else fixed. So I use the same model the same prompt the same tools the same data set what's the difference? Uh now ju now just doing this was a bit expensive. I I didn't expect this to to get over $500, but it did. And that's one of the reason we did afterwards follow-up experiments using cheaper models. But my colleague Sani will talk about this. Uh so what ARM so what basically what preset actually won. Uh, and so these are our um our results. And as you can see, h we weren't expecting this, but basically not touching um the context was actually the best the best strategy for recovering this fact over time over multiple uh messages. Um you can see that the production so the the defaults that we thought were good enough were actually not the best. Uh not doing anything is actually better. H we did two uh two different experiments where one was just one trial and the second was um two trials. So we have like a more statist I I guess it's it's better but the uh the numbers are not like can might not be accurate because it's just one trial and two trials but I guess the most interesting thing is just the order in which the techniques ended
所以基本上我们可以把所有东西先跑出来，之后再打分——跑一次，事后再评。那我们具体跑了什么？我们跑了 11 个 preset，每次实验会调整它们。首先是 full history，这是最主要的几个之一：完全不动 context，历史原样保留。然后是 production，就是我开场展示的那套默认配置。这两个是参照点，另外还有我想对比的六种技术：sliding window、prompt compression、selective retention 等等。我想看的就是：在其他条件全部固定的前提下——同一个模型、同一套 prompt、同样的工具、同样的数据集——memory recall 会差多少？光是跑这些就有点贵了。我没想到会超过 500 美元，但确实超了。这也是我们后来改用更便宜的模型做跟进实验的原因之一，不过这部分我同事 Samridhi 会讲。那到底哪个 preset 赢了？这就是我们的结果。你们能看到，这出乎我们意料：跨多轮把那个事实回忆出来这件事上，完全不动 context 反而是最好的策略。可以看到 production，也就是我们以为够好的那套默认配置，其实并不是最优的——什么都不做反而更好。我们做了两组实验，一组只跑一次，另一组跑两次，这样统计上会稍微稳一点，不过因为只跑了一到两次，数字未必准。我觉得最有意思的还是这些技术最后在表里排出来的顺序。
[44:40] Omar
up being in the table. So in this case keeping everything wins on the memory side but what about the cost? Uh this is what the production cost uh was for the single turn and the session. So almost 50 cents uh for a single turn and 24 cents uh for the multi-turn or each turn. Uh we actually had very good memory uh recall for pretty much all the techniques in the single turn because in single turn you don't have enough tokens to actually fire up the different strategies. Uh so for one response you don't need to do summarization compaction or anything like that. Uh so that's why you you see um high numbers. Uh but as you can see after in the multi-turn uh task you can see that the quality degraded to 38%. uh and we if we compare this to the full history. So here we don't touch the context uh the history of the model we can see that not touching uh is actually cheaper, it's faster and we have better recall overall. So keeping everything wins on on all of these three fronts. Uh so why does it so why is it actually uh like why is it uh why do we have less latency like what's we want to we wanted to understand that and it's basically because if you remove the tool outputs consistently then the agent needs to rerieve uh afterwards for information it already had. So you're just making the agent uh do more tool calls and that's why it ended up costing more uh and yeah using more tokens and having less uh uh less memory recall. So these are the results we initially got using uh Gemini 3.5 with the uh this this data set of 11 to 13 turns is not huge. Uh there's not many messages. That's why we wanted we now want uh in the follow-up part do more different experiments that my colleagues already will will show you.
就是这些技术最后在表里的排序。在这个案例里，记忆这一侧是「全部保留」胜出，那成本呢？这是 production 的成本，单轮和会话各一列：单轮差不多 50 美分，多轮的话每轮 24 美分。单轮任务里几乎所有技术的 memory recall 都很好，因为单轮的 token 量根本不够触发那些策略——回答一次用不着 summarization、compaction 这些东西，所以数字都很高。但到了多轮任务你就能看到，质量掉到了 38%。如果拿它跟 full history 比——这里我们完全不动 context、不动模型的历史——会发现不动它反而更便宜、更快，整体 recall 也更好。所以「全部保留」在这三条战线上都赢了。那为什么会这样？为什么延迟反而更低？我们也想搞清楚。原因基本上是：如果你一直把 tool 的输出裁掉，agent 之后就得为它本来已经拿到过的信息再去检索一遍。你等于是在逼 agent 做更多次 tool call，所以最后钱花得更多、token 用得更多，memory recall 反而更差。以上就是我们最初用 Gemini 3.5 得到的结果。这个数据集只有 11 到 13 轮，不算大，消息也不多，所以我们接下来想在后续部分做更多不同的实验，这部分交给我同事来讲。
[47:03] Samridhi
So yeah, let me in let me introduce you to something. Yeah. Um, so this is going to be my part and um, as we ended on the note where Omar just said that it costed it c um, it costed us almost $600 to um, run the evals that we ran. One question we were trying to um, answer with this extended evaluation was um, so when do you uh, when does compaction actually matter or should you actually compact or not? because we clearly saw that when we have um you know the full answers in the window it works really well but it costs a lot of money. So um I tried to you know do this evaluation in like three sorts of context. Uh first one was cash chats. So you know like when you're chatting with Gemini you'd be able to see that is the cash chat option. The second version is going to be document plus tool. So if you're pasting like a long document in the AI tutor or if there's just tool output what happens then and finally you know if you go local or if you scale this evaluation how well is it going to work out so um should you ever compact so before my part Omar just showed that on Gemini 3.5 flash um keeping everything one um but uh you know the why did we come up with that question was because um you know full history on a frontier model like Gemini would be very very expensive.
那我来给大家介绍点新的。接下来这部分归我讲。刚才 Omar 结尾提到，我们跑那批 eval 花了差不多 600 美元。这轮扩展评估我们想回答的一个问题是：compaction 到底什么时候才有意义？你到底该不该做 compaction？因为我们已经很清楚地看到，把完整答案都留在窗口里效果非常好，但很费钱。所以我试着在三类 context 下做了这个评估。第一类是缓存过的聊天，你用 Gemini 聊天时能看到有个 cached chat 的选项。第二类是文档加工具：如果你往 AI tutor 里粘一份很长的文档，或者纯粹是工具输出，那会怎么样。第三类是，如果你转到本地跑、或者把这套评估的规模放大，效果又会如何。所以，到底该不该做 compaction？在我这部分之前，Omar 已经展示了在 Gemini 3.5 Flash 上是「全部保留」胜出。那我们为什么还会提这个问题？因为在 Gemini 这种前沿模型上跑 full history 会非常非常贵。
[48:33] Samridhi
Um so um you know with all this extended experiment we are trying to figure out was it actually worth it or not. But uh before we start that I wanted to just talk about the different context we see in our AI tutor application. Um so first one is going to be like a long chat history uh where you know we have um a long chat but all of the details what the the students are asking or they're chatting about they can get buried in. Um the second is going to be a pasted document. So you know we do have a lot of students who are going to just like copy paste a lot of documents there. Um so you know um and also because we have like limited uh context window. How does that fit? And then it's going to be you know different tools that we use internally. Um but this is just going to be like bunch of logs that the tools have and since you know these are like different context each of them need some sort of different fix. So you know we had to evaluate it for all different um contexts that we had. So um you know the first obvious thing uh looking at the cost uh we were like okay let's try out a cheaper model and see you know um does it do better you know what sort of techniques work on that does compaction work on it or not.
所以这一整轮扩展实验，我们想弄明白的就是它到底值不值。不过在开讲之前，我想先说说我们在 AI tutor 这个应用里遇到的几类 context。第一类是很长的聊天历史——聊天拉得很长，学生问过什么、聊过什么，细节全被埋掉了。第二类是粘贴进来的文档，我们确实有不少学生会直接复制粘贴一大堆文档进来；而且我们的 context window 是有限的，这些内容怎么塞得下？第三类是我们内部用的各种工具，基本上就是工具吐出来的一堆日志。既然这是几类不同的 context，每一类需要的解法也不一样，所以我们必须针对手上所有这些不同的 context 分别评估。看到成本之后最先想到的一件事就是：好，那我们换个更便宜的模型试试，看它表现怎么样、哪些技术在它身上有效、compaction 在它身上到底管不管用。
[49:45] Samridhi
So um deepseek v4 flash was an obvious option and we were like we will uh try this out on this and see um how well it works out and you know you can see there is um you know a drastic cost difference between Gemini and Deepseek here. So here we can see um you know there's a drastic difference between cost when we checked the performance on deepseek v4 and the main reason was that that we were getting a cash discount. So the cash discount on GE um deepseek was um you know 50x as compared to Gemini and even in this um you know setting we saw that keeping all um you know all of the context still one. So you know we were just getting the best performance um in that case for deepseek as well. So, but you know the the plus thing from this experiment was that we figured out a cheaper model but um you know uh now we wanted to see that even though we have everything uh you know keeping everything uh makes it cheaper but does it remember better you know does uh keeping all of the context remember all of the details the student might be asking us. So um you know this is how I tested out the memory of the model. Um so we have conversation within our system where you know students are asking questions like you know about their setup about the errors that they're seeing what whatever they've already tried. So I just took these questions um so I just took these chats and you know asked questions about specific details just to see if the model is able to like figure that out if it is able to give me an output for that or not. And you know the results I saw is that 95% of the time if the model was able to uh you know give us the right exact details that I was trying to look for and even when it is keeping all of the details whereas if I summarize first or if I compact um the context I had it only gave me the answer back 32% of the time. And um if you think about it um the reason for that is that when you
所以 DeepSeek V4 Flash 是个很自然的选择，我们就想拿它试一把，看效果如何。你们能看到 Gemini 和 DeepSeek 之间的成本差距非常大。我们在 DeepSeek V4 上测性能时，成本差得非常悬殊，主要原因是我们拿到了缓存折扣——DeepSeek 的缓存折扣是 Gemini 的 50 倍。而且在这个设定下我们同样看到：保留全部 context 依然胜出。所以在 DeepSeek 上，全保留也是性能最好的那一档。这个实验的收获是我们找到了一个更便宜的模型。但接下来我们想知道：全部保留确实更省钱，可它记得更牢吗？把 context 全都留着，真的能记住学生可能问到的所有细节吗？我是这么测模型记忆力的：我们系统里有学生提问的真实对话，问的是他们的环境配置、遇到的报错、已经试过哪些办法。我就把这些对话拿过来，针对里面的具体细节提问，看模型能不能把它挖出来、能不能给我正确的答案。我看到的结果是：在保留全部细节的情况下，95% 的时候模型能给出我想找的那个准确细节；而如果我先做 summarize、或者对 context 做 compaction，就只有 32% 的时候能把答案给回来。想想原因的话，那是因为当你——
[51:43] Samridhi
summarize you you know you uh remove all of the necessary details. So you're not able to uh you know keep all of those details and the model keeps on missing those out. So uh you know we are able to see that keeping everything wins in terms of cost. If you have like a you know model like deepseeek um then it also is remembering things. So you know it's not that if you have a long conversation it is not able to remember things. So you know it is correct 95% of the time. So the next thing we wanted to see is the cost part of it. Uh you know how does it actually um does it cost most? Does it cost the lease? What happens in terms of tokens? So, um you know on Deep Seek we saw um the setup that was sending the most tokens is actually the cheapest to run. So the full history u you know setup that we had was sending the most tokens but uh we were still getting the best results out of it because 97% of the tokens that we had were cached and uh you know as you as I as we just saw in the previous slide cash tokens are really cheap. they are uh you know they are charged separately. So um summarizing uh works the other way uh and every turn it makes the model reads and writes uh new tokens and you know this is something that I ran on like 36 turn conversation and it was about 1.78 million tokens and keeping everything still came out ahead and um so you know this answers the question that it wasn't just u you know Gemini it it works same on deepseek as well So um the cheaper model is u what brought the cost down but the results are the same and um you know in in respect of this I wanted the next thing I wanted to ask is what happens when the conversations really grow because all of these were tested out on like short conversation if I really really uh you know increase the length of the conversation how does that work out so um you know when I tried to you know run this entire experiment on a longer
一做 summarize，那些必要的细节就被删掉了。所以你没法把细节都留住，模型就会一直漏掉这些信息。因此我们看到的是：在成本这件事上，「全都留着」反而赢了。如果你用的是 DeepSeek 这类模型，它其实是记得住东西的。也就是说，并不是说对话一长它就记不住了——它有 95% 的时候都答对。接下来我们想看的就是成本这一块：它到底花多少钱？是最贵还是最便宜？token 层面又发生了什么？我们在 DeepSeek 上看到的结果是：发送 token 最多的那套配置，跑起来反而最便宜。我们那套「保留完整历史」的配置发的 token 最多，但效果依然最好，因为其中 97% 的 token 都命中了缓存——就像上一页幻灯片里看到的，cached token 非常便宜，它是单独计费的。而 summarize 的效果正好相反：每一轮模型都要重新读、重新写新的 token。我在一段 36 轮的对话上跑过这个实验，大约 178 万 token，结果还是「全都留着」更划算。所以这也回答了前面那个问题——这不只是 Gemini 的特性，DeepSeek 上同样成立。便宜的模型是把成本压下来的原因，但结果是一样的。基于这一点，我接下来想问的是：当对话真的变得很长时会怎样？因为前面这些测的都是比较短的对话。如果我把对话长度大幅拉长，情况又如何？于是我把整套实验放到更长的
[53:45] Samridhi
context I saw saw that um you know if even if I'm pulling like one specific detail out of it um the model performed really well. So on the top part the green line those are all of the distinctive distinctive facts that the model is able to find out. So we can see that up until 800k tokens as well the model was not missing out on those facts. It was giving me like good and consistent results for uh you know some ambiguous facts. um the performance dropped um you know to half of what I observed on the distinct distinctive facts but for overall our for our AI tutor this uh you know result um was really good. So you know we saw that um the model even when you're not compacting anything holds really well even if you have a really long conversation so uh but so far everything that I've talked about is only per turn. So you know every cost has been per turn but does it help when you know we scale it because um a chatbot is not something um that you know it's not like one turn uh situation for a chatbot. Um the tutor is a long-term service where you know students are asking um you know questions on on a massive scale. So let's say we have uh you know if I I calculated this using this like if we have 100,000 to a million turns of questions every day what would you know be the cost on deepseek or so you know for deepseek the cost was uh approximately somewhere from 18,000 to like 180,000 a month and uh you know even though we don't see that sort of volume as of now but paying per token starts to add up and so the the alternative for that was going over to a local model just to see if we are getting the same sort of performance on local model or not. And um so one way um you know we were thinking that because local models cache as well can you know we use the same sort of setup on a local model and will we get the same result.
context 上重跑了一遍，我发现即使只是从里面捞某一个特定细节，模型表现也相当好。上半部分那条绿线，代表模型能找出来的那些「有明显区分度的事实」。可以看到，一直到 80 万 token，模型都没有漏掉这些事实，结果既准确又稳定。对于那些比较模糊、有歧义的事实，表现会掉到区分度高的那一半左右，但对我们的 AI tutor 来说，这个结果已经很好了。所以我们看到：即使你完全不做 compaction，哪怕对话非常长，模型也扛得住。不过到目前为止，我讲的都还是「单轮」的账——每一笔成本都是按每轮算的。那放大到规模上还成立吗？因为 chatbot 不是一轮就结束的事情。我们这个 tutor 是一个长期服务，学生会大规模地来提问。假设我们每天有 10 万到 100 万轮提问，在 DeepSeek 上成本会是多少？算下来 DeepSeek 大概是每月 1.8 万到 18 万美元。虽然我们现在还没有这个量级，但按 token 付费确实会越滚越多。于是我们想到的替代方案是转向本地模型，看看在本地模型上能不能拿到同样的表现。我们的想法是：既然本地模型也支持缓存，那能不能把同一套配置搬到本地模型上，拿到同样的结果？
[55:49] Samridhi
So uh you know because uh we had like hardware limitations we just tested it out on a MacBook. the maximum context window that we could go up to was 32K and uh so we thought that can we uh you know can we do that locally now uh but we can't because uh you know the the sort of uh lessons that we had are had are bigger than like 32k context window on their own and once the conversation doesn't fit in the window um caching was no longer helpful for us and we uh you know we have to make um the context smaller either by compressing it or by retrieving only the parts that we need. So uh you know the next question we were trying to answer is can we uh you know once you have to compact locally what actually works? So uh you know for um the chat memory going local uh and trying to keep everything stops winning because you can't keep everything and you know a simple uh question here would be that why can't you just keep increasing the length of the model you know why can't you use a bigger model because of course we have hardware limitations for us it was uh you know a MacBook but you know GPUs also have like hardware limitations but you know we went from like a 7B model 8B model to a 32-B model uh but here we landed Ed on the uh you know conclusion that even though you keep increasing the length of the model uh it is not going to increase your context window you know you cannot repair that part you have to you know make a choice there uh but um you know this is only for the chat history and what happens when you are trying to deal with like documents locally so um this was a little surprising because if you are retrieving results with like local documents um you know it was really good we got like 100% in accuracy in that case. So if uh you know when students are pasting something which is too big uh you know rag is a good option there you can use that and it can help you retrieve the exact uh data that you're looking for. Also the
因为有硬件限制，我们只是在一台 MacBook 上做了测试，能跑到的最大 context window 是 32K。我们本来想的是「那能不能干脆在本地做」，但结果是不行——因为我们的课程内容本身就超过 32K context window。一旦对话装不进窗口，缓存就帮不上忙了，我们就必须把 context 变小：要么压缩，要么只检索用得上的那部分。所以下一个要回答的问题就是：当你不得不在本地做 compaction 时，到底什么方案管用？对于本地的 chat memory 来说，「什么都留着」这条路走不通了，因为你根本留不下。这里很自然会有个问题：那为什么不把模型的规模一直往上加？为什么不用更大的模型？当然，我们有硬件限制——对我们来说是 MacBook，但 GPU 同样有硬件上限。我们从 7B 换到 8B，再换到 32B，最后得出的结论是：模型规模再怎么加，也不会把 context window 变大，这一点是修不了的，你必须在这里做取舍。不过这只是 chat history 的情况。那本地处理文档时又如何呢？这一块结果有点出乎意料：如果是对本地文档做检索，效果非常好，准确率能到 100%。所以当学生粘贴进来的内容太大时，RAG 是个好选择，能帮你精准检索到你要的数据。另外，
[57:52] Samridhi
processing time in this case was anywhere from 25 to 65 seconds. So you know which is uh pretty good in terms of the you know the output that we're getting. Uh but you know if you are trying to like stuff the window with like more context than you have. We saw that you know it took us like approximately 340 seconds um to get the output when you know our um conversations were really long and the output that we got was a single token. So you know you are um not getting anything but you're also wasting a lot of time when you are trying to do it. So um you know you have to be careful about what option you choose in this case. So um so you know you also have like different type of retrieval strategies that you could use. The default retrieval is of course semantic search where you're just trying to match uh you know the meaning of the text and that is the dense rag uh heading that you can see on the chart. Um it mostly works but uh you know we saw we tried to make it work from like uh 50k token to 200k and we saw that you know dense um rag worked really well. you know it was like 80%. But when we increased it to like 400k tokens it was not able to facts that were buried in the middle and it started giving us like 0% recall whereas uh you know something like BM25 it still got 100% every time. So semantic search uh on its own is not enough and that's why you know when Omar talked about our setup in the AI tutor we're actually using a hybrid search we're using a mix of both u you know dense and um you know BM25 we're using a combination of both those um so you know after all of this we came up we had like one other question which was like uh how does all of this um local setup compare to cloud because that's the uh real way we'll see the result. So we wanted to put them side by side just to see uh what is the output and for chat uh you know the local setup was not up to the par of um the cloud you know the cloud setup on
这种情况下的处理时间大概在 25 到 65 秒之间。考虑到我们拿到的输出质量，这个耗时算相当不错了。但如果你硬要往窗口里塞超出容量的 context，我们看到的结果是：在对话非常长的情况下，大约要 340 秒才出结果，而输出只有一个 token。也就是说你什么都没拿到，还白白浪费了大量时间。所以在这种场景下，你得非常小心地选方案。另外，检索策略也有很多种。默认的检索方式当然是语义搜索，也就是去匹配文本的含义，对应图表上标着 dense RAG 的那一栏。它大部分情况下是管用的，我们从 5 万 token 一路试到 20 万 token，dense RAG 表现都很好，大概能到 80%。但当我们把量提到 40 万 token 时，它就找不出埋在中间的事实了，召回率直接掉到 0%；而 BM25 依然每次都是 100%。所以光靠语义搜索是不够的——这也是为什么 Omar 讲我们 AI tutor 那套架构时提到，我们实际用的是混合检索（hybrid search），把 dense 和 BM25 两种方式结合起来用。走完这些之后，我们还剩一个问题：这整套本地方案跟云端相比到底如何？因为那才是真正能看出结果的对比。所以我们把两边并排放在一起看输出。在 chat 这块，本地方案达不到云端的水准——
[59:58] Samridhi
cloud keeping everything scores you know somewhere from from 92 to 95%. But locally it was stuck at like 33% and the context window was a limitation here. Also you can see that um you know local um local models actually work because um there is no cost like you cannot see because you already own the hardware though um you know there's a throughput limitation there. Uh but if you use a technique like retrieval uh you know you get like good accuracy even on a local setup. So um in our case what we found is that on memory uh you know keeping the whole chat recalled about like 95% of the details that we were providing it it was able to you know give us correct answer 95% of the time versus it was uh like just 32% if you summarize it um on long context uh you know finding a single fact uh is easy for the model. we went up to 800k tokens and you know we did not um see any sort of context rot in that case. Um on cost per turn we saw that the cheapest run is actually the one which is sending the most tokens because caching makes um resending the same context uh you know very cheap and on no cost at scale um um you know it scales up for like let's say if we have like thousand students uh you know Gemini costs us about like $40,000 40,000 a month whereas Deep Seek Deep Seek was around 1,900 a month um so um going local saves us on a cost a bit more. So the main thing to take away is that um do not compact by default. You have to name the constraint that you have and then you know look for a better alternative.
在云端，「全都留着」能拿到 92% 到 95% 的分数，而本地卡在 33% 左右，瓶颈就是 context window。同时也能看到，本地模型其实是可行的，因为硬件是你自己的，成本这一栏几乎看不到数字，只不过吞吐量是个限制。但如果你用检索这类技术，本地方案也能拿到不错的准确率。所以在我们的案例里，我们发现：在记忆方面，保留完整对话能召回大约 95% 的细节，也就是 95% 的时候能给出正确答案，而做 summarize 只有 32%。在长 context 方面，找单个事实对模型来说很容易，我们一路测到 80 万 token，都没有观察到 context rot。在单轮成本方面，最便宜的那次运行恰恰是发送 token 最多的那一次，因为缓存让「重复发送同样的 context」变得非常便宜。在规模化成本方面，假设我们有一千名学生，Gemini 大约要每月 4 万美元，而 DeepSeek 只要 1900 美元左右。所以走本地路线还能再省一些。最核心的一条 takeaway 是：不要默认就去做 compaction。你得先说清楚自己面对的约束是什么，然后再去找更好的方案。
[1:01:48] Samridhi
So what did we finally decide after all of these different experiments that we ran? Um so we decided on deep uh V4 flash because uh we had hardware limitations. So for us um the cloud structure worked out really well. It is also the cheapest considering the current um intake of students we have. So you know it works out well for us. Um and we're using on top of that model we are using a mix of u you know using hybrid retrieval to uh you know get good results. For memory uh you know we have chosen to keep everything. uh we have got um you know a default limit that after 30k tokens we are going to you know have compaction but up until that we're planning to like keep everything and uh you know that is the tutor setup that we have so you know because we had like time limitations so these are all the evaluations and experiments that I could pack into this time but if you would like to you know learn more about these evaluations or you would want to build a tutor yourself um this is the fullstack AI engineering course uh on academy.towardsai.net. Um so you can go to this link and uh you know go through the course. Uh but thank you so much everyone. Uh and now we can take any questions.
那么跑完这一堆实验之后，我们最终定下来的是什么？我们选了 DeepSeek V4 Flash，因为我们有硬件限制，所以云端这套架构对我们来说效果很好。按我们目前的学生量算，它也是最便宜的，所以对我们是划得来的。在这个模型之上，我们还叠了 hybrid retrieval 来拿到更好的结果。在记忆方面，我们选择「全都留着」，设了一个默认上限：超过 3 万 token 之后才做 compaction，在那之前打算全部保留。这就是我们这套 tutor 的配置。因为时间有限，我能塞进这段分享的评测和实验就这些了。如果你想更深入了解这些评测，或者想自己动手做一个 tutor，这是我们的全栈 AI 工程课程，在 academy.towardsai.net 上，你可以点这个链接去上这门课。非常感谢大家，接下来我们可以回答提问。
[1:03:07]
[applause]
[掌声]
[1:03:22]
[music]
[音乐]