The State of Model Routing — NVIDIA, Cognition, OpenRouter

频道: AI Engineer
视频: https://www.youtube.com/watch?v=QHBjufYK8TA
原文语言: en
统计: 共 107 轮 · 主持人 15 · Walden Yan 19 · Carter 1 · Tuhin 3 · Alex Atallah 17 · NVIDIA 代表 14 · NVIDIA 8

[0:01]

[music]

[音乐]

[0:12] 主持人

have been really exciting. We've tried to get a bunch of the industry leaders together to talk about some of the problems that are that we're facing as we try to run more on local. If you guys were here for the first panel, one of the things that we talked about was model routing. We firmly believe that we're in a multi-model world. I think you heard this from many of the panelists. Anyone who is deploying AI in production and who is doing so locally is seeing that multi-model world. That's why we released these NeMo Triton models at NVIDIA. Everything is released from the data sets to the weights with recipes so that you can customize them. We do that because we know that people customizing models is going to be huge. And so, this panel is really exciting cuz we're going to talk specifically about model routing. So, as you are picking which model to use, how does that how essentially how does that tooling itself look? Um, do you guys want to introduce yourselves?

……真的挺让人兴奋的。我们尽量把行业里的一批领军者请到一起，聊聊我们在往本地部署推进的过程中遇到的一些问题。如果你们听了第一场圆桌，其中一个话题就是 model routing。我们坚信现在是一个多模型并存的世界，这一点你们应该从很多嘉宾那里都听到了。任何一个在生产环境里落地 AI、而且是在本地跑的人，都能感受到这个多模型的世界。这也是我们在 NVIDIA 发布 Nemotron 系列模型的原因——从数据集到权重再到配方全部放出来，让大家可以自己定制。我们这么做，是因为我们知道「用户自己定制模型」这件事会变得非常大。所以这场圆桌特别有意思，因为我们要专门聊 model routing：当你在挑用哪个模型的时候，这套工具链本身到底长什么样？呃，几位要不先自我介绍一下？

[1:03] Walden Yan

Yeah, sure. Um, I'm Walden. I'm the co-founder of Cognition. We build Devin, AI software engineer. Uh, in addition to the product, we spend a lot of time partnering with our customers to figure out how they should deploy these models uh and these agents. And one of the things they're constantly asking us nowadays is basically how do I know the ROI of our models and how do I know which tasks I can actually let our engineers spend the most expensive models on versus, you know, letting them use a more cost-efficient model. And so, that's why we're we're also thinking a lot more about multi-model model routing nowadays.

好的。我是 Walden，Cognition 的联合创始人，我们做的是 Devin——AI 软件工程师。除了产品本身，我们还花很多时间和客户一起研究他们该怎么部署这些模型和 agent。现在客户天天问我们的一个问题基本上就是：我怎么知道模型的 ROI？我怎么判断哪些任务值得让工程师用最贵的模型去跑，哪些任务用更省钱的模型就够了？所以我们最近也在大量思考多模型的 model routing。

[1:32] 主持人

Totally.

完全同意。

[1:33] Carter

Yeah. I'm Carter. You guys heard from me a little bit earlier, but if you weren't here, I'm a developer tech engineer at NVIDIA. And uh ultimately, I spend a lot of time thinking about how to get intelligence into as many developers' hands as possible. And uh something that is continually becoming a not an issue, but something that is top of mind for a lot of developers is uh as you use more intelligence and the frontier models get more expensive, uh it becomes somewhat cost prohibitive to to use the best uh tools, what feels like the best tools, as much as you would like to use them. And so, this has become a recent uh you know, focus is, how can we, you know, still get the same desired outputs, but actually uh both as an individual developer, but also imagine startups and small companies, how can you leverage this incredible tool without totally breaking the bank?

我是 Carter，前面大家听我讲过一点，如果你当时不在场的话——我是 NVIDIA 的 developer tech engineer。说到底，我花很多时间思考怎么把智能送到尽可能多的开发者手里。有件事正在越来越成为——倒也不能说是问题吧，但确实是很多开发者最挂心的事：随着你用的智能越来越多、frontier 模型越来越贵，想按自己心里想的频率去用最好的工具、或者说感觉上最好的那些工具，成本上就有点吃不消了。所以最近的一个重点就是：怎么在拿到同样想要的输出的前提下——不光是对个人开发者，你也可以想想那些创业公司和小团队——怎么用上这么强的工具，又不至于把钱包彻底掏空。

[2:21] Tuhin

Uh I'm Donne. I have worked on model evaluations uh both in terms of its accuracies and efficiency and cost understanding of the model. Uh and then I try and understand those implement those learnings and uh help build a router. So, it's basically my job is to understand the behavior of of the model on an intimate level and then use those learnings to both improve the model and try and design a system of model that can work together with each other.

我是 Tuhin。我一直在做模型评测，既看准确率，也看效率和成本这一块的理解。然后我会试着把这些认知落到实处，帮忙搭一个 router。所以我的工作基本上就是：非常细致地理解模型的行为，再用这些认知去改进模型，并试着设计一套能让多个模型互相配合的系统。

[2:52] 主持人

Totally. Um yeah, I love a lot of the research that you're doing at Nvidia as we kind of see the space through. I think what's really interesting is model routing itself is pretty new still. And so, what you'll notice is there isn't a very clear solution here. That was something that came up on the first panel is that there is a lot of space for startups and for companies in the ecosystem to fill in a solution here cuz we're still figuring out how to best do these patterns. And I think, Walden, I want to kind of ask you. So, um Cognition just released Fusion, your guys's model router. And when you guys released it, you in your blog said that you're actually getting better performance than Fable, um than these frontier models. And I feel like that was a very surprising statement to hear because we were thinking that you're getting as good or close enough usually when we're running on edge, when we're running local in these compute strained, smaller footprint models. But you guys are getting better. Can you explain how?

完全同意。我很喜欢你们在 NVIDIA 做的很多研究，正好我们也在一起把这个领域看清楚。我觉得特别有意思的一点是，model routing 本身还很新，所以你会发现现在并没有一个特别明确的解法。这也是第一场圆桌上提到的：这里给创业公司和生态里的公司留了很大的空间去补上一个解法，因为我们还在摸索这些模式到底怎么做最好。Walden，我想问你一下——Cognition 刚发布了 Fusion，你们的 model router。发布的时候你们在博客里说，它的表现其实比 Fable 更好，比这些 frontier 模型更好。这句话我听着挺意外的，因为我们本来的预期是：在边缘端、在本地跑、在算力受限、体积更小的模型上，通常是做到差不多好、够用就行。但你们居然做得更好。能讲讲是怎么做到的吗？

[3:43] Walden Yan

Yeah, absolutely. Um so, I also want to be be clear about something here is like, you know, we're not saying that that we gap above Fable level performance in in the same way that maybe Fable level performance gaps above other models. I think actually there's this really unintuitive dynamic where smarter models actually get better and better at delegating work. And so one of the philosophies we had with building a model router is we don't want to route people to a dumber model and then suddenly you're stuck with a model that doesn't know how to do your task. Next thing you know you're you're switching yourself back to smarter model anyways and now taking that expensive cost. And in general we think a lot of the existing model routing systems out there are probably the same ones people have been using like a year ago. And so we really wanted to put out a new framework that actually lets people still feel like and and still have a frontier model in their system while getting all these like cost benefits. So yeah we we're reducing the cost of Fable level intelligence by 40%. The way we do that is we allow Fable to still do like the planning and the the hard decision making but delegate a lot of the work to an implementation model. And the implementation model can be you know one of these open source models be it a cheaper mini model. Um the unintuitive thing is even though it's cheaper because you're delegating the work to another model you can let that model go at the task with much more depth and intensity than you might otherwise. Like you can spin off like you know three sub agents to go and like explore the code base and maybe that's actually more comprehensive than if you had just let Fable explore the code base itself. So you're actually getting this like nice trade-off where it's both more cost efficient and it's also more comprehensive overall.

当然。不过我想先把一件事说清楚：我们并不是说我们在 Fable 的水平之上又拉开了一个身位，像 Fable 相对别的模型那样拉开差距。实际上这里有个很反直觉的现象——越聪明的模型，越擅长把活派出去。所以我们做 model router 的一个理念是：我们不想把用户路由到一个更笨的模型上，然后你突然被卡在一个根本不会做你这个任务的模型上，接下来你还是得自己切回聪明的模型，那笔贵的成本照样得付。总体上我们觉得，市面上现有的很多 model routing 系统，大概还是一年前那批东西。所以我们特别想拿出一个新框架，让用户系统里始终有一个 frontier 模型、体感上也确实是在用 frontier 模型，同时还能拿到所有这些成本上的好处。所以，我们把 Fable 级智能的成本降低了 40%。做法是：让 Fable 继续负责规划和那些难的决策，但把大量具体的活儿交给一个实现模型（implementation model）去做。这个实现模型可以是某个开源模型，也可以是更便宜的 mini 模型。反直觉的地方在于：正因为它便宜，你把活派给它之后，反而可以让它以比原来更深、更狠的力度去啃这个任务。比如你可以一口气派出三个 sub agent 去把代码库摸一遍，这可能比让 Fable 自己去看代码库还更全面。所以你其实拿到了一个很妙的权衡——既更省钱，整体上还更周全。

[5:15] 主持人

Interesting. I see. So you're saying by using a bunch of smaller models you're essentially like for one example scouring the code base you can you you can explore it potentially better than if you were to just have one model I don't know figure out with what it's limited with its limited context with whatever path it's on.

有意思，我明白了。所以你的意思是，用一堆更小的模型——就拿翻代码库这件事举例——你可能比只用一个模型探索得更好，因为那一个模型受限于它有限的 context，只能顺着它当时走的那条路往下走。

[5:31] Walden Yan

Yeah, totally.

对，完全是这样。

[5:32]

But also if you think about the budget of if you were to say the frontier model costs this amount per token and the smaller model is this amount per per token and it's significantly cheaper then you can use a lot more tokens from the smaller model with

另外你也可以从预算的角度想：假设 frontier 模型每 token 是这个价，小模型每 token 是那个价，而且便宜得多，那你就能用小模型跑多得多的 token，

[5:45]

still within the budget that it would have been from the Frontier model.

而总花费仍然在原来 frontier 模型那份预算之内。

[5:48] Tuhin

I would also like to encourage everyone to think there are jagged capabilities in most models, right? So, coding is not one domain. Within, let's say data visualization, there'll be scikit-learn, there'll be matplotlib, there'll be something else. It largely comes down comes down to the training corpora that went into each of the models, so right? So, one model, while you're trying to do X type of work, let's say data visualization, and the other type is Y, that means let's say model building. Let's say you're trying to have a data science work stream, where you're trying to optimize for some kind of prediction, and then visualizing your results. Within that task, different models will have different strengths. So, not it's not necessary that model A if scores higher on a coding benchmark, is just plain better at every task that is. So, routing is a task of intimately intimately understanding of behavior of and strengths and weaknesses of different models, and then applying them thusly, right? I would I would encourage everyone to think, "Hey, models are strong at different things, rather than like there's one model to rule them all."

我还想请大家意识到一点：大多数模型的能力都是参差不齐的（jagged capabilities），对吧？所以「写代码」并不是一个单一领域。比如就说数据可视化，里面有 scikit-learn，有 matplotlib，还有别的。这在很大程度上取决于每个模型的训练语料里到底进了什么东西。所以，当你要做 X 这类活——比如数据可视化——和 Y 这类活——比如建模——的时候，情况是不一样的。假设你在跑一条数据科学的工作流，一边要优化某种预测，一边要把结果可视化出来。就在这么一个任务里，不同模型的强项也各不相同。所以并不是说模型 A 在某个 coding benchmark 上分数更高，它就在所有任务上都更强。所以 routing 这件事，本质上是极其细致地理解不同模型的行为、强项和弱项，然后照着这个去分派任务。我想请大家换个思路：模型各有各的强项，而不是「有一个模型能通吃一切」。

[7:01] 主持人

I see. And by the way, real quick, thank you Alex for joining. Yeah.

明白。对了插一句，谢谢 Alex 赶过来。

[7:05] Alex Atallah

Sorry I'm late.

抱歉来晚了。

[7:05] 主持人

No. Oh, is that so?

不会不会。哦，你那个麦克风开着吗？

[7:07] Alex Atallah

I might need yours in. Sorry I'm late.

那我可能得借你的用一下。抱歉迟到了。

[7:10]

[laughter]

[笑声]

[7:11] Alex Atallah

I'm Alex from OpenRouter. Um thanks for having me now.

我是 OpenRouter 的 Alex，谢谢邀请。

[7:14] 主持人

Yeah, of course. Thank you so much. You came right from the airport, so this is perfect. I think um tonight that's that's super interesting. So, um the way that you're thinking through through model routing, it's not even just uh delegating to necessarily a smaller model, but like and maybe this is kind of what you're saying is can you can you put essentially a swarm of agents to accomplish the same task, and suddenly routing the task between them is is much is a is a problem to solve in and of itself.

当然，非常感谢。你是直接从机场赶过来的，这太好了。Tuhin，我觉得你刚才说的这点特别有意思。所以你们思考 model routing 的方式，甚至不只是「把活派给一个更小的模型」——可能这也正是你想说的——而是：你能不能派出一整群 agent 去完成同一个任务？这样一来，在它们之间怎么分派任务，本身就变成了一个需要解决的问题。

[7:36] Tuhin

Yeah. So, if you look at like let's say let's take an easy example. Let's take a science or like scientific discovery as an example, right? Usually these are one-shot problems. It's incredibly hard. You have models think through this process, right? So, in that you have tons of sub-domains. Like tons and tons and tons. So, in that aspect, if you think about post-training like the post-training process of a model, they'd be tuned with different teachers. They'd be tuned on different sub-tasks. So, those those um those overlapping strengths will be readily apparent when you're trying to understand failures of each models on different different sub-tasks. Once you understand that, you can orchestrate your system to leverage that arbitrage essentially, and that essentially becomes free. So, I think this is on LM router bench. There was there are tons of benchmarks out there. But, if you use these techniques, you can get like up to 10% higher accuracy even, right? It depends on the model pool. Depends on the task at hand. But, I would encourage to think about the complementary nature of models.

对。举个简单的例子吧，比如科学发现。这类问题通常是一次性（one-shot）的，难度极高，你得让模型把整个过程想清楚。而这里面有海量的子领域，非常非常多。从这个角度看，如果你去想模型的 post-training 过程：它们会被不同的 teacher 调教，会在不同的子任务上做微调。所以当你去分析各个模型在不同子任务上的失败情况时，那些互相重叠、又互相错开的强项就会很明显地显现出来。一旦你摸清了这个，你就可以把系统编排成去吃这个套利空间，而这部分收益基本上是白捡的。我记得这是在 LLM router bench 上——外面 benchmark 很多——但如果你用这些方法，准确率甚至能提高最多 10%。当然这取决于你的模型池，也取决于具体的任务。总之我建议大家多想想模型之间的互补性。

[8:48] 主持人

I see. Do you see so in kind of the way that you were describing the way that the task is broken up, do you see that the some of the smaller models because the token cost is cheaper, are they using more tokens? Like is it Are Are you specifically routing so that they do or that they so that they are chattier?

明白。那按你刚才描述的这种任务拆分方式，你有没有发现：有些更小的模型，因为 token 成本更便宜，它们会用掉更多 token？还是说你是专门这样路由的，就是要让它们更「话痨」一点？

[9:03] Walden Yan

Uh oh yeah, they absolutely do use more tokens. I I actually I want to kind of like riff on something that Tuhin was saying, which is like you know, a lot of times when you look at these different benchmarks, you'll see that the small models will perform better than like even the frontier models in certain cases. I think a lot of people they look at this and they immediately jump to, "Oh, how can we just route like, you know, the task where the small models do better just straight to the smaller models?" I I think that one of the things we really want to emphasize with our recent blog post and recent Devin Fusion was that this kind of like naive like initial routing to based on the task type is extremely fragile, especially the more agentic the task you you work on is. So, for example, like a real developer, you might ask your agent first, "Oh, how does this code base work?" And then you you go deeper and you're like, "Okay, actually, can you implement some features for me?" And then you go deeper and you're like, "Oh, can you like now go to like a live test of this feature and debug deep cases?" The complexity changes and the type of task changes over time, and you don't want to be left with some subpar model for the the task that you're now on. I think this is why people like frontier models so much is they're they're just like generally intelligent and they're capable of shifting between various different domains, even if you can eke out better performance in very specific tasks. Um, and and what So, the challenge is, how do you get a small model to know that it's out of its depth and you need to now like go switch to another model or go like, you know, go to a smart smarter model. And and our solution to this is you kind of just always have this like main frontier agent that's watching, even if it's not the one doing the work. It should at least be keeping tabs to figure out, "Okay, we like the the agent I delegated

哦，对，它们确实会用掉更多 token。我其实想接着 Tuhin 刚才说的再展开一下：很多时候你看这些不同的 benchmark，会发现小模型在某些情况下表现甚至比 frontier 模型还好。我觉得很多人看到这个，第一反应就是「那我们干脆把小模型更擅长的那类任务直接路由给小模型不就行了」。我们最近那篇博客和 Devin Fusion 特别想强调的一点是：这种「按任务类型做初步路由」的朴素做法极其脆弱，而且你处理的任务越 agentic 就越脆。举个例子，一个真实的开发者，可能一开始问 agent：「这个代码库是怎么运作的？」然后再往深走：「行，那你能不能帮我实现几个功能？」再往深走：「你现在能不能把这个功能拿去跑个实测，把那些很深的 case 调通？」任务的复杂度在变，任务的类型也在随时间变，而你并不希望自己被留在一个配不上当前任务的次等模型上。我觉得这也正是大家这么喜欢 frontier 模型的原因——它们就是通用地聪明，能在各种不同领域之间来回切换，哪怕你在某些非常具体的任务上确实能榨出更好的表现。所以真正的挑战是：你怎么让一个小模型意识到「这事我搞不定了」，得切到另一个模型、或者切到更聪明的模型去。我们的解法是：你基本上永远留一个主 frontier agent 在旁边看着，哪怕干活的不是它。它至少要一直盯着盘，能判断出「好，我派出去的那个 agent……

[10:36] Walden Yan

to now is like out of its depth. I need to kind of like move it to to something else." Um, and and overall, just the guarantee of always having frontier intelligence present, I think reduces the the fragility of of these systems quite a lot.

……现在已经超出它的能力范围了，我得把它换到别的地方去」。总的来说，光是「系统里永远有 frontier 级智能在场」这个保证，我觉得就把这类系统的脆弱性降低了很多。

[10:48] 主持人

How does the sharing of context between one of those smaller agents who has basically completed up to some level of a task and decides, "Actually, I don't think I'm the right person for this. I need to hand it back to the the foundational model." Um, of course, you don't want to have the entire trace of that

那 context 是怎么在它们之间共享的？比如某个小 agent 已经把任务做到某个程度了，然后它判断「其实我不太适合干这个，我得把它交回给基础模型」。当然，你肯定不希望把那个小 agent

[11:05] Walden Yan

Yeah.

对。

[11:05] 主持人

smaller agent be passed back to the larger model. But so, how do you get that level of specificity while basically providing the information it needs, but not more?

的完整 trace 全都塞回给大模型。那你怎么做到既足够精确、又只给它需要的那部分信息，多余的不给？

[11:13] Walden Yan

Yeah, absolutely. So, I I I think like the the context here is it's actually very easy to actually create a system that's more expensive as soon as you're running like, you know, multiple models together cuz oh, no, like, you know, this one file reading. Now, every every one of these models is now reading this one file reading. So, now you're you're you're being charged like three times as much. Um, the the the the trick that we we spent a lot of our time on is um, most of the context by default will only be going to like one model. So, like most of the context let's say will be going to the small model. But, the thing you need to then tune very well is okay, like maybe you still show like what files it's reading, maybe you show like the high-level thinking of what it's doing back to the main model. Maybe you have the small model um, you tune its ability to present the context back to the main model. Um, and actually a lot of these problems already have been well studied in many domains already like context compaction is something you already have to solve if you want to do like really long-running agents. And so, this problem of taking long context compacting it in a way that is now understandable is the one that you can also apply to this domain and just kind of give the compacted context back to back to the main agent.

当然。我觉得这里的关键在于——一旦你开始把多个模型放在一起跑，其实非常容易搞出一个反而更贵的系统。因为「哎不对，就这一次文件读取」——现在每个模型都要把这同一个文件读一遍，于是你被收了三倍的钱。我们花了大量时间琢磨的诀窍是：默认情况下，绝大部分 context 只会流向一个模型，比如说大部分 context 都给小模型。但接下来你要非常仔细地调的是：也许你仍然要把它读了哪些文件展示出来，也许你要把它在做什么的高层思路回传给主模型，也许你还要专门调教小模型「把 context 汇报回主模型」的能力。而且其实这些问题在很多领域已经被研究得挺透了——比如 context compaction，你要做真正长时间运行的 agent，本来就得先解决这个。所以「把长 context 压缩成一份仍然读得懂的东西」这件事，同样可以套用到这个场景：把压缩后的 context 交回给主 agent 就行。

[12:13] 主持人

Context compaction is something that you know, I'm familiar with but I hadn't really thought about as you're doing model routing and as you're trying to share context across now potentially many models, you're you're expanding the amount of what could be seen as wasteful tokens or redundant tokens just just because you have to process that across the many models.

context compaction 这件事我其实挺熟，但一直没往这个方向想过——当你做 model routing、要在可能很多个模型之间共享 context 的时候，你等于是在放大那些可以算作浪费的 token、冗余的 token，因为同一份东西你得在多个模型上各处理一遍。

[12:30] Walden Yan

Yeah. Yeah. Um, I I I think there's the way I describe it is I think we are early in in the in the model routing domain. Um, I I I hope that a year from now that even the techniques we kind of use for Devin Fusion you people look back on that and are like, "Oh, these are some like really legacy ideas and and now we have like much better methods at routing between models." Um, and when people actually start co-designing their models with this in mind we're going to be be in a much better world.

对对。我的说法是，model routing 这个领域我们还处在非常早期。我希望一年之后，大家回头看我们在 Devin Fusion 里用的这些技术，会觉得「这些想法已经很老古董了，现在有好得多的模型间路由方法」。而且等大家真的开始按这个思路去协同设计自己的模型，整个局面会比现在好得多。

[12:55] NVIDIA 代表

Yeah, I I echo what you said that um, I think routing will evolve as the task evolves when when you start task, right? So, um, it's more useful to see things in terms of subtasks and sessions uh, than individual problems that you're trying to solve because more than likely you're when you're working through a problem you're asking a lot of questions, you're you're exploring different things. Um, and it is imperative that you try like people who design routers is imperative that they try and understand these phases of different complexities and then try and apply some logic for essentially side kicking tasks or leveraging expertise from other models that's that's pretty on point.

对，我也认同你说的——我觉得 routing 会随着任务本身的演进而演进。所以更有用的看法，是把事情看成一个个子任务和 session，而不是一个个孤立的待解问题。因为你推进一个问题的过程中，多半会问很多问题、探索很多方向。所以设计 router 的人必须去理解这些不同复杂度的阶段，然后据此设计逻辑——要么把活分给 sidekick，要么去借用别的模型的专长。这一点说得非常到位。

[13:48] 主持人

Yeah, I'd love to hear from the the router guy.

对，我特别想听听我们这位「专做 router 的人」怎么说。

[13:51]

[laughter]

（笑声）

[13:53] Alex Atallah

Yeah, um I think those are these are like important points and one of the biggest debates I think we have internally is whether that outer model that's doing the orchestration should be the big model or the small model. You get like very different results depending on your choice and in fact and it's not even clear what the pricing impact would be because if your outer model that's doing the orchestration is the big model it can leverage its caching to like make more of its decisions and it's caching is going to be like a dramatic price savings compared to the small models caching a lot of the time especially for like perform for you know issues that are on the bright line like zooming out a little bit I think the what you want from all the models out there when you do model fusion is to benefit from all the data that is being trained on across all the labs and and not just the data from one lab right or one source and a model is just like a combination of like the data and its understanding of the data both its compute and the quality of its RL so long term I think you want you want models where they know that oh this is like in distribution like this is in my data you can use small models pretty easily and get a cost savings but if it's out of distribution small models may actually increase your cost because of how often they'll like call tools and how crazy their loops will be. Like if you run terminal bench on Opus and Haiku, like Opus will do about three times better at 1/10 the cost of Haiku, even though Haiku's significantly cheaper per token.

嗯，这些点都很关键。我们内部最大的争论之一，就是负责编排（orchestration）的那个外层模型到底该用大模型还是小模型。选择不同，结果差别非常大。而且连价格影响都说不清——因为如果做编排的外层模型是大模型，它可以靠自己的缓存来做更多决策，而它缓存带来的省钱幅度，很多时候比小模型的缓存要夸张得多，尤其是那些处在边界线上的情况。往上抽象一层看，做 model fusion 的时候，你想要的是从所有实验室、所有来源训练出来的数据里都受益，而不只是某一家实验室、某一个来源的数据。一个模型无非就是数据、它对数据的理解、它的算力，以及它 RL 的质量的组合。所以长期看，你想要的是模型自己知道「这个在分布内、这在我的训练数据里」——那种情况下小模型很好用，能省不少钱；但如果是 out of distribution，小模型反而可能把成本推高，因为它会疯狂调工具、陷进各种离谱的循环里。比如你拿 terminal bench 去跑 Opus 和 Haiku，Opus 的表现大概好 3 倍，成本却只有 Haiku 的十分之一——尽管 Haiku 单 token 便宜得多。

[15:40] Alex Atallah

So it really becomes a huge problem if you use a too small of a model, particularly on tasks that are out of domain for the train data. When you're doing something like classifying text, like hey, is this like a person's name or is this an organization's name? That's super in domain. So you would you don't want that kind of task to go to a large model. You want it to go to a a small model. Everyone has that in their domain. Um so deep like being able to understand in domain, out of domain is a lot of work that we're doing for for um open router fusion. And uh and then also figuring out like what like how to orchestrate the outer and inner models um for different types of task. And like um yeah, it's it's an early industry. It's like an early field of research. Uh most research on model fusion has not been very detailed, not been very like, you know, optimistic sometimes. Um it's only like just, you know, recently getting uh more optimistic. And I think um I'm personally very optimistic about it. And uh you know, like we we're very like ecosystem-driven, collaborative company. And a lot of our like we work with a lot of partners to try like help improve their orchestration pipelines with good primitives, like the, you know, sub-agent and like the advisor tool, um which is kind of similar to what you were talking about.

所以模型用得太小真的会出大问题，尤其是在训练数据覆盖不到的任务上。反过来，像文本分类这种——「这是人名还是机构名？」——就是极度 in domain 的活儿，这种任务你不该丢给大模型，就该交给小模型，谁家都有这类落在自己 domain 里的活。所以，能判断 in domain / out of domain，是我们在 OpenRouter Fusion 上投入很大的一块工作。另外就是搞清楚不同类型的任务该怎么编排外层模型和内层模型。这还是个很早期的行业、很早期的研究领域。之前关于 model fusion 的研究大多不够细，有时候结论也没那么乐观，最近才开始变得乐观一些。我个人对它非常乐观。我们是一家很生态导向、很讲协作的公司，跟很多合作伙伴一起，用好的原语去帮他们改进编排流程，比如 sub-agent、advisor 工具，这跟你刚才讲的东西挺像的。

[16:59] 主持人

I'm curious. So uh help me understand when uh if a it makes total sense that um a small model, if it's in domain, would be cheaper. But if it's not, then it's going to thrash around uh as it tries to get an answer. When you're describing whether the uh con- like the main agent should be the the the local model or the cloud model? Does that is that a decision that's then dependent on whether the task is something that it that's in domain or not? Does my question kind of make sense?

我好奇一点，帮我理解一下：小模型在 in domain 的时候更便宜，这完全说得通；但如果不 in domain，它就会为了憋出答案而反复瞎折腾。那你刚才说的「主 agent 该放在本地模型还是云端模型」——这个决定是不是也取决于任务在不在分布内？我这个问题说清楚了吗？

[17:30] Alex Atallah

I don't know. Um, I basically I don't it's kind of early to

我也说不好。基本上……现在下结论还有点早，

[17:36] Alex Atallah

um to say. I I think the I wait the results that we published, which are a couple weeks ago, which were focused on deep research, not coding.

还太早。我们几周前发布的那批结果，针对的是 deep research，不是写代码。

[17:47]

Mhm.

嗯。

[17:47] Alex Atallah

Um we had the smart model be the wrapper model, be the outer model, and we got the best results from doing that. Um, but it for deep research it works the best. Um, for other tasks, it's like kind of unclear. Like we there it's fusion is not super well optimized for coding. And uh it might be that like a smaller model ends up being like a higher um efficiency per you know, fewer dollars um per like completed successfully completed task, but it's kind of early to say.

我们当时是让聪明的那个模型当外面那层的 wrapper 模型，这么做出来的结果最好。但那是在 deep research 场景下最优。换成别的任务就不好说了——Fusion 在写代码这块还没怎么优化过，也可能反倒是小模型效率更高，也就是每成功完成一个任务花的钱更少，但现在下结论还太早。

[18:22]

Yeah.

对。

[18:23]

[clears throat]

（清嗓）

[18:23] Walden Yan

One thing you said earlier is like, oh, you get the caching benefit from like the main line agent. Um, you actually can get the caching benefit from the side agent. And um, this is actually one of the key things we talked about with our Devin fusion launch is that um, you kind of are leaving a lot on the table if you do a main agent and sub agents type system. Um, so we don't use sub agents. We use what we call a sidekick, which is um, one sub agent that continually has a running context. So the main agent doesn't need to re-provide uh, context from earlier. Um, it's all still in the KB cache, right? Like it's 10 times cheaper on on all those cache tokens. Um, and then if you want to like switch the smart model to be like the one on the side or the one in charge, um it's actually totally fine, and you can kind of like do do the swapping back and forth. We're also spending a lot of time right now thinking about how do you train models to actually work collaboratively with other models? I actually I think there's a lot of literature out there on how you RL one model to do a task end-to-end. How can you RL model to also be good at collaboration? And when we think about it, we actually try both of these setups where let's RL the model being the orchestrator and the one deciding what gets delegated to other models, see how well that performs. Um and we also orchestrate it in a way where the model we're training is actually the executor, the sidekick, and see how well is it at executing other models' instructions. Um and we expect that to to be a probably a big lift in this next step of like multi-model orchestration is don't just like take models as they are and orchestrate them, but like can you actually co-design your models with the orchestration system?

你刚才提到一点，说缓存的好处来自主线那个 agent。其实旁边那个 agent 同样能吃到缓存红利。这也是我们发布 Devin Fusion 时重点讲的事情之一：如果你做成「主 agent + 一堆 sub-agent」那种结构，其实会白白浪费很多。所以我们不用 sub-agent，我们用的是我们叫 sidekick 的东西——一个持续保有运行中 context 的子 agent。这样主 agent 就不用把之前的 context 再喂一遍，那些内容都还在 KV cache 里，对吧？这些缓存 token 便宜 10 倍。而且如果你想把聪明的模型从旁边换到主位、或者反过来，完全没问题，可以来回换。我们现在也花很多时间在想：怎么把模型训练成真正会跟别的模型协作的模型？关于怎么用 RL 把一个模型训到端到端完成任务，文献已经很多了；但怎么用 RL 训一个模型、让它擅长协作？我们两种设置都试了：一种是把正在训练的模型当编排者，由它决定什么活派给别的模型，看效果如何；另一种是让我们训练的模型当执行者、当那个 sidekick，看它执行别的模型的指令执行得怎么样。我们觉得多模型编排的下一步能不能有大提升，关键就在这儿——不是把现成的模型拿来编排一下，而是能不能让你的模型和编排系统协同设计。

[19:53] 主持人

Yeah, that makes sense. I mean, with Nemo Tron and you know, with all the foundational models where um we're essentially post-training them for the harnesses that they're getting used. If the harness is going to include a lot of routing, then that makes sense that makes its way in to the post-training.

对，说得通。我是说，像 Nemotron，还有那些基础模型，本质上都是针对它们将来要跑的 harness 去做 post-training 的。如果 harness 里会包含大量 routing，那这部分自然也该进到 post-training 里去。

[20:05] 主持人

Yeah. Um are you guys thinking a lot about the kind of like the model training at at Nvidia for this kind of purposes?

对。那你们在 NVIDIA 是不是也在为这类目的大量思考模型训练的问题？

[20:12] NVIDIA 代表

Yeah, so we have a technology called

是的，我们有一项技术叫……

[20:16]

The mic's go?

麦没声了？

[20:18]

Right.

对。

[20:18]

Hello.

喂。

[20:20] NVIDIA 代表

Testing. Oh, this one works.

测试一下。哦，这个能用。

[20:22]

[laughter]

（笑声）

[20:23] NVIDIA 代表

Okay. Uh so we have a technology called Flex Run. So you have uh we we have a setup where uh there's a the uh there's a main model, then we distill it into smaller uh footprints. And then based on the based on the task at hand, you can switch which model does the decoding. Right? So there's a lot of fancy stuff you can do uh within a model artifact, too. Uh to essentially only activate a class of model or a section of weights, depending on the task at hand or the complexity at hand. Most In most cases, you can essentially understand the novelty of a question to a model if you have access to the recipe with which it was trained. So, this works very well for open models, right? Like or any model you have access to its data for, right? Because you can literally decide if it's in like see if it's in distribution or not. Uh again, if you have studies from when it was trained, you can also see how much essentially how much was your distillation gap across teachers and the artifact that you trained, right? Because sure, you have domain data from all all the different domains you're tuning, but it's not guaranteed that it uh absorbed all the that data evenly across the models, right? So, um it becomes it becomes very interesting uh to start thinking about these flexible weights and flexible model sizes essentially.

好。我们有一项技术叫 Flex Run。它的做法是：先有一个主模型，然后把它蒸馏成若干个更小尺寸的版本；接着根据手头的任务，切换由哪个模型来做 decoding。而且在单个模型产物内部也能玩不少花活——比如根据任务或复杂度，只激活某一类能力、某一部分权重。大多数情况下，只要你拿得到它的训练配方，就能判断一个问题对这个模型来说有多「新」。所以这套东西对开放模型特别好使，或者任何你能拿到训练数据的模型都行——因为你可以直接判断它在不在分布内。同样，如果你有训练时的各种记录，还能看出蒸馏过程中各个 teacher 模型和你训出来的那个产物之间的差距有多大。因为你虽然喂了各个领域的数据，但并不保证模型对这些数据的吸收在各处是均匀的，对吧？所以，开始思考这种可伸缩的权重、可伸缩的模型尺寸，就变得非常有意思。

[21:56] NVIDIA 代表

There's also I wanted to add with the context piece, right? So, how do you think about ASTs and and context compression representations? Uh compaction in its very nature is lossy, right? So, just like headroom is there, RTK is there, right? Uh these these spaces code bases are usually designed to have representations that we carry forward through life and you essentially give models the capability to further expand on them. It's it's more more like loss less-ish compression, which can retain states of a models or states of agents. What do you think about that?

关于 context 我还想补一点。你们怎么看 AST 这类 context 压缩表示？compaction 本质上就是有损的，对吧？而这些现成的东西本来就在那儿。代码库天然就带着可以一路带下去的结构化表示，你只要给模型在这个基础上继续展开的能力就行。这更接近「近乎无损」的压缩，还能保留模型或 agent 的状态。你怎么看？

[22:37] Walden Yan

Yeah, I think this gets to like kind of like a fundamental philosophy of how agents and context should work. One exercise I like to do is like you know, as a human, like how many numbers can you like if I just start spitting out numbers now, right? Like how many can you remember before, you know, like you start losing track of them. I think it's actually very few, right? So, in some ways you could argue that your context window is actually shorter than these language models. And yet, you can actually be very effective at that, right? Your context is is very lossy. Um, I think one of the nice things that people are starting to realize with agents is like you have a lot of non-lossy systems that you can fall back to. So, you have a file system. Like uh if in your memory all your memory is that you read some file earlier, you don't need to remember the whole file. You maybe remember the important parts, but you can still have the full version of the file on your system. And that's kind of my goal when I'm thinking about how do we build a good context engineered harness is the harness should have everything in needs to find what it needs to have even if it doesn't have everything immediately available.

我觉得这就触到了 agent 和 context 该怎么运作的根本哲学问题。我喜欢做一个小实验：作为人，我现在开始一个个报数字，你能记住几个才开始跟丢？其实非常少，对吧？所以某种意义上你可以说，你的 context window 比这些语言模型还短。可你照样能做得非常好——你的 context 是高度有损的。我觉得大家现在开始意识到 agent 的一个好处是：你有一堆无损的系统可以兜底。比如文件系统。如果你记忆里只留着「我之前读过某个文件」，你不需要记住整个文件，可能只记得关键部分，但完整版本还好好地躺在你的系统里。这也是我在想「怎么做一个 context engineering 做得好的 harness」时的目标：harness 得具备找到它所需内容的一切条件，哪怕这些内容并不都摆在眼前。

[23:35] NVIDIA 代表

In that case, uh do you think that the context sharing problem will become cheaper and less of a problem in future?

这么说的话，你觉得 context 共享这个问题，将来会不会变得更便宜、不再那么棘手？

[23:45] Walden Yan

Yeah, it's definitely possible as well. I've seen cases where um you know, the psychic agent does a bunch of work. It tells the main model, "Oh yeah, like here's here's all the things I found." Instead of dumping the full thing, it just references them by file. And then the main model is actually generally you find these larger, smarter models, they're actually more token efficient with how they use tools and how they read. And so, they actually read the files in a way where they only see the important parts, right? Or they decide that, "Oh, actually I only need to look at a subset of this." Or, "Oh, I can run a single command and just know if everything is done properly." Um, it's actually quite amazing um the fact that you know, these multi-model systems they actually seem to scale and get better with intelligence, which is um not something we should just take for granted, right? It's not obvious that actually more expensive models are actually creating an overall cheaper system.

对，完全有可能。我见过这样的情况：sidekick agent 干了一堆活，然后告诉主模型「我找到的东西都在这儿」——它不把内容整个倒出来，而是只给出文件引用。而主模型——你会发现这些更大、更聪明的模型，在用工具和读东西上其实 token 效率更高——它读文件的方式是只看关键部分，或者判断「其实我只需要看其中一小段」，又或者「我跑一条命令就能确认是不是都弄对了」。挺神奇的：这些多模型系统看起来真的能随着智能水平的提升而变得更好，这可不是理所当然的事——更贵的模型反而造出一个整体更便宜的系统，这一点都不显然。

[24:33] NVIDIA

Yeah, like the scaling laws uh if you have a larger model, it's going to be more efficient with its tokens. Uh smaller models less efficient with its tokens. I guess I had a question for you, Alex. Uh do you guys uh like weigh more importance on uh the actual infrastructure side of routing? So, for instance, KV cache aware routing or uh like this where most of the businesses right now Uh or the are you seeing strong pull off like people actually deploying routers in production?

对，就像 scaling laws 讲的那样——模型越大，用 token 的效率越高；小模型用 token 就没那么高效。我这边想问 Alex 一个问题：你们会不会更看重路由里偏基础设施的那一层？比如 KV cache aware 的路由。也就是说，现在大多数企业……或者换个问法，你们有没有看到很强的实际需求，真的有人把 router 部署到生产环境里？

[25:07] Alex Atallah

So open router is a like a marketplace for language models. We we exist at like we can't see into the KV caches of models unless we're running them ourselves, which is pretty rare. Um we we we do spend a lot of time optimizing for cash hits and we we like pass through cash hits directly to users. Um but like in terms of KV cash optimizations we we we can't do any of like specific work there. What we do for um model routing is we we try to like find the best model or best combination of models for the prompt and then when we see a cash hit, we will will like use up the duration of the cash and send the downstream customer like the the full savings of the cash hit. Um there's more work that we can do here where we could say, okay, this looks like something where there's significant benefit to switching the model right now, but you haven't used up the full cash. You still have like 2 minutes left. Um and it we think it's probably worth switching the model and losing the rest of your cash um and letting people kind of like tweak their tolerance for that behavior. Um we've we've been doing a little bit of that, but we haven't like exposed it to customers yet.

OpenRouter 本质上是一个语言模型的 marketplace。我们所处的位置决定了——除非模型是我们自己在跑（这种情况很少），否则我们看不到模型内部的 KV cache。我们确实花了很多精力去优化 cache 命中，也会把 cache 命中直接透传给用户。但具体到 KV cache 层面的优化，我们做不了什么。我们在 model routing 上做的事情是：为这条 prompt 找到最合适的模型，或者最合适的模型组合；一旦看到 cache 命中，我们就把这段 cache 的有效期用满，并且把命中省下来的钱原封不动地让给下游客户。这里其实还有更多事可以做，比如我们可以判断：现在换模型的收益明显很大，但你的 cache 还没用完，还剩 2 分钟；我们觉得为此丢掉剩下的 cache 也值，那就换——然后让用户自己去调节对这种行为的容忍度。这块我们已经做了一点，但还没开放给客户。

[26:29]

[clears throat]

（清嗓）

[26:30]

Awesome.

很棒。

[26:30]

What what is next for you guys in terms of your your your model routing? Cuz as you mentioned, you know, it is kind of like a different direction from like the the marketplace business that exists today. So I'd love to hear.

你们在 model routing 上下一步打算做什么？因为像你刚说的，这跟今天已经跑起来的 marketplace 业务算是两个方向。挺想听听的。

[26:41] Alex Atallah

So I mean we've been doing like we we've had a an auto router for like 2 years almost. Um but when we launched it, there was like no adoption of it. It was people really wanted to use specific models. Um and the the auto router just had like no real usage. It was it we mostly saw it as like a discovery play point. Like, hey, this is how you discover which model might be good for your prompt. And then um at around like January this year with open claw, it exploded. And the reason it exploded is because there's this fundamental um idiosyncrasy in open claw where it sends heartbeats every like 10 minutes to your model of choice just to see like if the you know, if the the the client was still active.

其实我们做这事儿有一阵了，auto router 差不多两年前就上线了。但刚推出来的时候基本没人用，大家还是想指定某个具体模型，auto router 几乎没有真实用量。我们当时更多把它当成一个「发现」入口——就是让你知道哪个模型可能适合你的 prompt。然后大概今年 1 月，随着 OpenClaw 起来，它一下子爆了。爆的原因是 OpenClaw 有个很特别的机制：它大概每 10 分钟就往你选定的模型发一次心跳，纯粹是为了确认客户端还活着。

[27:31]

[laughter]

（笑）

[27:32] Alex Atallah

And that means that if you set Opus to be your default model, it would be like using a lot of tokens on this like heartbeat process. And so um this was the very beginning of like a very popular app with two completely different intelligence needs, completely different. And the models the open source models have improved to a point where like it makes sense to segment the market and at at least those two areas. And so that's that was how it got started. And then we saw a lot more segmentation blossom afterwards. And so um and now a whole bunch of agents and apps on open router use the the different routers that we have. We have a couple of them. We have Pareto code which gives you the Pareto Pareto optimal model for coding tasks given a certain like threshold that you can tune. Um we have fusion which orchestrates multiple models and gives you a fused result. And um and then we'll have other experiments in the future. What we want to do is basically like create good primitives that developers can use to like get really advanced with how they use model orchestration. Um kind of like sidecar sort of like that. Um but also give people like a really easy thing that they can just like set a slug to that works with all harnesses and just like gets the job done.

这就意味着，如果你把 Opus 设成默认模型，光这个心跳流程就会烧掉大量 token。所以这其实是头一回出现这种情况：一个非常流行的应用，内部却有两种完全不同的智能需求，差别极大。同时开源模型也进步到了一个程度，把市场至少切成这两块是说得通的。事情就是这么开始的。后来我们看到更多细分场景冒出来。现在 OpenRouter 上一大堆 agent 和应用都在用我们的各种 router。我们有好几个：一个是 Pareto Code，在你设定的某个阈值下给你编程任务上帕累托最优的模型，阈值可以自己调；还有 Fusion，它会编排多个模型，给你一个融合后的结果。之后还会有别的实验。我们想做的，本质上是提供一套好的原语，让开发者可以在模型编排上玩得非常高级——有点像 sidecar 那种思路。但同时也要给大家一个特别简单的东西：设一个 slug 就行，所有 harness 都能用，活也照样干完。

[28:52]

I feel like it's super interesting how much of a perfect storm there is for model routing right now because on one hand, you know, ignore agents, ignore open claw for a second, like just to squeeze better performance, it seems like we should be smarter about how we tackle problems. That's obvious, right? If you make a plan, if you make a strategy, that's a that's a better way to to go about your day. So, I'm not surprised that you're going to see, you know, better code get written or more performing code get written and less buggy code get written if you break the problem down. And so, routing specifically for that uh use case makes a ton of sense. But then hearing this, yeah, like the profile of workloads changed with agents, right? They're very they it went from like I ask questions, I get a response, then it went to reasoning where I ask questions, it reasons, and then it comes back, and then it went to uh yeah, this heartbeat, right? Like if um if my agents are running optimally, uh there's a token being generated every second. Um and suddenly that is its own need for model routing, and it feels like um hearing the different solutions uh to kind of tackle each of those is very interesting. Even

我觉得特别有意思的是，model routing 现在简直是天时地利全占了。一方面，先把 agent、把 OpenClaw 放一边，单纯为了榨出更好的性能，我们本来就该更聪明地去拆解问题，这不是明摆着的吗？你先做个计划、定个策略，一天肯定过得更顺。所以我一点也不意外：把问题拆开之后，写出来的代码会更好、性能更高、bug 更少。所以专门为这个场景做路由，非常合理。但另一方面，听你们这么一讲，agent 确实把工作负载的形态改掉了：一开始是我问一句、它答一句；后来变成 reasoning——我问一句，它先推理一会儿再回来；再后来就是这个心跳，对吧？如果我的 agent 跑在最优状态，那基本每秒都在生成 token。这一下子就成了 model routing 的一个新需求。听大家针对这几种情况各自给出的方案，我觉得非常有意思。

[29:53] NVIDIA

Well, I was just going to say I think that yeah, the the use cases for model routing are are there are many of them. And so, one could be getting a better answer, one could be, you know, saving money and trying to get the same answer. One that we haven't even talked about yet, which is probably the most relevant maybe even to this crowd, is uh when do you want to actually run a model locally versus when do you actually need something like a frontier model to to perform that task. And that might be something to the effect of like uh for privacy protecting information. You know, when I'm running some local, it can you detect that my prompt has sensitive information? And if so, do that on on device, um but then maybe even anonymize some of that information to go do the more advanced workloads up on top of that information in the cloud. Um another example would be again for the cost savings, but it's like, "Hey, I bought this DGX Spark and I'm not I know I'm not at 100% utilization. How can I make sure that as part of my workloads, whether it's the heartbeat and open claw or what have you, that I'm leveraging that compute um to the fullest of its ability because I'm only paying for the electrons that are coming in for my power bill, um but I'm paying full price for the tokens in the cloud?" And I think that's a whole other area of model routing that I know that we're doing some work with at NVIDIA um that I think will be will be really cool as it starts to uh the hybrid of local and cloud starts to to really emerge as its own sector.

我正想说，model routing 的用例其实非常多。一个是为了拿到更好的答案，一个是在答案差不多的前提下省钱。还有一个我们今天还没聊到、但可能对在座各位最相关的：什么时候该把模型放在本地跑，什么时候才真的需要 frontier model 来完成这项任务。这里面有隐私保护的成分——比如我本地跑着一个模型，它能不能检测出我的 prompt 里有敏感信息？如果有，就在设备上处理；甚至可以先把这部分信息脱敏，再把更复杂的负载拿到云上去跑。另一个例子还是省钱：「我买了台 DGX Spark，我知道自己肯定没跑到 100% 利用率，那怎么保证我的一部分工作负载——不管是心跳、OpenClaw 还是别的什么——能把这块算力用足？毕竟本地我只付进来的那点电费，而云上的 token 是按全价付的。」我觉得这是 model routing 里完全独立的一块。我知道我们 NVIDIA 内部也在做一些相关的工作，等本地和云的混合真正长成一个独立赛道，这块会非常有看头。

[31:14] NVIDIA

Yeah, I'd be curious to know what you guys take uh take is on. Like if you self-host a model, the cost dynamics change, right? You have a considerably higher cost at a higher context length because your throughput slows down as the context gets deeper. So, rather than switching to a cheaper model, even if you have self-hosted models in data center, you can use compaction uh to bring your throughput back up. Uh have you guys uh thought about this part of it? Like compaction versus uh just routing because one is you have fewer tokens to work with. One is we have cheaper tokens.

对，我挺想听听你们的看法。如果模型是你自己部署的，成本结构就变了，对吧？context 越长成本越高，因为 context 一深，吞吐就掉下去了。所以哪怕你在自己的数据中心里托管模型，与其换一个更便宜的模型，你也可以用 compaction 把吞吐拉回来。你们有没有想过这一层？就是 compaction 和 routing 之间的取舍——一个是让要处理的 token 变少，一个是让 token 变便宜。

[31:50] Walden Yan

Yeah, I I think in practice um compacting alone doesn't solve the the cost or or throughput problems because a lot of times it's just like the uh the differential in like model intelligence and cost is just so big where also by the way, when you compact, you're taking a cache miss. So, you're actually then now like paying 10 times as much for the for those input tokens if you didn't compact. Um the main reason we compact is actually intelligence. Um all all these like multipliers they advertise some like insane context window, like a million tokens. Uh I would like never recommend using like these models past like 200K tokens, under 100K if you can. Um the the the intelligence uh just kind of like falls off a cliff at some point. Um sorry, Anthropic if you're if you're watching. But I I think that uh you know, compaction is like a very useful tool if you are going to have to take a cache miss anyway um one way or another, like when you're routing to another model and you want to just like minimize the the window there.

对，我觉得实际做下来，光靠 compaction 是解决不了成本和吞吐问题的。很多时候模型之间在智能和成本上的差距实在太大了。而且顺带说一句，你一做 compaction 就等于吃了一次 cache miss——那些 input token 你现在要付的钱，是不 compact 时的 10 倍。我们做 compaction，主要原因其实是智能。各家都在宣传自己有多离谱的 context window，动不动一百万 token。我基本不会建议这些模型用到 200K token 以上，能压在 100K 以内最好。到了某个点，智能就是断崖式往下掉。抱歉了 Anthropic，如果你们在看的话。但我确实觉得，如果你反正都要吃一次 cache miss——比如你正要路由到另一个模型、想把窗口压到最小——那 compaction 就是个非常有用的工具。

[32:43] Alex Atallah

Do you find that, um, in the sidecar,

那你们有没有发现，在 sidecar 里……

[32:46]

Yeah.

嗯。

[32:47] Alex Atallah

um, when small models are generating lots of tokens, um, is that like an is that like one of the best reasons to switch it to a larger model? Like, basically, when when small models generate lots of tokens, I wonder if that's like a box of root the root cause of of intelligence problems down the road. You want your big model to generate the big token chunks, the small models to generate smaller token chunks, right?

……当小模型开始大量生成 token 的时候，这是不是切换到大模型的最好信号之一？就是说，小模型狂吐 token 的时候，我怀疑这本身就是后面一连串智能问题的根因。你希望大模型去生成大块的 token，小模型只生成小块的，对吧？

[33:13]

that question, you mentioned like a small model essentially needing to flag that it needs help from the larger model. What is that mechanism? Cuz that seems like the what's the indicator and then what's the mechanism for it to do so?

接着这个问题——你刚才提到，小模型本质上要能主动示意「我需要大模型帮忙」。那这个机制具体是什么？也就是说，判断的信号是什么，它又通过什么机制把这个信号发出来？

[33:25] Walden Yan

Yeah, totally. Uh, so there are a lot of mechanisms we we talk about in our blog post about how uh we just detect that we need to like change the model up. Um, to to uh I guess to answer your question first, how does the the small model detect? Um, actually, the the thing that we spend a lot of time on is, um, how do we make sure the small model is like good at detecting it. Unfortunately, there's a lot of cases where you do need the big model to detect it. Um, one thing that we could don't go into the blog post is, um, you have some kind of cadence on which you're refreshing the cache anyways, cuz by default there's some like 5-minute lifetime on these caches. If you're going to go refresh a cache anyways, you basically can get a free like big like frontier model call uh if you kind of like ask the right question. So, it's at that point where you might say, "Hey, just take a look at what the small model is doing. Does it feel like it's kind of like going into some rabbit hole and and need some help now?" Um,

对，完全可以聊。我们在博客里写了不少机制，讲我们怎么判断该换模型了。先回答你的问题：小模型是怎么检测的？其实我们花了大量时间在「怎么让小模型擅长做这个检测」上。不幸的是，很多情况下你确实需要大模型来做这个判断。有件事博客里没写：你本来就得按某个节奏去刷新 cache，因为这些 cache 默认只有 5 分钟左右的存活时间。既然你反正都要去刷一次 cache，那只要问法得当，你基本上就能白拿一次 frontier model 的调用。就在那个时间点，你可以问一句：「你看一眼小模型现在在干嘛，它是不是钻进某个死胡同了、现在需不需要帮忙？」

[34:12]

What's the need for the 5-minute refresh?

为什么非得每 5 分钟刷新一次？

[34:14] Walden Yan

Uh, it's just like a practical like you you you have to pay some kind of like cost to like keep these like KV cache caches warm. Um, and so most caches just get evicted on some kind of cadence.

就是很现实的原因——要让这些 KV cache 保持热的，是要付出成本的。所以大多数 cache 都会按某个节奏被清掉。

[34:24] NVIDIA

works is like at inference time you only have so many so many cache you can keep kept loaded in the GPU. So, once if a cache is not being used again and again, it's offloaded.

……原理是这样：在 inference 的时候，GPU 里能常驻的 cache 数量是有限的。所以一段 cache 如果没被反复用到，就会被换出去。

[34:35]

I see.

原来如此。

[34:35] NVIDIA

So, it's lost essentially. So, that's why the inference provider asks you for money. Uh but if you self-hosted, you can get around this problem. You can you can make it as long as you want based on your big business logic.

基本上就等于丢了。所以 inference 服务商才会跟你收这笔钱。但如果是你自己部署，这个问题就能绕开——想留多久留多久，完全按你自己的业务逻辑来。

[34:48]

But do do you see a world where we'll have like much more dynamic cache durations rather than just the 5-minute, 1-hour?

那你觉得未来会不会出现动态得多的 cache 时长，而不是只有 5 分钟、1 小时这几档？

[34:57] NVIDIA

It depends on who's do who's deploying the model where, right? So, if you have a GPU which has like a lot of memory uh which like the ratio of let's say SCMs to memory is memory more heavily skewed or if you're working with unified memory uh and you have systems like where I Rubin, uh you you have a lot of tricks to play here, right? Uh the 5-minute uh window is what a lot of providers right now put, but that's uh that's that's more an operational operational operational determination rather than a like a science-based or like a core physics law determination. So, you can technically see over time maybe some uh some APIs are priced differently, uh but uh if you do self-deploy again, you can you can get past a lot of this. The the cost economics really change when you move from self-hosted models to uh API providers uh because you have a lot more control and uh you don't have to guess the shape of your workload. So, let's say if your workload is 32K uh like on average 32K cache, 1K input, 1K output, uh and someone else's like let's say 64K uh 1K 1K, uh if you use some provider, they are amortizing everyone's use uh use case and then giving you a price, right? And they have optimized, quote and quote, for general use. If you self-host, you can optimize specifically for your use, and you'll likely pay much less.

这要看是谁、在哪里部署这个模型。如果你的 GPU 内存很大，比如说 SM 和内存的比例更偏向内存那一侧，或者你用的是统一内存、用的是 Rubin 这类系统，那你能玩的花样就多了。现在很多服务商设的 5 分钟窗口，更像是一个运营层面的决定，而不是什么科学结论或者底层的物理定律。所以从技术上讲，随着时间推移，完全可能出现定价方式不一样的 API。但话说回来，你要是自己部署，这些坎大部分都能迈过去。从 API 服务商转向自托管模型，成本结构是真的会变，因为你控制力强得多，也不用去猜自己负载的形状。比方说你的负载平均是 32K 的 cache、1K input、1K output，而别人是 64K 加 1K、1K；你用某家服务商，他们是把所有人的用法摊平之后再给你一个价格，他们优化的是所谓的「通用场景」。你要是自托管，就能只针对自己的场景做优化，花的钱大概率会少很多。

[36:39] Walden Yan

Yeah. This is like kind of like the level of like, you know, hardware-software frontier that we kind of like think about. When When we started Cognition, we were working on the first agents, I think one reason why no one else worked on agents is they were just extremely expensive. This was before cash tokens was a thing that API providers paid for. Like if you were sending 100,000 tokens and the same 100,000 tokens, you were paying full price for those tokens back in 2024 when we started. One of the key things that let us build Devin and build these first agents was we actually bought direct compute capacity from these providers, and instead of paying on a per-token basis, we just paid for the underlying compute, knowing that the economics of the compute was that we were actually paying far less for for the cash tokens that we'd send over. Um And And then nowadays, you know, that uh you know, there's there's like similar, you know, things I would like about like, you know, having a version of the cache that maybe like you can just back out to like storage in S3 or something and just like hold for much longer.

对，这就是我们一直在琢磨的那种软硬件交界处的问题。我们刚创立 Cognition、做第一批 agent 的时候，我觉得别人不做 agent 的一个原因就是太贵了。那时候 API 服务商还没有 cached token 这回事。2024 年我们刚开始的时候，你哪怕发的是同样的 10 万个 token，也得按全价付。我们能把 Devin 做出来、能把第一批 agent 做出来，关键之一是我们直接从这些服务商那里买了算力，不按 token 计费，而是直接为底层算力付钱——因为算过这笔账我们知道，这样一来那些反复发送的 cache token，我们实际付的钱要少得多。到了现在，也有一些类似的东西是我想要的，比如能有一种 cache，可以直接落到 S3 之类的存储里，保留时间长得多。

[37:34] NVIDIA

Yeah. Now, this is not extremely relevant to a DGX organization like setup, but if anyone's looking to do what you guys have wanted to try out Dynamo. We have a lot of pre-fixed cache optimizations in there.

对。这个跟 DGX 那种规格的部署关系不算大，不过如果有人想做你们做过的这种事，可以试试 Dynamo，我们在里面做了很多 prefix cache 的优化。

[37:48] Walden Yan

Yeah. Um And then going back to your question, Alex, I think you said like, "Oh, like you When a small model is going off and generating a ton of tokens, is that like a interesting time to back off?" Um to be honest, we haven't explored that right yet. So, that that might actually be a very interesting thing to to take a look at. Um It It is weird. I think like some small models do tend to kind of be less like token efficient than others, but um they also seem to be trained on like their own traces, so maybe in a way it ends up like balancing out. A lot of these things I I feel like we have to be like very empirical about to actually know.

对。然后回到你刚才的问题，Alex，你说的是「小模型开始狂吐一大堆 token 的时候，是不是一个该退出来换模型的有意思的时机」。老实说我们还没试过这个方向，所以这可能真的值得去看一看。这事儿挺微妙的：有些小模型确实在 token 效率上不如别的模型，但它们看起来又是拿自己的 trace 训出来的，所以某种程度上可能又抵消掉了。这类事情我觉得都得非常讲经验数据地去测，才能真知道答案。

[38:20] NVIDIA

So, uh just to add on that um um you have a lot of like these days there are a lot of hallucination probes. So, probes that work on either the internal state like internal state of the models directly. So, you can have some form of either magnitude analysis done or linear probes or just uh the end types of probes that you can see and you can essentially rate like how how much you think is is tending towards hallucination. Uh so, that kind of gives you a proxy for how lost it is. Uh like how lost a model is in its thinking. Uh so, you can use like different kinds of probes to understand like the perplexity within a model.

补充一点：现在有很多幻觉 probe（hallucination probe），有些是直接作用在模型的内部状态上的。你可以做某种幅度分析，或者用 linear probe，还有其他各类 probe，基本上都能给出一个评分——判断它现在有多倾向于产生幻觉。这相当于给了你一个代理指标，看模型在自己的思考里迷失到了什么程度。所以你可以用不同类型的 probe 去理解模型内部的困惑度（perplexity）。

[39:05] NVIDIA 代表

Well, that's interesting. So, yeah, instead of using the quantity of tokens that are as indicative of a of a model being lost, it's the it's hallucinating more and you're

这挺有意思的。所以说，判断一个模型是不是已经「迷失方向」，看的不是 token 消耗量这个指标，而是看它是不是开始更频繁地产生幻觉，然后你们……

[39:14]

Yeah, so uh so, essentially what is cache, right? It's It's the prefill It's the prefill stage, right? So, you What is a prefill stage? It's just a vector at the end of the day. So, you can do tune all kinds of classifiers to understand uh different aspects of those collections of vectors. So, with those kind of probes, you can guesstimate a lot of uh states of a model.

对。本质上，cache 是什么？它就是 prefill 阶段的产物。那 prefill 阶段又是什么？说到底就是一堆向量。所以你可以在这些向量集合上训练各种分类器，去识别它们的不同特征。有了这类探针（probe），你就能相当靠谱地推测出模型当下处在什么状态。

[39:41] NVIDIA 代表

I see. One One question I have is um you know, different models behave uh differently and um the kind of means that these prompts aren't portable. So, as you're doing model routing, how do you handle essentially if you're if you're going to a different model architecture, um what do you have to do to the prompts and how much is that a factor into either of you guys' model routing solution?

明白了。我有个问题：不同模型的行为差别很大，这就意味着 prompt 是没法直接迁移的。那你们在做 model routing 的时候是怎么处理这件事的？如果要切到另一种模型架构上，prompt 得做哪些改动？这个因素在你们各自的 model routing 方案里占多大分量？

[40:03]

Like what like how how is the prompt itself a factor in the in the routing?

你是说，prompt 本身在 routing 这件事里是怎么起作用的？

[40:07]

Yeah.

对。

[40:08] Alex Atallah

Yeah, um well, I think with building agents, there are all kinds of paper cuts and edge cases that are domain-specific and like the value of an agent company like the value of Devin is all these like doom loops that you've discovered that are across all industries and the best ways to recover from them and like man here I mean it manifests big time in what the prompts are going to be both for like you know how the the advisor model gets called um uh you know the smart friend

嗯，我觉得做 agent 这件事，会遇到各种细碎的坑和领域特有的边界情况。一家 agent 公司的价值——比如 Devin 的价值——就在于你们摸索出来的那些跨行业的「死循环」场景，以及从里面脱身的最佳办法。这一点在 prompt 上体现得特别明显：比如那个 advisor 模型是怎么被调用的——就是那个「聪明朋友」——

[40:46]

[laughter]

[笑声]

[40:47] Alex Atallah

the how the like subtask agents get called and and the best thing is that like anyone can like like any engineer or any like agent can inspect the traces and like adjust the prompt and then see the like live accuracy long time. So I mean basically I just think that that's part the prompt is part of the the startup building process and is also really easy to observe and like and have and have like multiple people and agents collaborating on them.

——还有 subtask agent 是怎么被调起来的。最棒的一点是，任何人——不管是工程师还是 agent——都能去翻 trace、去改 prompt，然后实时看到准确率随之发生的变化。所以我基本上会说，prompt 本身就是创业公司做产品这个过程的一部分，而且它特别容易观测，也很适合让多个人、多个 agent 一起协作打磨。

[41:16] Walden Yan

Yeah. One thing I'd love to do with our fusion product and we don't have this yet and so this is kind of maybe a preview of some some things we work on is you know you can tune it against a data set but the the real thing you want when you're building a real agent someone uses is just like tune it against what actual people use it for and and what actual models they get routed to. And so there's a lot of signals for this like if if someone sends a prompt and then Devin is working and then you see that the user decides themselves like upgrade to a different model or they decide to downgrade or the system detects that we originally sent to the wrong one we now got to replace. Like that's actually a really useful stream of signals and we're actually getting into this world of like auto research where like maybe we can just have like a constant stream of prompts what it should have been what it was instead and build a system internally that's just capturing all of this and then reiterating on our routing system until it eventually kind of like fits the real production data. Um that's kind of like now that we it's it's it's public and and people are using it, this is now something that we're we're thinking about.

对。我们的 Fusion 产品有件事我特别想做，现在还没做，算是提前剧透一下我们正在搞的东西：你当然可以拿一个数据集去调优，但真正做一个有人在用的 agent 时，你真正想要的是拿真实用户的使用场景、以及他们实际被路由到的模型来调优。这里面其实有大量信号——比如用户发了个 prompt，Devin 开始干活，然后你看到用户自己主动升级到了更强的模型，或者主动降级；再比如系统自己发现最初路由错了、现在得换一个。这其实是一条非常有用的信号流。我们现在也开始进入一种「自动研究」的状态：也许我们可以持续收集这样一条数据流——当时实际路由到了哪个模型、本该路由到哪个模型——在内部搭一套系统把这些全都记录下来，然后不断迭代我们的 routing 系统，直到它最终贴合真实的生产数据。现在产品已经公开、有人在用了，这就成了我们正在琢磨的事情。

[42:15] NVIDIA 代表

Have you guys looked into prompt tuning and do you find it useful like say Japa?

你们研究过 prompt tuning 吗？觉得有用吗？比如像 GEPA 这类的。

[42:20] Walden Yan

Yeah, so there are like these prompt tuning frameworks from like a few years ago that tried to do some kind of like gradient descent type thing. I'm like I'm actually personally less bullish on these kind of like low level mechanical prompt tuning harnesses versus just telling like a smart model like here is the decision that was made and the context figure out why it went wrong. Sometimes you can do something as dumb as asking the model why did you do this instead of this and cite the prompts and then just have your agent your dev and just go and update the prompts, rerun the the test as a regression, make sure make sure it changes. Like it's a lot heavier weight of a system, but I kind of trust the intelligence of a system like that a lot more.

研究过。几年前就出现过一批 prompt tuning 框架，试图用类似梯度下降的方式来做。不过我个人对这种偏底层、偏机械的 prompt 调优工具其实没那么看好。相比之下我更倾向于直接告诉一个聪明的模型：这是当时做出的决策，这是当时的 context，你来分析一下为什么走偏了。有时候可以用很笨的办法——直接问模型「你为什么选了这个而不是那个」，让它把相关的 prompt 片段引出来，然后让你的 agent、你的 Devin 去把 prompt 改掉，再把测试当成回归跑一遍，确认结果真的变了。这套系统当然重得多，但我对这种系统的智能程度信任得多。

[43:03] NVIDIA 代表

So we're we're running out of time, so we're going to wrap up real quick, but I think what's really interesting is just from talking to you guys, we can kind of see how new this space is, right? How much of this is actually just research. We're starting to see new products come in. I'm really excited about your guys' solutions as you guys enter the space. The the ways and the needs that you need routing for, you know, even on a DGX Spark when you're doing local inference, you have more compute and if the memory is filled, one or if it's if the memory utilization is high, one thing you need to do is increase the compute utilization. And so one way you can do that is by spawning multiple agents that are working collaboratively. So that collaborative piece is something that not only is optimal for all of these cloud workloads that you guys are doing, but but specifically that is how you extract more performance out of this edge hardware. And I think um you know, a question here is and maybe to to end on is a router going to be something that we see as a product or is that going to be seen as part of the plumbing here? Are models going to get good at routing to other models because they know they need to be collaborative or or harnesses going to know that they are working across multiple models.

时间快到了，我们抓紧收个尾。我觉得特别有意思的一点是，跟你们几位聊下来，能明显感觉到这个领域有多新——里面有相当大一部分其实还是研究性质的工作，同时新产品也开始冒出来了。你们几家进入这个领域，我对你们的方案非常期待。routing 的用法和需求其实很多：哪怕是在 DGX Spark 上跑本地 inference，你的算力是有富余的，如果显存占用已经很高，你要做的就是把算力利用率提上去。做法之一就是同时起多个 agent 协同工作。所以「协同」这件事，不只是对你们做的这些云端 workload 是最优解，对边缘硬件来说，它恰恰是压榨出更多性能的方式。最后想问一个问题作为收尾：router 最终会成为一个独立的产品，还是会变成底层管道的一部分？模型自己会不会变得擅长把任务路由给别的模型——因为它们知道自己需要协作？还是说 harness 会知道自己正在跨多个模型工作？

[44:11] Walden Yan

I think we already see this. Like you know, at Cognition we're training our models to be able to be good collaborators. I think it's very clear that new frontier models like the Fable models and GPT-5.5 and 5.6 models are like themselves like naturally collaborative and and better at delegation. So I think we're ready there at that point.

我觉得这件事我们已经看到了。在 Cognition，我们就在训练自己的模型，让它成为一个好的协作者。而且很明显，新一代前沿模型——比如 Fable 系列、GPT-5.5 和 5.6——本身就天然更会协作、更擅长把任务委派出去。所以我觉得在这一点上我们已经到位了。

[44:27] NVIDIA 代表

Interesting.

有意思。

[44:28]

Yeah, I I think that the systems are are kind of becoming not muddied in some sense, but I think that ultimately we're understanding that as we step up the the abstraction ladder and build more more things to create this smarter blob, which obviously we should hopefully and we do understand how we're building it and why we're building it, that it's going to it's going to become a system that you look at kind of both the the different components of the system, but it's not just going to be just models. There's not going to There's not going to be a thing as like a really great harness that is in absence of a really great model and vice versa.

对，我觉得这些系统正在变得……倒也不是说变浑浊了，而是说我们最终会意识到：当我们沿着抽象的阶梯往上走、不断往上叠东西，去造出这么一个更聪明的「整体」时——当然我们希望、而且实际上也确实清楚自己在造什么、为什么这么造——它最终会变成一个你既要看整体、也要看各个组件的系统，而且它不会只由模型构成。不会出现那种情况：一个特别出色的 harness，背后却没有一个特别出色的模型；反过来也一样。

[45:03] NVIDIA 代表

Yeah. Makes sense.

嗯，有道理。

[45:05]

I think applications, especially as built on non-deterministic systems like models, operate in a very low-trust environment. So yes, most of the improvements will likely be distributed across both models and the harnesses, but I think overall it's it's it's mostly it's There There will There will have to be some form of controller trying to have some form of arbitration because even from the model perspective you aren't in a perfectly visible world. You don't know the behavior of every model, so it's it's going to be at the orchestration level where you have these kind of things. And this has traditionally been shown by other industries like when web when web launched, you know, you had traffic-based routing. So it's different. But all the sort of routing controls have been centralized over time.

我觉得应用层——尤其是建立在模型这种非确定性系统之上的应用——运行在一个信任度非常低的环境里。所以没错，未来的改进大概率会分摊在模型和 harness 两边，但总体上，最终一定得有某种 controller 来做某种形式的仲裁。因为哪怕站在模型自己的角度，它也不处在一个完全可见的世界里——它并不知道其他每一个模型的行为特性。所以这类事情最终只能落在编排（orchestration）这一层。而且这在别的行业早有先例：当年 Web 刚起来的时候，就有基于流量的路由。形态当然不一样，但所有这类路由控制权，随着时间推移都逐渐集中化了。

[45:59] NVIDIA 代表

Makes sense.

有道理。

[46:01] Alex Atallah

I I it's most likely going to be good news in the future. Um and and I I think like caching is a big reason for that. Even if you I think like a to take the flip side of of this argument, um the you know, it might be that in the future we have like one big model that's like, "I know I am the like most efficient at everything and I'm like way more efficient than Haiku. I'll solve every task better than Haiku can at like a lower price. Um why should I ever delegate to Haiku?" There's something like that actually could could be a model that we have in the future. Um but you're always going to have these like, you know, for example, caching. It could be that like you tell the model that this other model like does have the right context in cash and uh you know, the the orchestrator model just always has more context and the models have to be aligned. So, I I think like it's I I don't really see a world where like we wouldn't be able to get models to collaborate really well and and I think they're going to get better over time. Um in part because they're you know, they just have limited memory. So, I I think that's kind of one one deciding factor and another is that um there will be like uh there will continue to be like if you just look at like the the the rankings on OpenRouter, if you look at our our public data and you look at like the top model being used by dollar spent on classification tasks, well, guess what it is. It's Opus.

我觉得未来大概率是好消息。cache 是其中一个很重要的原因。哪怕我们反过来想这个问题——未来也许会出现这么一个大模型，它会说：「我知道我在所有任务上都是最高效的，我比 Haiku 高效得多，任何任务我都能做得比 Haiku 好、价格还更低，那我凭什么要把活派给 Haiku？」未来真有可能出现这样的模型。但你永远会遇到一些别的因素，比如 cache：可能你会告诉模型，另一个模型的 cache 里已经存着正确的 context 了；而且 orchestrator 模型天然掌握更多 context，模型之间必须保持对齐。所以我基本想象不出一个「我们没法让模型之间很好协作」的世界，而且我觉得这方面只会越来越好。一部分原因是它们的记忆容量终归有限，这算是一个决定性因素。另一个因素是——你去看 OpenRouter 上的排行榜，看我们公开的数据，看在分类任务上按消费金额排第一的模型是谁，猜猜是什么？是 Opus。

[47:35]

[laughter]

[笑声]

[47:35] Alex Atallah

I think there are there are there are big opportunities for like using small models for in distribution easy tasks and the and like as time goes on, that's going to be a larger and larger percentage of tasks relative to like the most valuable tasks that um very smart models spend most of their time on.

我觉得用小模型去处理那些「分布内」的简单任务，机会非常大。而且随着时间推移，这类任务在总量里的占比会越来越高——相对于那些最有价值、要靠非常聪明的模型花掉大部分时间去啃的任务而言。

[47:54] NVIDIA 代表

Totally. Well, I want to thank you guys so much. Can we all give everyone a round of applause? Thank you.

完全同意。非常感谢几位今天的分享！大家一起给他们一点掌声好吗？谢谢！

[47:59]

[applause] [music] [music]

[掌声] [音乐] [音乐]