Dashboards Are Dead — Sarah Simionescu, Composio

频道: AI Engineer
视频: https://www.youtube.com/watch?v=YiFqcu9YA38
原文语言: en
统计: 共 8 轮 · Sarah 7

[0:12] Sarah

Hello everyone. My name is Sarah and I have a confession to make. So, I've been using Data Dog every day for the last six months. And the other day my co-orker comes up to me and he's like, "Hey Sarah, can I look at this alert?" I'm like, "Okay." So, I open up Data Dog and I'm logged out. It's kind of awkward, so I have to log in and he's like looking over my shoulder and I'm like overly conscious about everything that I'm doing on my computer. And I open up the data the data dog dashboard and I just froze. I had no idea where anything was. And this is actually not a knock on data dog. The problem was I had genuinely never opened the dashboard. I've been using it every day without ever looking at it. And so today, I want to make an argument that sounds kind of insane and then show you that it's obvious. The dashboard is dead. And I think that's great news for everyone in this room except probably for me because I am my team is responsible for the Composio dashboard. Alas, how did we get here? I believe a postmortem is in order. Shall we? The year is 2022. It is the dark ages. Someone pings me on Slack. They say, "Hey, I get this error when I try and search for prod." I would have instantly spawned five different windows. Read the context in Slack. Write a query in data dog. Check post for this session. Uh, fix the bug in VS Code. Open the PR in GitHub. That is five tools and five UIs I have to learn and relearn every time they ship a new redesign. But frankly, redesigns are the least of my problems because every single tool has its own query language.

大家好，我叫 Sarah，我得先坦白一件事。过去六个月，我每天都在用 Datadog。结果前几天，同事走过来跟我说：「嘿 Sarah，能帮我看一下这条告警吗？」我说：「行啊。」于是我打开 Datadog，发现自己是登出状态。场面有点尴尬，我只好重新登录，他就站在我身后盯着屏幕看，搞得我对自己在电脑上的每一个动作都格外不自在。等我把 Datadog 的 dashboard（仪表盘）打开，整个人直接僵住了——我完全不知道东西都在哪儿。这其实不是在吐槽 Datadog。问题在于，我是真的从来没打开过这个 dashboard。我天天都在用它，却一眼都没看过它。所以今天，我想提一个听上去有点疯狂的论点，然后让你们看到它其实显而易见：仪表盘已死。而且我觉得，这对在座的每个人都是天大的好消息——大概只有我除外，因为我……我们团队负责的正是 Composio 的 dashboard。唉，我们是怎么走到这一步的？我觉得有必要做一次复盘（postmortem）。我们开始吧？时间是 2022 年，那是黑暗时代。有人在 Slack 上 ping 我，说：「嘿，我搜 prod 的时候会报这个错。」换作那时候，我会立刻开出五个不同的窗口：在 Slack 里读上下文，在 Datadog 里写查询，去 PostHog 查这次 session，呃，在 VS Code 里修 bug，在 GitHub 上开 PR。这就是五个工具、五套 UI，我得一个个学会，而且每次它们上线一版新设计，还得再重学一遍。不过说实话，改版还算是我最小的麻烦，因为每一个工具都有自己的一套查询语言。

[1:57] Sarah

So, data dog has its own query syntax. Jira has JQL, search, Slack has its search modifiers. And it gets worse because even when these tools claim to speak the same language, let's say SQL, they don't even agree on SQL. Every dashboard you've ever used, every weird obscure query language ever written was a translation device between you and your data because the machines on the other end could not understand what you actually wanted. You never wanted a dashboard or it's cursed query language. You wanted the answer. And in 2023, everything changed. The sparkle button. The sparkle button was born. And with a click of this magical button, an LLM would write a sometimes correct query to get you the answer you need. That is if your question does not require more than two database joins. And soon enough, these sparkling buttons were everywhere. So dashboards became AI native. Problem solved, right? Well, not quite. In November of 2024, Enthropic announced the MCP protocol with a promise to create an open standard for connecting AI systems with its data sources and that it did. Instead of using a sparkle button, your own agent, Claude, whom you've already been using every day, could generate the query for you, execute it on your behalf, and give you the answer directly. And you might be thinking, problem solved, right? This is the end of the story. Well, far from it, because MCP is a protocol. It's a channel for communication between agents and your service. And it's up to you, the service, to choose how to communicate with the agent. And it turns out that makes all the difference. If you've ever actually wired up a dozen MCP servers, you will know the reality is a mess for three reasons. Agents don't learn. Every conversation starts from zero. It has no memory of how it fumbled formatting links properly in Slack yesterday, and so it's just going to fumble again today. And we patched this with skills,

Datadog 有它自己的查询语法，Jira 有 JQL 搜索，Slack 有它的搜索修饰符。更糟的是，就算这些工具号称讲的是同一种语言，比如 SQL，它们连 SQL 都谈不拢。你用过的每一个 dashboard，人类写出来的每一种古怪冷门的查询语言，都是夹在你和你的数据之间的一台翻译装置，因为另一头的机器听不懂你真正想要的是什么。你从来就不想要 dashboard，也不想要它那套该死的查询语言。你要的是答案。到了 2023 年，一切都变了。sparkle button。sparkle button——那个 AI 闪光按钮——诞生了。点一下这个神奇的按钮，LLM 就会帮你写出一条有时候正确的查询，把你要的答案拿回来。前提是，你的问题不需要两个以上的数据库 join。没过多久，这些闪闪发光的按钮就遍地都是了。于是 dashboard 变成了 AI 原生。问题解决了，对吧？嗯，还差点意思。2024 年 11 月，Anthropic 发布了 MCP 协议，承诺要建立一个开放标准，把 AI 系统和它的数据源连接起来——它也确实做到了。不用再点 sparkle button 了，你自己的 agent，也就是你本来就天天在用的 Claude，可以替你生成查询、代你执行，再把答案直接交给你。你可能会想：问题解决了，对吧？故事到这儿就结束了。嗯，还差得远呢，因为 MCP 是一个协议。它是 agent 和你的服务之间的一条通信通道。至于怎么跟 agent 沟通，得由你——也就是服务这一方——来决定。而事实证明，差别全在这里。如果你真的动手接过十来个 MCP server，就会知道现实是一团乱，原因有三个。agent 不会学习。每次对话都从零开始。它不记得昨天自己在 Slack 里是怎么把链接格式搞砸的，所以今天照样会再搞砸一次。我们用 skill 给它打了补丁，

[4:03] Sarah

but skills is just a band-aid because agents become dumber with the more context you provide. And loading more tools, more skills takes more context. And if you connect enough servers, you're dumping thousands of tool definitions straight into the context window. I mean, our GitHub like toolkit alone has over 200 tools. The model just drowns. It grabs the wrong tool or struggles to resolve dependencies and it can't figure out which one it needs to call first. And third, every app is isolated. Each MCP server knows about itself and nothing else. So the moment a task spans two apps and like they always do, it is your job to piece that together. And so MCP gave agents a door into every app. But it left them standing in thousands of separate rooms with no map and no memory of ever being there. I'm going to show you a mockup demo here of how I would solve this bug report uh from the very beginning how I would solve it in 2026. So imagine in this mockup I have my claude connected to the composio MCP. What I would do is I would literally just copy and paste the link to the message from Slack and say please use century data dog find the root cause a draft PR make no mistakes. And the first thing it does is it calls composio search to state the task it wants to accomplish. So in this case it stated three. It wants to fetch slack messages.

但 skill 只是一块创可贴，因为你给的上下文越多，agent 就越笨。而加载更多工具、更多 skill，都要占用更多上下文。如果你接的 server 足够多，就等于把几千条工具定义直接倒进上下文窗口。我是说，光是我们的 GitHub 这一个 toolkit，就有 200 多个工具。模型直接被淹没了。它会抓错工具，或者理不清依赖关系，搞不明白该先调用哪一个。第三，每个 app 都是孤立的。每个 MCP server 只了解它自己，对别的一无所知。所以一旦某个任务横跨两个 app——而任务总是这样的——把它们拼起来就成了你的活儿。所以说，MCP 给了 agent 一扇通往每个 app 的门，却把它们丢在几千个互不相通的房间里，没有地图，也不记得自己曾经来过。接下来我给大家看一个 mockup 演示，看我会怎么解决这条 bug 报告，呃，从头开始，看看在 2026 年我会怎么解决它。假设在这个 mockup 里，我的 Claude 已经连上了 Composio 的 MCP。我要做的，真的就只是把 Slack 那条消息的链接复制粘贴过来，然后说：请用 Sentry 和 Datadog 找出根因，起草一个 PR，不许出错。它做的第一件事，是调用 Composio Search，说明它想完成的任务。这一次它列了三项。它要拉取 Slack 消息。

[5:28] Sarah

It wants to search sentry issues. It wants to search data dog logs. And for each of these three, Composeio returns not only the correct tools it needs, but actually a plan of how to use them. So for example, for Slack, you actually have to find the Slack channel ID before you can start quering for messages. Now that it's armed with all the context it needs, it has a plan. It starts pulling the message from Slack to see what the user complaint was. Then it begins pulling from data sources in parallel. Data dog sentry and it begins scanning the codebase. Once it identifies the root cause, PR is up with a fix in less than five minutes. I didn't have to build a workflow or write a skill to teach it how to do this. This is literally just claude using Composeio's MCP. And it has become so much easier to do things this way that opening up Cloud has just become like muscle memory for me to do literally anything. And the dashboards I used to open every day are becoming increasingly unfamiliar to me. And the impact of designing for agents is measurable. So these are some early unreleased uh results comparing Composio against each app's own native MCP that's listed in the clawed uh marketplace. So it's the same tasks, it's the same model, and we see a clear difference.

它要搜索 Sentry 的 issue。它要搜索 Datadog 的日志。针对这三项中的每一项，Composio 返回的不只是它需要的正确工具，还有一份怎么用这些工具的计划。比如说 Slack，你其实得先找到 Slack 频道的 ID，才能开始查询消息。现在它手里有了所需的全部上下文，也有了计划。它先把那条消息从 Slack 拉出来，看看用户到底在抱怨什么。接着它开始并行地从各个数据源拉数据——Datadog、Sentry——同时开始扫描代码库。一旦定位到根因，带着修复的 PR 就提上去了，全程不到五分钟。我不用搭什么工作流，也不用写一个 skill 去教它怎么做。这真的就只是 Claude 在用 Composio 的 MCP。用这种方式做事已经变得太轻松了，以至于不管要干什么，打开 Claude 都成了我的肌肉记忆。而那些我以前每天都要打开的 dashboard，对我来说变得越来越陌生。为 agent 做设计带来的影响，是可以量化的。这是一些早期的、还没对外发布的，呃，结果，拿 Composio 和每个 app 自己的原生 MCP——也就是 Claude，呃，marketplace 里上架的那些——做了对比。同样的任务，同样的模型，我们看到了明显的差距。

[6:44] Sarah

But why? Why is this experience so much better than native MCPs? It's because at Composeio, we are developing a brand new interface specifically designed for agents. We translate messy, sparsely documented, everchanging APIs from all of the apps that you live in every day into something that agents really love to use. So, we build on top of MCP CLI and native tools to deliver a cohesive, unified, and excellent experience for your agents that allows them to perform complex operations across these apps seamlessly. I'm going to show you a more classic example of like what I usually use it for every day. So, let's say I want to dig into some user data. I want to see the distribution of what vertical my users select during onboarding. I can simply just ask Claude. Claude will again run Composio search. It will write a query in Post Hog, execute it nice and simple. There's my results. But what if I wanted to take this up a notch? So I want to look at that e-commerce section right there. And I want to see what tool kits they like to use. Toolkits is another word for apps we like to use at Composio. So let's see if cloud can figure out how it can pull this data from metabase uh given the user ids in post hog. So once again it uses composio search.

但为什么呢？为什么这种体验比原生 MCP 好这么多？因为在 Composio，我们正在开发一套全新的、专门为 agent 设计的接口。你每天都泡在里面的那些 app，它们的 API 杂乱、文档稀缺、还一直在变——我们把这些 API 翻译成 agent 真正爱用的东西。所以，我们在 MCP、CLI 和原生工具之上构建，为你的 agent 提供一套连贯、统一、而且出色的体验，让它们能顺畅地跨这些 app 完成复杂操作。我再给大家看一个更经典的例子，差不多就是我平时每天拿它来干的事。比方说我想深挖一些用户数据，想看看用户在 onboarding 时选的垂直行业是怎么分布的。我直接问 Claude 就行。Claude 还是会先跑 Composio Search，然后在 PostHog 里写一条查询并执行，干净利落。我的结果出来了。但如果我想再加点难度呢？我想看看那边电商的那一块，想知道他们喜欢用哪些 toolkit。toolkit 是我们在 Composio 喜欢用的、对 app 的另一种叫法。那我们就来看看，Claude 能不能想明白，怎么拿着 PostHog 里的用户 ID，呃，去 Metabase 把这份数据拉出来。它又一次用上了 Composio Search。

[8:06] Sarah

It queries the user ids of those who selected e-commerce from post hog and it actually saves it without loading the full results into its context. Then it makes some queries uh in metabase to get the database schema sample some data get a feel for how things are structured. And once it's confident, it actually uses a very cool tool called Composio Remote Workbench to dynamically generate an SQL query with a reax string to search for all those user IDs again without ever loading the entire thing into its context window. And there we go. So we can see the most popular toolkits amongst this user persona in less than a few minutes. Uh and that's data pulled from both sources. If you want to play with the live version, come visit us at the Composio booth. Uh, I'd love to have you play with this live. And so, the dashboard has died. And I'm thinking more and more about what this means for the future. Many startups have begun making their landing pages and their websites AI friendly for GEO purposes. But so few have really prepared their applications to be used by agents. Compose began as a tool to help developers. We were taking these popular apps that users like and turn them into these tools that agents could use.

它从 PostHog 里查出选了电商的那些用户的 ID，而且它是直接把结果存起来，没有把完整结果加载进自己的上下文。然后它，呃，在 Metabase 里跑了几条查询，拿到数据库 schema，抽样看一些数据，摸清楚结构大概是什么样。等它有把握了，它就用上一个非常酷的工具，叫 Composio Remote Workbench，动态生成一条带 regex 字符串的 SQL 查询，把所有这些用户 ID 都搜出来——同样，全程没有把整份数据加载进它的上下文窗口。好了，出来了。不到几分钟，我们就能看到这类用户画像里最受欢迎的 toolkit。呃，而且这是从两个数据源一起拉出来的数据。如果你想玩玩真实版本，欢迎来 Composio 的展位找我们。呃，我很乐意让你现场上手试试。所以，dashboard 已经死了。我也越来越多地在想，这对未来意味着什么。很多创业公司已经开始为了 GEO（生成式引擎优化），把自己的落地页和网站做得对 AI 友好。但真正把自己的应用准备好、能让 agent 来用的，少之又少。Composio 最初是一个帮助开发者的工具。我们把用户喜欢的这些热门 app，变成 agent 可以使用的工具。

[9:28] Sarah

But the humans are tired of dashboards and their agents are tired of poorly designed MCP servers. And now we're getting requests from startups saying their clients are begging for a way to use their services through their agents. So this is a lesson for everyone building anything right now. You are now serving a new species of user. They don't have eyes. They are not going to click your sparkle button. It shows up with a goal and a set of tools and it judges you on exactly one thing, whether it can get the job done. For a decade, we've built pro uh we've built products to be easy for humans to use. And the next era belongs to those that are easy for agents to use. So, picture me again. I'm frozen in front of that data dog dashboard. And I was thinking to myself in that moment that I'm falling behind. Uh, but now I think of that moment as a preview of what's to come. Thank you. [music]

但人类已经受够了 dashboard，他们的 agent 也受够了设计糟糕的 MCP server。现在我们开始收到创业公司的请求，说他们的客户在求着要一种办法，能通过自己的 agent 来使用他们的服务。所以，这是给眼下正在做任何产品的每一个人的一课：你们现在服务的是一个全新物种的用户。它们没有眼睛。它们不会去点你的 sparkle button。它带着一个目标和一套工具出现，评判你的标准只有一条：它能不能把事情办成。十年来，我们做产……呃，我们做产品，都是为了让人类用起来容易。而下一个时代，属于那些让 agent 用起来容易的产品。所以，再想象一下那个我：僵在那个 Datadog 的 dashboard 前面。那一刻我心里想的是，我要落伍了。呃，但现在我把那一刻看作一次预告，预示着即将到来的一切。谢谢大家。（音乐）

[10:37]

[music]

（音乐）