The End of the Static Screen: Architecting Intent-Driven UX — Gus Iwanaga, commercetools
频道: AI Engineer
视频: https://www.youtube.com/watch?v=QrMcNe2jjt8
原文语言: en
统计: 共 21 轮 · Gus 19
[0:12] Gus
How's it going? The end of uh the end of the conference, how's everybody feeling? Tired, drinking from the fire hose as well? Are you guys a little bit tired of hearing loop engineering, harness engineering, software factory, evolves, pre-training, post-training data, and whatnot? Uh but anyways, those topics were more than valid, right? Uh hi. Some uh really uh common faces here. By the way, I'm super excited to be here and talk to you guys about this title. And And I'm sorry, when I was when I submitted the application, I was thinking of potentially a catchy title, but I don't like this at all. So, with all due respect to the organizers, I'll have to make a change. Uh and the actual topic that I want to focus here today is lessons learned uh from a team that is building proper generative uh UX and UI. And I was going to uh touch on agentic orchestration, but come on. Uh over the last 3 days, this is what we heard all the time. So, I'd rather focus on what I didn't hear enough about here in the conference, and hopefully you walk away, if not with something very tangible, but with a new mental model that can spark meaningful discussions down the road.
大家好吗？到大会尾声了，大家感觉怎么样？累了吧，是不是也被海量信息灌得有点懵？loop engineering、harness engineering、software factory、evals、预训练、后训练数据……这些词大家是不是都有点听腻了？不过话说回来，这些话题当然都非常值得聊，对吧？嗨，台下有不少熟面孔。说真的，我特别高兴能来这儿，跟大家聊这个题目。还有，得先说声抱歉：当初提交申请的时候，我想着起个吸睛一点的标题，但现在我一点也不喜欢它。所以，对主办方绝无不敬之意，但这个标题我得改一改。我今天真正想聚焦的主题，是一个在打造真正意义上的生成式 UX 和 UI 的团队总结出来的经验教训。我本来还打算讲讲 agentic 编排，但拜托，过去 3 天我们一直在听这个。所以我更想聊聊这次大会上讲得还不够多的东西。希望大家离开的时候，就算没带走什么特别具体的东西，至少也能带走一个新的心智模型，引发后续一些有意义的讨论。
[1:29] Gus
Sound good? All right. Very good. So, hi everybody. Uh I'm Gus. I'm a general manager at commercetools. I lead product, UX, and engineering for zero-to-one products. I tell people I'm in a very privileged position because we get to build really cool stuff. So, we cook really interesting stuff at commercetools. Fantastic. And this is what you guys can expect uh at least during the presentation. Uh I'd like to make sure that we're on the same page with respect to the problem uh the problem space uh followed by a quick demo of the product because I'm not sure if you guys agree, an image speaks more than 1,000 words. So, it would just uh make it more tangible for uh everybody here. Followed by a rapid discussion on the emergence of UI protocols, and I'm not sure if you guys joined maybe some of the talks here. Even the founder of some of these protocols were here uh this week, and that was really cool. Followed by and last but not least, the challenges that uh my team uh and I faced and we still face. And some of the mitigation tactics uh that we put in place to overcome some of these challenges.
听起来不错吧？好，很好。那么，大家好，我是 Gus，commercetools 的总经理，负责从 0 到 1 新产品的产品、UX 和工程。我常跟人说，我的位置特别幸运，因为我们能做很酷的东西——我们在 commercetools 捣鼓的东西都特别有意思。太棒了。下面是这次分享大致会讲的内容。首先，我想确保大家对问题、也就是问题空间有一致的理解；接着是一段简短的产品 demo，因为不知道大家是否同意，一图胜千言，这样能让在座各位看得更直观。然后快速聊一聊 UI 协议的兴起——不知道大家有没有去听这里的相关演讲，这周连其中几个协议的创始人都来了，真的很酷。最后但同样重要的，是我和团队遇到过、至今仍在面对的挑战，以及我们为克服这些挑战而采取的一些缓解手段。
[2:33] Gus
So, with that let's continue. The problem space. This is This is more like a a a statement um and we're still adapt to the software that we ship, not the other way around. Right? And then uh although you could argue when GPT came out, this was November uh 2022, we had a really good glimpse of uh real personalization, but everything else uh remained static. And even with AI, we keep shipping a lot of stuff much faster, but to a significant extent, it is still static. And my question is, why? So, for the uh over the last 40 years, uh we kept shipping a static experiences. And if I put it myself in the shoes of uh some of uh my customers, they need several SaaS applications for uh the day-to-day work, and each with its own mental model and its own way to get anything done. And over time, it just kept getting worse, just accruing uh debt. The cognitive load that we wanted to remove and that we wanted to transfer to the machine, uh it's on us.
好，那我们继续。先说问题空间。这更像是一个判断：我们至今仍在迁就自己交付的软件，而不是让软件来适应我们，对吧？当然你可以说，2022 年 11 月 GPT 刚出来的时候，我们确实很好地瞥见了真正的个性化是什么样子，但除此之外的一切依然是静态的。即便有了 AI，我们交付东西快了很多，但在很大程度上，交付出来的仍然是静态的。我的问题是：为什么？过去 40 年里，我们一直在交付静态体验。如果我设身处地站在一些客户的角度想，他们日常工作要用好几个 SaaS 应用，每个都有自己的一套心智模型、自己的一套办事方式。时间一长，情况只会越来越糟，债越积越多。我们本想消除、本想转交给机器承担的认知负担，结果还是压在我们自己身上。
[3:47] Gus
And now with AI, things can be different. I don't know if you guys agree, but this is uh what we what we think. And I do have a couple of examples. I'm not going to say out loud the name of these apps, but let's take a look. Here. First one. You know, that icon is very well known. This is probably the most famous CRM of all times. But when I look at this screen, there's a lot going on. I don't even know where to start. Right? Not only the information overload, how many features how many teams do you think are somewhat involved just to ship this? Many, probably. Right? And this is just one. Let's have a look. I have a couple of other examples. So, this here. Look at that. Oh my god, this is a fancy table. I don't even know where to start. But anyways, one more. Does everybody know this one here? Beautiful. Beautiful UI. Right? It's a fantastic, super intuitive, and something else just to highlight.
而现在有了 AI，事情可以不一样了。不知道大家同不同意，但这就是我们的看法。我准备了几个例子。我不会说出这些应用的名字，但我们来看看。这里，第一个。那个图标大家都很熟悉，这大概是有史以来最有名的 CRM 了。但我看这个界面，上面的东西太多了，我都不知道从哪儿看起，对吧？不光是信息过载——你们觉得，光是为了上线这一个界面，要牵扯到多少功能、多少个团队？估计很多吧，对吧？而这还只是一个。我们接着看，我还有几个例子。比如这个。看看。天哪，好一张“花哨”的表格，我都不知道从哪儿下手。算了，再来一个。这个大家都认识吧？漂亮，多漂亮的 UI，对吧？简直出色、超级直观——另外还有一点我想特别强调一下。
[4:52] Gus
Do you guys know how much time these companies need to invest in onboarding people? So, that was the the trade-off. Right? So, you got to allocate a lot of time for a lot of people just to onboard newcomers. As a result of this complexity that has been introduced over time. So, this is just at least the hardcore evidence. So, different apps, different logic every single time. And then more apps are coming out. So, imagine just put yourself in the shoes of average user, and then oh, now I have five five apps. And then every single one I need to learn how to navigate, how to browse, and so on and so forth. And then this has been the history up until now. But then this was August last year, I sat down with my boss. It happens to be the founder of the company, so big shout out to my boss. And then we asked this uh question because at Commerce Tools, we are an API first company, 300 plus still counting. And we ask this posing question, through the lens of artificial intelligence, what are the foundational shifts that could be made if we could change drastically the way that we interact with software?
你们知道这些公司得在新人上手培训上投入多少时间吗？这就是代价，对吧？光是为了带新人上手，你就得调配很多人、花大量时间。这都是长期累积起来的复杂度造成的。这至少算是实打实的证据：不同的应用，每次都是一套不同的逻辑。而且新的应用还在不断冒出来。想象一下，设身处地站在普通用户的角度：噢，我现在有五个应用了，每一个都得去学怎么导航、怎么浏览，诸如此类。一直到现在，历史都是这么过来的。但就在去年 8 月，我跟老板坐下来聊——他恰好就是公司的创始人，在这儿特别感谢一下我老板。我们提出了这样一个问题，因为 commercetools 是一家 API 优先的公司，API 有 300 多个，而且还在增加。我们抛出的问题是：从人工智能的视角来看，如果我们能彻底改变人与软件的交互方式，可以带来哪些根本性的转变？
[6:03] Gus
Not in a uh static uh fashion. And the answer to that question led to the product that I'm going to demo right now. Let's go. How about quick demo? Guys like the idea? Give me a thumbs up. Bye. I know everybody's tired. Let's go. Yeah. All right. Cool. Look at this. I know, I'm going to zoom in. No worries. So, I have this query here. Create a sales report for Q1. And the UX side of me, when I look at uh what was generated, and by the way, everything here on the right side has been auto-generated, guided by us, but this is AI, right? Deciding on the placement, on the information architecture, deciding which uh components had to be actually retrieved from the catalog, but I don't like it at all. Right? It's a Even if you don't know a lot uh about UX, just let's, you know, I have at least four different variations because those were four different terms for the same query. Let's check it out. First one here, it was Q1 uh Urban Thread, whatever that is. Uh Q1, I see all of these KPI cards.
而不是用静态的方式。这个问题的答案，最终催生了我接下来要演示的产品。开始吧。来个快速 demo 怎么样？大家觉得行吗？行的话竖个大拇指。哎呀，我知道大家都累了。来吧。好。行，不错。看这个。我知道，我放大一下，别担心。我这里有一个 query：“生成一份 Q1 销售报告”。我身上做 UX 的那一面，看到生成出来的东西——顺便说一句，右边这一整块全是自动生成的，由我们引导，但这是 AI 在做决定，对吧？决定怎么摆放、信息架构怎么安排、要从组件目录里实际调取哪些组件——可我一点都不喜欢。对吧？这……就算你不太懂 UX，也看得出来。我这里至少有四个不同的版本，因为同一个 query 跑了四轮。我们来看看。第一个是 Q1 的 Urban Thread，管它是什么呢。Q1，我看到一大堆 KPI 卡片。
[7:12] Gus
There's a lot going on here. And then my intuition tells me, man, this doesn't add up. Okay. Second one, it's not Q1 anymore. Now this is January and March. There's no consistency. Does that help? Yes or no? No, right? This is This is You only create confusion. If this is a heavy personalized experience for the user, imagine if every single time you need to prompt, and then uh at least the model will output something different. This is not good, and this is just the second turn. Let's have a look at the third one. Oh my god, now there is even more stuff here on the right side. But so, this is I have this component, the KPI cards, a bunch of text, a bunch of charts, and then this this was the beginning of a journey. One more? Yes. Okay, now it's still Q1, but still for me this still it's confusing. And because this has been a very experimental journey, at least okay, let's let's move on here.
这里的东西太多了。我的直觉告诉我：天哪，这根本对不上。好，第二个，已经不是 Q1 了，变成了 1 月和 3 月。完全没有一致性。这有帮助吗？有还是没有？没有，对吧？这只会制造困惑。如果这是一个为用户深度个性化的体验，想象一下，你每次都得写 prompt，而模型每次都会吐出点不一样的东西。这可不行，而这才只是第二轮。来看第三个。天哪，现在右边的东西更多了。所以你看，这里有 KPI 卡片这个组件、一堆文字、一堆图表——而这只是一段旅程的开端。再看一个？好。行，这次还是 Q1，但对我来说还是很乱。因为这一路走来都非常实验性，至少……好，我们继续往下。
[8:18] Gus
My feedback to the team and to myself was no, no, and no. There's no way that I would ship this to to prod whatsoever, right? And then I have my colleagues here just to confirm what I just said. But then things evolved, and I'll like to demo the current state of the product. It's much more sophisticated, and let's have a look. I I'll like to plan a campaign, and for what it's worth, I'm going to save you from all the nitty-gritty details for everything that is domain specific, but I'm going to pick this query here, and then let's see what happens. And this is the the agentic orchestration part that I was going to highlight. Underneath the hood we have the orchestrator, and the orchestrator can then just extract the intent of the query. Based off of the intent of the query, it can locate the tools, right? Those can be first party, third party tools, and the outputs of this different it could be agents on MCP servers, combined will give enough what I call ammunition and context for the UX agent to eventually render something that we call meaningful. So, compared to the previous turns, this is this is decent, right? I want to just remove my bias, but the overall aesthetics, the look and feel of this query, uh it it resonates with me. Would you agree? Give me a thumbs up if you like if you agree. Okay, at least um the vast majority here. And then see, it is decent. And let me just continue here.
我给团队、也给我自己的反馈就是：不行，不行，还是不行。这种东西我无论如何都不可能发到生产环境，对吧？我的同事们就在现场，可以作证。不过后来事情有了进展，我想演示一下产品现在的样子。它成熟多了，我们来看看。我想策划一场营销活动——说实话，所有跟具体领域相关的细枝末节我就不展开了，我直接选这个 query，看看会发生什么。这就是我本来想重点讲的 agentic 编排部分。底层有一个 orchestrator，它会先提取 query 的意图；基于意图，它能找到对应的工具，对吧？可以是第一方工具，也可以是第三方工具；这些不同来源的输出——可能来自 agent，也可能来自 MCP server——合在一起，就能给 UX agent 提供足够的、我称之为“弹药”的东西和上下文，让它最终渲染出我们所说的“有意义”的东西。所以跟前面几轮比，这个还算像样，对吧？我想尽量抛开自己的偏见，但这个 query 出来的整体美感和观感，确实挺打动我的。你们同意吗？同意的话竖个大拇指。好，至少绝大多数人同意。你看，还算像样。我接着往下演示。
[9:48] Gus
And then once again, right? This was decided by AI guided by us. I would just want to make make that clear here. And then I'm going to touch on the on the UI protocols and then how you can make this happen. But okay, let's see. If I approve here, and then this is already live. And this is pre-prod, right? So, uh this is great. Let me go back to my presentation. Perfect. I have one more question. Are you guys as skeptical that this is possible? Because I can tell you this it is possible. If you're still skeptical, don't worry. I have all of these guys uh here uh also uh every day just looking at me and then challenging whether uh this uh can be made possible at scale, right? And then uh okay. And then this is the part that I'll like just to touch uh touch base on the uh on the three ways that you can render what you just saw. And they're different UI protocols. Are you guys familiar with uh generative uh UI? Have you guys played with it? Let me see here.
再强调一次，对吧？这是 AI 在我们的引导下做出的决定，这一点我想说清楚。接下来我会讲讲 UI 协议，以及怎么把这件事做出来。好，我们看看。如果我在这里点批准，它就已经上线了。当然这是预发环境（pre-prod），对吧？所以，挺好。我切回演示文稿。完美。我再问一个问题：你们是不是也在怀疑这事到底能不能做成？因为我可以告诉你们，这是做得到的。如果你还是怀疑，也没关系——我这几位同事每天都盯着我，质疑这东西到底能不能规模化落地，对吧？好。接下来这部分，我想简单讲讲把你们刚才看到的东西渲染出来的三种方式，它们分别对应不同的 UI 协议。大家熟悉生成式 UI 吗？有人上手玩过吗？我看看。
[10:47] Gus
Okay, well, that's really cool. Uh okay. So, uh what I would like just to share with you guys it's all about how much control you want to exercise over the experience. And this matters a lot because uh you know, as a non-deterministic solution, uh you can decide if you want something I really um like this. So, let's have a look. Here, this is uh ChatGPT. And my query was help me find a Japanese restaurant uh in SF today. If you guys see here, this component this component is very opinionated. Would you agree with that? Here? Right? So, uh you can have complete control over this uh component. So, depending upon the nature of your business, this works really well, right? And for that, let me just go back here. And then this is what I call a control. Essentially, you ship the component as it is. The agent will pick and it will display exactly the way that you describe. However, depending upon the nature of your business, at least for us, right, at my company, where a B2B SaaS, the there's so much configuration that we don't want to be over uh prescriptive because uh the feedback that I keep getting from my customers, "Oh, the flows are so confusing. There's so much configuration. How can you remove the cognitive load uh for me?"
好，挺酷的。那么，我想跟大家分享的核心就是：你想对体验施加多大程度的控制。这一点非常关键，因为作为一种非确定性的方案，你可以决定自己想不想要……嗯，像这样的东西。我们来看一下。这是 ChatGPT，我的 query 是“帮我找一家旧金山的日料店，今天去”。大家看这里，这个组件是非常“有主见”（opinionated）的，同意吧？就这儿，对吧？也就是说，你对这个组件拥有完全的控制权。所以，视你的业务性质而定，这种方式效果非常好，对吧？说到这个，我先切回来。这就是我说的“受控”型。本质上，你把组件原封不动地交付出去，agent 负责挑选，并完全按照你描述的样子展示出来。不过，还是得看业务性质——至少对我们来说，对吧，我们公司是做 B2B SaaS 的，配置项多得不得了，所以我们不想规定得太死。因为我从客户那里不断听到的反馈是：“哎呀，这些流程太让人困惑了，配置项太多了。你们怎么才能帮我把认知负担去掉？”
[12:11] Gus
But uh if you're like Booking, for example, this approach uh works uh really well. And then let's see uh how it works. So, essentially, you have the agent. The agent will just pick the component from your catalog and then it will render uh as it is, right? And uh in the interest uh of time, I'm not going to touch base on the code snippets uh that I have for this uh three different types. But afterwards, if you guys are interested, uh I can share the presentation and then you can have a look, all right? Very good. This is uh at least on the left side. Let's discuss a little bit on the right side because this is when you give full autonomy to the LLM. If you guys remember uh at least the previous attempts for my product, this exactly what we did. So, we just said, "So, hey LLM, how uh how would you compose this experience knowing that you have these components?" Right? But uh that was uh that was a little bit uh a little bit of our opinion because uh it has uh it had uh some of the components available. But it could happen that uh you can just delegate fully to the LLM. And then right now, I'm here on Claude and I asked Claude, "So, hey, create an org chart with three levels."
但如果你是像 Booking 这样的公司，这种做法就非常合适。我们来看看它是怎么运作的。本质上，你有一个 agent，agent 会从你的组件目录里挑出组件，然后原样渲染出来，对吧？时间关系，这三种类型我各自准备的代码片段就不展开讲了。之后大家要是感兴趣，我可以把演示文稿分享出来，你们自己看，好吗？很好。以上至少是左边这一侧。我们再来聊聊右边这一侧，因为这是把完全自主权交给 LLM 的情况。大家如果还记得我们产品之前那几次尝试，我们做的正是这个。我们就是跟它说：“嘿，LLM，你手上有这些组件，你会怎么组合出这个体验？”对吧？不过那次其实还是带了一点我们自己的主观设定，因为它当时手上有一部分现成组件可用。但你也完全可以把事情全权交给 LLM。比如现在我在 Claude 里，我跟 Claude 说：“嘿，做一张三层的组织架构图。”
[13:23] Gus
That was it. And then Claude just rendered this diagram and it works really well. Right? But here, if I put myself in the shoes of a company, I'm not sure I would delegate fully to the LLM. Because I cannot control at least the app the output and the outcome. And me personally, me guys, as a UX leader, the UX side of me will always say no. You got to be in control. There's been a couple of talks here at least this week on design, on taste, and judgment. And this matters a lot. If you guys want to embark on this journey of leveraging these protocols, you don't want to delegate too much of the actual experience to the LLM. You got to find alternatives and I'm going to touch on that in just a little bit. Okay, cool. And then this is how the open-ended approach works. So essentially there there's going to be an MCP tool and then this will literally ship the HTML and then in a sandbox I frame environment, this will be rendered in the host of your choice. But it can be a chat. It can be this cloud. It could be a chat GPT. It could be perplexity or it could be any other chat. If you're willing just to to give full control to the LLM, good luck. But the one that I would like to highlight is this here and this was our choice that we call the declarative. Declarative is in the middle.
就这么简单。然后 Claude 直接把这张图渲染了出来，效果非常好，对吧？但如果我站在一家公司的角度，我不确定自己会完全放手交给 LLM。因为至少输出和最终结果我是控制不了的。而我个人，各位，作为一个 UX 负责人，我身上 UX 的那一面永远会说不行，你必须把控制权握在自己手里。至少光这周，这里就有好几场演讲在讲设计、讲品味、讲判断力，这些真的非常重要。如果你们想走上这条路、用好这些协议，就别把真正的体验过多地交给 LLM。你得找到替代方案，这个我待会儿马上就会讲到。好，那么这就是开放式方案的运作方式。本质上会有一个 MCP tool，它会直接把 HTML 发出去，然后在一个沙箱化的 iframe 环境里，渲染到你选定的宿主应用中。可以是某个聊天应用，可以是 Claude，可以是 ChatGPT，可以是 Perplexity，也可以是任何其他聊天应用。如果你愿意把控制权完全交给 LLM，那祝你好运。但我想重点讲的是这里这个，这也是我们的选择，我们称之为声明式（declarative）。声明式介于两者之间。
[14:52] Gus
If you guys heard some of the protocols and I don't want to get into the specifics of each because they have different characteristics. But HTMX from Google, JSON render from Vercel, OpenUI by Thesis, those are some of the protocols that will give you this in between here. And let let's have a look, right? So at least for my product, the one that I just showed, the orchestrator agent will eventually, if you think of the whole traversal, the user will enter the query and then there's going to be the intent classification. Based off of the intent classification, then the tools will be invoked, the data will be retrieved, and somewhat somewhat in between Alice, there will be the mapping of the eligible components from your catalog to the entities of of the tools, right? And then the orchestrator will just broadcast this UI description. It's like a UI spec. This UI spec will be also we have this component catalog here and we use the Zod schema and then you got to be compliant with this protocols. This is just one of the requirements and then you just render that. Right? And then their final output will be the native UI, in this case the React components.
你们可能听说过其中一些协议，我不想逐个展开讲细节，因为它们各有各的特点。不过像 Google 的 A2UI、Vercel 的 JSON Render、Thesys 的 OpenUI，这些协议就能提供这种介于两者之间的方案。我们来看一下，对吧？至少拿我的产品来说，就是我刚才演示的那个，orchestrator agent 最终会……如果你把整条链路过一遍：用户输入查询，接着会做意图分类。根据意图分类的结果，调用相应的 tools，取回数据，而在这中间的某个环节，会把你组件目录里符合条件的组件映射到 tools 的实体上，对吧？然后 orchestrator 就会把这份 UI 描述下发出去，它就像一份 UI spec。这份 UI spec 也会……我们这里有这份组件目录，用的是 Zod schema，然后你必须遵守这些协议的规范。这只是其中一项要求，然后把它渲染出来就行了，对吧？最终输出的就是原生 UI，在这个例子里就是 React 组件。
[16:09] Gus
This is it. The good thing about the declarative approach is it will be compliant with your design system everywhere. This matters a lot. So, in our case, we did not want to delegate to the LLM because you guys saw over there you can't change the copy, right? So, it's not key one. Sometimes it's going to be March January to March. It matters a lot. So, within UX we have different factors, right? We have the actual UX's if you think of the overall experience. There is UI, there's also copy, UX writing, and so on and so forth. But this approach gives us this in between. It is less deterministic and I think this is a really good segue to some of the challenges because imagine, I'm going to use my example once again, the orchestrator will just fetch the eligible components for that query, but then it's up to the LLM how to place in the UI. And this can get really really messy, but and those are the challenges that I'll like to share with you guys here.
就是这样。声明式方案的好处是，它在任何地方都会遵循你的设计系统，这一点非常重要。所以在我们这边，我们不想把这件事交给 LLM，因为大家刚才在那边也看到了，文案你是左右不了的，对吧？所以它不一定写 Q1，有时候会写成“1 月到 3 月”。这很要紧。在 UX 里有不同的构成要素，对吧？如果从整体体验来看，有 UX 本身，有 UI，还有文案、UX 写作，等等。而这种方案给了我们一个折中。它的确定性要弱一些，我觉得这正好可以引出一些挑战。想象一下——我再拿我自己的例子来说——orchestrator 只负责为这条查询取来符合条件的组件，但它们在 UI 里怎么摆，就全看 LLM 了。这就可能变得非常非常乱，而这些正是我想在这里跟大家分享的挑战。
[17:14] Gus
I got to be careful because I've got only 30 minutes, but let's go. So, the first challenge is if the agent picks the components, who's in charge or what entity is in charge of arranging them? And this is information architecture, right? This is a critical aspect of UX. And then once again, if you put yourself in the shoes of the average customer, it matters a lot. So, if you're just left alone, the placement can be totally random. So, at least in my team, we borrow this concept of atomic design. And atomic design, it goes like this. Let me just change here. Uh yeah. So, as you can see, atomic design, and then I have the definition, is a methodology composed of five distinct stages working together to create interface design systems in a more deliberate and hierarchical manner. This helps a lot. Why? Because those are the individual elements, and if you think of the overall structure of the page, it gives me the ability to steer it as I see fit. And what we've done in my team, so this UX agent, we harnessed this UX agent. So, we eventually taught this this UX agent what good looks like, what is the optimal layout for a given situation, and we have a catalog of different templates. But I'll like to show you at least this because this detail is very important.
我得注意时间，因为我只有 30 分钟，不过我们继续。第一个挑战是：如果由 agent 来挑选组件，那由谁、或者说由哪个实体负责把它们编排好？这就是信息架构，对吧？这是 UX 里至关重要的一环。还是那句话，如果你站在普通客户的角度去想，这一点非常重要。如果放任不管，组件的摆放可能完全是随机的。所以至少在我的团队里，我们借用了原子设计（atomic design）这个概念。原子设计是这样的，我这里切换一下。呃，好。大家可以看到，原子设计——这里放了它的定义——是一种由五个不同阶段组成的方法论，这些阶段协同运作，以更有章法、更有层级的方式来构建界面设计系统。这帮助非常大。为什么？因为这些都是一个个独立的元素，而当你考虑页面的整体结构时，它让我能够按自己认为合适的方式去掌控它。我们团队做的是，针对这个 UX agent，我们给它搭了一套 harness。最终我们教会了这个 UX agent 什么才叫好、在特定情境下最优的布局是什么，而且我们还有一套收录各种模板的目录。不过我至少想给大家看看这个，因为这个细节非常重要。
[18:41] Gus
Here, it is the overall hierarchy hierarchy. So, if you remember part of the orchestrator, the orchestrator will eventually just fetch the eligible components to accomplish the query of the user, but then the next big question is, how do we arrange that? The approach that we used was think of this hierarchy. So, you have the overall page, the layout. The layout will contain different slots. So, think of this one here, the header. You can have the main, and then you can have sub sub slots. Sub slots can have sub slots. And within the sub slots, you can have eligible component categories. And then this will allow us to steer eventually the the the optimal placement of the components that have been retrieved by the orchestrator. So this is literally us codifying our UX knowledge into this agent. So the next time, you know, it doesn't matter. We'll just follow the same approach. And then this is the hierarchy that we're using. So layout to slot to sub slot to components. However, because of the orchestrator, we we flipped the order.
这里是整体的层级结构。如果你们还记得 orchestrator 那部分，orchestrator 最终只负责取来能完成用户查询、符合条件的那些组件，但接下来的大问题是：我们要怎么编排它们？我们用的方法就是按这个层级来思考。你有整个页面，也就是布局（layout）。layout 里会包含不同的 slot（插槽）。比如这里这个 header。还可以有 main，然后下面还能有 sub slot（子插槽）。sub slot 里还可以再套 sub slot。在 sub slot 里面，可以放符合条件的组件类别。这样我们最终就能引导 orchestrator 取回来的组件摆到最优的位置上。所以这实际上就是我们把自己的 UX 知识编码进这个 agent 里。这样下一次，你知道，情况怎样都无所谓，它都会按同样的方法来。这就是我们在用的层级：layout → slot → sub slot → 组件。不过因为有 orchestrator，我们把顺序反过来了。
[19:52] Gus
So from components, components will map to sub slot, sub slots to slots, and slots to templates. And then we can just arrange as needed. So this was a hell of a challenge. It is still a challenge, by the way, and it's a really good segue to the second one, the second challenge, which is the actual design, your design system and your catalog. This becomes the heartbeat of the whole thing, right? I cannot stress enough, if you guys see the potential of leveraging this UI protocols for your product, this is going to be a big deal, right? So we're pushing the boundaries and then we're testing and we keep on testing these different protocols that I mentioned, ATUI, JSON Render, Open UI, and so on and so forth. But this has been a quite challenging because the the catalog is the contract between the agent and the UI. So every property mat- matters. And not only for the catalog, but for the layout as well. So if you remember, the layout has its own components, the slots and the sub slots. Each of those components will have its own attributes. And all of that, this curation, let me just encapsulate into curation. This curation is absolutely needed so that you can deliver something meaningful. Not as some demo that you will see out there for the sake of demo, right? So, this this this will give you a control will allow you to steer from UX perspective.
也就是从组件出发：组件映射到 sub slot，sub slot 映射到 slot，slot 再映射到模板。然后我们就能按需编排。这真是个要命的挑战。顺便说一句，它到现在都还是个挑战，而这也正好引出第二个挑战，就是设计本身——你的设计系统和你的组件目录。它会成为整件事的心脏，对吧？我怎么强调都不为过：如果你们看到了在自己的产品里用这些 UI 协议的潜力，那这一块会是重中之重，对吧？所以我们在不断突破边界，一直在测试、持续测试我刚才提到的这些协议，A2UI、JSON Render、OpenUI 等等。但这一直相当有挑战，因为组件目录就是 agent 和 UI 之间的契约，所以每一个属性都很重要。不光组件目录如此，layout 也一样。如果你们还记得，layout 有它自己的组件，也就是 slot 和 sub slot。每个组件都有自己的属性。所有这些，我就统一概括成一个词：curation，精心梳理。这种梳理是绝对必要的，这样你才能交付真正有意义的东西，而不是外面那种为了演示而演示的 demo，对吧？所以这会给你控制权，让你能从 UX 的角度去掌舵。
[21:20] Gus
Very good. And last but not least, one significant challenge that we had is that my teams do not design the pixel anymore. I don't know if you could see that, right? So, we're not here designing at the entire flow. Now, AI can dictate that to a significant extent, but the nature of the work shifted quite a bit and it's been an interesting journey to say the least, a really good one, but even for the non-technical PMs and UX designers, it it was a big hit because right now we talk about the schema. Let's talk about this curation of the catalog. Let's talk about the rules. Let's talk about the synthetic data that we can generate. How can we generate the queries that will map to a given component as part of this mapping logic? Let's talk about interaction patterns. So, this if once again, if you guys are going to embark on this journey, be aware that the people element is very important. When I talk to other leaders, I I talk about the three P's: people, product, and process, right? And then a very lightweight process, but this matters a lot.
很好。最后但同样重要的是，我们遇到的一个重大挑战是：我的团队不再逐像素做设计了。不知道大家有没有看出来，对吧？我们不再在这里把整个流程都设计出来。现在 AI 能在很大程度上决定这些，但工作的性质也随之发生了很大转变。说得保守一点，这是一段很有意思的旅程，而且是很棒的一段，但对非技术背景的 PM 和 UX 设计师来说，这也是个不小的冲击，因为现在我们聊的是 schema。我们聊组件目录怎么梳理，聊规则，聊能生成哪些合成数据：作为映射逻辑的一部分，怎么生成能映射到某个特定组件的查询？我们还聊交互模式。所以还是那句话，如果你们打算走上这条路，要意识到人的因素非常重要。我跟其他管理者交流时，会讲三个 P：people、product 和 process，也就是人、产品和流程，对吧？流程要非常轻量，但这件事非常重要。
[22:28] Gus
And with that, I'm going to leave a couple of resources here. So, and by the way, those are talks from AIE for what it's worth. So, yeah, people that have been talking about this protocols over and over and over. So, please just take advantage, take a screenshot. And if you guys want to connect with me here, yeah, my LinkedIn or just take a screenshot. I'll love to talk more about the topic. I can tell this is just a matter of time, right? So, this is coming. So, thank you so much.
说到这儿，我在这里给大家留几份资源。顺便说一句，这些都是 AIE 上的演讲，供大家参考。这些人一直在反复讲这些协议。所以请好好利用，拍个截图。如果大家想跟我联系，这是我的 LinkedIn，或者直接截个图。我很乐意就这个话题再多聊聊。我敢说这只是时间问题，对吧？它一定会到来。非常感谢大家。
[23:02]
I know.
我知道。
[23:15]
[music]
（音乐）