#97: What it means to be a forward-deployed product leader | Chase Schwalbach (SVP @ Millie)

频道: Supra Insider
视频: https://www.youtube.com/watch?v=XzbJsEYB1Ng
原文语言: en
统计: 共 125 轮 · 主持人 66 · Chase 59

[0:09] 主持人

All right, we're live, Chase. Really excited to have you with us.

好，我们开始录了，Chase。非常高兴请到你。

[0:13] Chase

Thank you. I'm excited to uh get the group back together.

谢谢。我也很开心能和大家再聚一次。

[0:16] 主持人

Welcome. Welcome. Yeah. Um, people probably would not have any way of knowing this, but you were the facilitator in my first uh my first Supra cohort uh with all the early stage heads of product and I really enjoyed you as a facilitator and I suspect you're going to do a lot more talking today than you did in those sessions as a facilitator.

欢迎欢迎。是这样，听众大概不知道，你其实是我第一届 Supra cohort 的引导人，那一届都是早期阶段的产品负责人。我特别喜欢你当引导人的风格。我猜今天你说话的量会比当年那些引导 session 多得多。

[0:34] Chase

I always talk a lot. I mean, it's hard.

我一向话就多。这没办法。

[0:37] 主持人

No, you you what I loved about your facilitation style is you did not shy away from participating in the conversation and bringing your experience to what we're talking about. So you were not trying to blend into the background. I think you were part of it. So I really appreciated that about you.

不不，我特别喜欢你引导的地方在于，你不会回避参与讨论，会把自己的经验带进我们聊的话题里。你不是想着往后缩、融进背景里，而是真正参与其中。这点我特别欣赏。

[0:50] 主持人

Yeah. I mean Chase Chase has done the thing a few times. So yeah, he has a lot of wisdom to share.

是啊。我是说，Chase 这种事已经干过好几回了，所以他有很多经验可以分享。

[0:56] 主持人

For sure. So um so yeah, like I I want to kick this off with why we were excited to chat with you. So he recently shared some stuff with me and Mark over Slack that we thought was just this could be a really interesting conversation. Lots of I I personally had a lot of curiosity kind of like light bulbs going off and I wanted to save them all for the recording. But you mentioned to us that you recently had I want I want to kind of just paint the broad strokes and let you really tell the story, but like you recently went very deep as kind of like a rolling your sleeves up IC even though you're the SVP of product and technology at Millie and did you know we jokingly started calling it it's almost like you're like a Ford deployed product leader. you went into like a new space, figured out a bunch of stuff, tried to protect or shield or just like isolate the rest of the team from like the messiness perhaps of that very early stage,

绝对的。所以呢，我想先讲讲我们为什么这么期待和你聊。他最近在 Slack 上跟我和 Marc 分享了一些东西，我们当时就觉得这可以是一场特别有意思的对话。我个人当时冒出很多好奇、脑子里不停有灯泡亮起来的那种感觉，我就想把它们全都留到录节目的时候再聊。你当时跟我们提到，你最近有段经历——我先把大框架勾勒一下，具体的故事留给你来讲——就是你最近深深扎进去，像个卷起袖子亲自上手的 IC 一样干活，哪怕你其实是 Millie 产品与技术的 SVP。我们后来还开玩笑,说你这几乎就是个「forward-deployed(前线部署)的产品负责人」。你一头扎进一个全新的领域,搞清楚一大堆东西,然后尽量保护、屏蔽、或者说隔离团队其他人,让他们不用去趟那种极早期阶段可能会有的一地鸡毛。

[1:47] 主持人

and then when things got to a certain threshold, brought the team along. And we just want to want to dive into that story because I think a lot of product leaders potentially find themselves in a boat where they could see the value of doing what you did. And I want to maybe give people something actionable to go with. But how does that sound for how we spend the time? at least kickoff.

然后等事情到了某个临界点,你再把团队带进来。我们特别想深挖这段故事,因为我觉得很多产品负责人可能都会碰到类似的处境——他们看得到你这么做的价值。我也想给听众一些能实际拿去用的东西。你觉得我们这样安排这段时间怎么样?至少开场先这么来。

[2:07] Chase

Yeah. Love it.

好啊,我喜欢。

[2:10] 主持人

Cool. So, yeah. What what what have you been up to? What what was that whole that whole project?

太好了。那就聊聊,你最近都在忙什么?整个项目是怎么回事?

[2:16] Chase

Yeah. I mean, I think we've got to go back to even explain it, but you know, I've I've been in product for 15 years. Um I was briefly an engineer before for about three or four years and then went into product because I like talking too much for most engineers, I think. Um and I was always on technical product and then the last two stops were really technical. So setting up data warehouses. It was all kind of about data for me in my last two stops at pretty enterprisey companies. So data bricks and kubernetes and all this crazy stuff and not you know just kind of far away but very hands-on kind of architect role. You know, at Millie, I they reached out to me four months after my first child was born. They had no idea about our story. We had had, you know, a really rough birth and it was one where I thought the, you know, monitoring of my wife's situation and the feedback and everything else was ignored um and not escalated properly. And I thought the rough situation, which is better now, so all good or as good as you can be from from some of this, but I I thought it could have been avoided, right? And I was still on paternity leave. I wasn't planning anything, but it was something that was in my head. And I got a call from a recruiter and he was like, you know, we're trying, you know, the the founding story is the the founder had preeacclampsia, self-dagnosed, doctors tried to say no, went to the hospital, and basically saved her own life, right? and they were like, "Are you interested in this kind of thing?" And I was like, "You have no idea." Um, so, you know, stepping in there, I learned that they really were focused on business model innovation in the early stages and and data. So, that was really all their focus. It wasn't tech enablement nearly as much as as it was like making sure that they were doing this the right way operationally. So, there was a lot of manual work, but it was all kind of data driven. but you know your classic Excels and everything else. So they reached out for me and our head of engineering to start building. So pulling all of this into databases and everything else. Um

好。我觉得得先往前倒一点才讲得清楚。我做产品已经 15 年了。之前有大概三四年短暂当过工程师,后来转做产品,因为对大多数工程师来说,我大概是话太多了。我一直做的都是偏技术的产品,而最近两份工作技术性尤其强,都是在搭数据仓库那类事。在我上两家挺偏企业级的公司,我干的基本都是围着数据转:Databricks、Kubernetes,各种复杂到不行的东西,是那种离一线有点远、但又非常亲自上手的架构师角色。到 Millie 这边,是我第一个孩子出生四个月后他们联系上我的。他们完全不知道我家的经历。我们当时那场分娩非常凶险,我一直觉得对我太太状况的监测、那些反馈之类的信息全被忽视了、没有被恰当地上报。虽然现在好多了,一切都还好,或者说在经历过那些事之后能有多好就多好吧——但我一直觉得那本来是可以避免的。当时我还在休陪产假,没打算找工作,但这件事一直搁在我心里。然后我接到一个招聘的电话,他大概说:你知道吗,我们的创始故事是——创始人当年得了先兆子痫(preeclampsia),自己诊断出来的,医生一开始不认,她硬是去了医院,基本上是救了自己一命。他们问我:你对这类事情有没有兴趣?我当时就想:你根本想象不到我有多有兴趣。所以真正走进去以后我才发现,他们在早期阶段真正聚焦的是商业模式创新和数据,这才是他们全部的重心。当时的重点根本不在技术赋能上,而是在确保他们在运营层面把这件事以正确的方式做出来。所以那时有大量人工的活儿,但都是数据驱动的——你能想到的那些经典的 Excel 表格之类。他们联系我和我们的工程负责人,就是来从零开始搭建的,把这一切汇进数据库里等等。

[4:24] 主持人

so because before you got there like what was the like the business or the service they provided was it basically just monitoring of the pregnancy a little bit more hands-on or

那在你去之前,他们做的业务或者说提供的服务是什么?基本上就是对孕期做那种更亲力亲为一点的监测吗?还是……

[4:34] Chase

No, I should explain that. You're right. Um you know we run women's health clinics so we're frontto back. Um, we partner with hospital systems who currently in something like a low-risk birth try to use uh obgyns who are extremely expensive and and not available. And we're moving that towards a more of a midwife model. So that's our that's where we started was maternity patients. We have our own clinics. We operate them front to back and we uh help with delivery in the hospital setting. So, we're partners with a hospital, but we're kind of doing all the marketing and bringing in patients and then we've got the full ninemonth 40week episode with them. We've expanded into gynecology and we're expanding into more clinics and and everything else in the meantime. So, what we were doing kind of before he came was just being really good at the operations. So you know yes you know instead of using epic for inbox we're using zenesk we're using slack we're using the basic tools that actually don't infiltrate most healthcare systems the outcomes data was everywhere we did have a very strong mobile app so that was kind of the big tech differentiator was all about like a mobile app experience where you could do your own blood pressure monitoring and check for things like pre-claim which was you know again our founding story and that was all really built out by a external team. So it wasn't connected to our operations and our operations was running very just traditionally and that's kind of where I came in was to tie these things together and start building out the operations in a scalable way, right? So it's kind of do the things that don't scale. Make sure you know our NPS is high, our patients are happy, we're saving money for the health care system as a whole, we're saving money for everyone and they're having a better experience. We're having better outcomes, less niku rates, less C-sections, etc., etc. So they were working on that for three years in their first clinic started as scaling started to bring in the tech inhouse. So that was kind of the story of when I came in and then from there so kind of getting into the forward deployed PM thing kind of um

不是,这个我得解释一下,你问得对。我们是开女性健康诊所的,从头到尾全流程都自己管。我们和医院系统合作——他们现在对低风险分娩这类情况,往往会动用极其昂贵又难约到的 obgyn(妇产科医生),而我们是把这个往更偏助产士(midwife)的模式去转。我们最早切入的就是产科患者。我们有自己的诊所,从头管到尾,并在医院的场景下协助分娩。所以我们是和医院合作的,但市场推广、把患者引进来这些全是我们在做,然后整个九个月、四十周的孕程都由我们全程陪着。我们后来又扩展到了妇科,同时还在开更多诊所、拓展更多业务。所以在他来之前我们做的,基本就是把运营这件事做到极好。比如收件箱我们不用 Epic,而是用 Zendesk;我们用 Slack,用的都是那些其实压根渗透不进大多数医疗系统的基础工具。结果数据(outcomes data)散得到处都是。不过我们确实有一个很强的手机 App,这是当时最大的技术差异化点——整套体验都围绕手机 App 展开,你能自己做血压监测、自查先兆子痫这类东西,这又回到我们的创始故事了。而这个 App 完全是外部团队做出来的,所以它跟我们的运营是脱节的,运营那边就非常传统地跑着。我进来要做的,就是把这些东西拼接到一起,并开始以一种可扩展的方式把运营搭建起来。也就是所谓「先去做那些不可规模化的事」:确保我们的 NPS 高、患者满意,替整个医疗系统省钱、替所有人省钱,同时让他们体验更好、结果更好——更低的 NICU(新生儿重症监护)入住率、更少的剖腹产等等等等。他们在第一家诊所干了三年,规模开始起来后,才开始把技术往公司内部收。这大概就是我进来时的背景。从这儿开始,就慢慢聊到 forward-deployed PM 这件事了。

[6:46] 主持人

and you've been at the company almost a year now right

你在这家公司差不多快一年了,对吧?

[6:50] Chase

about a year.

差不多一年。

[6:51] 主持人

Yes, almost a year. Yep. Almost a year.

对,快一年了。嗯,快一年了。

[6:53] Chase

So I spent that first six months doing the healthc care stuff that sucks. So, HIPPA everywhere in a bunch of new places where we hadn't needed them before. So, you know, getting all the AWS set up, all the data bricks set up, hex for analytics, getting that HIPPA compliant, everything else, moving all of our data to one place. So moving where we still had sheets, moving that out, moving Zenesk into something that was actually a database, setting up the fiverr and getting those HIPPA, you know, just all of the infrastructure probably took

所以头六个月我都在干那些医疗行业里最招人烦的活儿。比如 HIPAA(医疗隐私合规)——一堆我们以前根本用不着它的新地方,现在处处都得合规。把整套 AWS 搭起来,把 Databricks 搭起来,分析用 Hex,再把这些全弄成 HIPAA 合规;把我们所有数据往一处归拢:还有些地方在用表格的,把那些迁出来;把 Zendesk 里的数据迁进一个真正意义上的数据库;还得把 Fivetran 搭起来、也弄成 HIPAA 合规……总之,光是这些基础设施,大概就花掉了……

[7:25] 主持人

like no customerf facing like no no changes to like customer experience or user interfaces for that that period.

也就是说这段时间里,基本没有任何面向客户的东西,对客户体验、对用户界面都没有任何改动?

[7:31] Chase

Yeah.

对。

[7:32] Chase

Yep. Yeah. And I mean I I think that would be one of the things that I would will hit on a lot is yeah without the infrastructure there if you're set up in the kind of old school way in healthcare with Epic running everything and security departments being you know the top dog you you can't experiment because there's no such thing as a staging or sandbox environment in those

没错。我觉得这恰恰是我特别想反复强调的一点:如果底层没有这套基础设施——如果你是按医疗行业那种老派方式搭起来的,一切都跑在 Epic 上、安全部门又是最大的话事人——那你根本没法做实验,因为在那种环境里,压根不存在所谓的 staging 环境或 sandbox 沙箱。

[7:56] 主持人

Can you talk about Epic? What what is Epic?

你能讲讲 Epic 吗?Epic 到底是什么?

[7:58] Chase

Oh sorry yes we're not on a healthcare podcast. Epic is the classic um electronical medical record, electronic medical record that almost everyone uses. So whenever you're in the hospital, they're typing into Epic, which is a huge monopoly now. And yes, there's others, but it's almost always Epic. So that's where your your doctor is saving your official record. They are they don't play nice. So they don't want you.

哦,抱歉,对——毕竟这不是个医疗行业的播客。Epic 就是那个几乎人人都在用的经典电子病历(EHR,electronic medical record)系统。你在医院的时候,医生敲进去的就是 Epic,它现在基本是个巨无霸垄断。当然也有别的,但绝大多数情况下都是 Epic。你的官方病历就是你的医生存在 Epic 里的。而它们不太好打交道——它们不想让你(把数据拿出去)。

[8:22] 主持人

You have to rip out you had to rip out Epic and kind of build replace it with your own.

你得把 Epic 整个拆掉,再用你们自己的东西去替换它?

[8:28] Chase

It's actually the opposite way. So, we're kind of doing duplicate work on the front end with modern tools and then entering that into Epic. And the thing that you'll hear me say a lot if you talk to me in healthcare is that the the Epic system is a billing system masquerading as like patient care. So, it is really optimized for getting billing codes. So, you can submit those to insurance and Medicare. And doctors mostly don't like it. It's not really meant for like delivering the absolute best care. that's just not where it makes money, right? Um, having a quality score of 92 or 93 doesn't matter. What matters is you got the billing codes right and got reimbursed. So, we actually have a lot of frontline tech that we've put in front of Epic. So, scheduling is a big one. Intake forms and getting information from people is a huge one so that we can send SMS reminders and things like that. The mobile app is a big one where now we're getting a bunch of data in. Now I'm putting AI on front of all of that so that

其实正好反过来。我们是在前端用现代化的工具做一遍重复的活儿,然后再把那些数据录进 Epic。有句话如果你跟我聊医疗,你会常听我念叨:Epic 这套系统本质上是个「披着患者护理外衣的计费系统」。它真正被优化到极致的,是拿到计费代码(billing codes),这样你才能把它们提交给保险公司和 Medicare(联邦医保)。医生大多不喜欢它。它压根就不是奔着「提供最顶级的医疗护理」去设计的——因为那不是它赚钱的地方。你质量分拿到 92、93 都无所谓,真正要紧的是你的计费代码填对了、钱报销回来了。所以我们其实在 Epic 前面加了很多面向一线的技术。排班就是一大块。填表、从患者那儿收集信息也是很大一块——这样我们才能发短信提醒之类的。手机 App 也是一大块,现在我们能从里面拿到一大堆数据。而现在我正把 AI 铺到所有这些东西的前面,这样……

[9:26] Chase

all of their history and memories with us are kind of stored. And then it'll be the same actually with transcription. So ambient charting is a big thing where you take a transcript, it listens and then it puts it into the chart. Since we partner with hospital systems, they'll all kind of use their own Epic instance and no Epic instance is the same. So it's not like they have an API. Each one is central to this hospital and requires a very large implementation to do anything. So instead of trying to use these cool ambient charting tools, I'm just going to record it. So we'll already have to consent people for the recording,

……患者跟我们之间的所有历史和记忆,都能被存下来。转录这块之后也会是一样的路子。所谓 ambient charting(环境式病历记录)是很热的一个方向:你拿一段对话录音,系统边听边把内容填进病历里。但因为我们是和医院系统合作的,他们各用各自的 Epic 实例,而没有哪两个 Epic 实例是一样的。并不是说它们有个统一的 API——每一套都是围着那家医院量身定制的,想干任何事都得走一遍非常庞大的实施流程。所以与其去硬用那些很炫的 ambient charting 工具,我干脆就自己把对话录下来。反正为了录音我们本来就得先取得患者同意。

[10:01] Chase

I'm just going to dual record it so I can actually use this data because we'll never get it out of Epic. They have no interest in that.

我索性同步再录一份自己的,这样我就真的能用上这份数据——因为我们永远别指望能把它从 Epic 里弄出来,人家对这事毫无兴趣。

[10:06] 主持人

Yeah.

嗯。

[10:07] 主持人

So it's like it's like a dual technology layer.

所以这就像是一个双重的技术层。

[10:10] 主持人

Yeah. It's almost like I think of Epic as like a little bit of Salesforce but like way more gated and way more custom. Like just people really customize it. And one of the benefits of Epic too, right, is like you can share like electronic health records across hospitals too. So there's like some benefit like some benefits there as well, right, of of like there that you can just get information from other doctors or clinics and

对。我一般把 Epic 想成有点像 Salesforce,但门槛封闭得多、定制程度也高得多——大家真的会把它改得很个性化。不过 Epic 也有它的好处,对吧,比如你可以在不同医院之间共享电子健康记录(EHR)。所以那儿确实有一些好处,就是你能直接从别的医生或诊所那儿拿到信息……

[10:31] 主持人

there's like data portability kind of like implications that make it make it very easy.

……有点像是数据可迁移性(data portability)带来的那种便利,让这件事变得非常容易。

[10:36] Chase

Yeah. But yeah,

对。不过话说回来，

[10:38] 主持人

but it sounds like the biggest like uh and I know we're not doing a deep dive on Epic, but this is helpful to kind of get the framing is like

听起来最大的问题——我知道我们不打算深入聊 Epic，但先把大框架搭起来是有帮助的——就是说，

[10:44] 主持人

you came into you came into the picture when

你进来的时候，正好是

[10:47] 主持人

that abstraction layer on top of Epic or I guess like the front end layer on top of Epic didn't exist is what it sounds like.

Epic 之上那层抽象层、或者说 Epic 之上那层前端还不存在的时候，听起来是这样。

[10:56] Chase

Yeah, there was pieces there but we weren't tying them together. We weren't utilizing them across but we were doing scheduling outside of Epic for example which was kind of core to us.

对，当时是有一些零件在的，但我们没把它们串起来，也没有跨场景去用。举个例子，我们的排班是在 Epic 之外做的，而排班对我们来说算是核心业务。

[11:05] 主持人

Got it. And then you mentioned there was like an uh external team that was building like the mobile app for example. That's like a is that like a dev shop or like an agent like an agency?

明白了。你刚才还提到有个外部团队在做，比如那个移动 App。那是个开发外包公司，还是说像代理机构那样的？

[11:14] Chase

Yeah, they were here before me. So we had kind of a static mobile app with I I'm downplaying how good it was for healthcare and and still is, but it's you know kind of just shows your week by week pregnancy. If you have any kids, you'll know there's a bunch of these apps. Shows how big your baby is or whatever else. But it also showed some articles and things that we had built. So we had spent those three years building out thousands of pages of documents. Every time we got asked a question, it would basically go into some document as an answer somewhere. So I don't think the team I mean the team definitely didn't do it because LMS weren't around. But the idea of like a knowledge base for vector databases and AI to look at wasn't the reason. It was just operational efficiency. But that along with hundreds of hours of YouTube videos are all these resources available in the app. And every time we kind of went down a path where every time there was a question we were like we have to put it somewhere so that we don't have to answer it again. And what happened? Everyone stopped reading everything. So then we were back to okay now everyone just assumes there's way too much and just ask. So now we're back in the hell of answer everything. And that's kind of where I came in was we had had a great mobile app, all these resources being utilized. It was being, you know, people were logging in. 90 plus% of people were logging in, but we were still getting a lot of questions from certain subgroups that just had no interest in really looking every week at all the information you were sending them because it was a lot. And some people, it's a persona thing. Some people love it. I was one of the people who would have loved it. Some people are like, I'm not reading any of this. Like, you're just have to tell me what I need to know in the appointment. Yeah, I mean I think in a way they they were very lucky to hire you because I think I don't know if everyone would have had the foresight of like hey we need to create this funding. I know where we're moving to this future where everything's going to be, you know, like LLM power and there's going to be a lot of um advantages to having agents taking action on top of valuable data. Um and even though it might be counterintuitive to spend in the first, you know, as a head of product, you probably want to have impact right away and spending the first six months just building foundational and not having any customer um outcomes to to show for and it might be perceived that as a risky move, but you know, now as you're sharing, it's like it's like a very obvious h first step, right, to to then the velocity that it unlocks later on is huge. So it's interesting. Yeah. Did you did

对，他们在我来之前就在了。所以我们当时有一个比较静态的移动 App。我这么说其实低估了它——放在医疗行业里它已经算做得很好了，现在也是。它基本就是给你看孕期一周一周的进展，如果你有孩子就知道，市面上这类 App 一大堆，会告诉你宝宝现在有多大之类的。但它同时也放了一些我们自己写的文章和内容。我们花了三年时间，建了成千上万页的文档。每次有人问一个问题，答案基本上就会被写进某个文档里存起来。所以这个团队——当然团队当时肯定不是冲着这个去的，因为那会儿还没有 LLM——但那种给向量数据库和 AI 去读的知识库的思路，并不是他们当初建这些的原因，纯粹是为了运营效率。但就是这些文档，再加上几百个小时的 YouTube 视频，全都是 App 里能查到的资源。我们一路走下来，每次一有问题，我们就想：得把它放到某个地方，这样以后就不用再回答一遍了。结果呢？大家干脆什么都不看了。于是我们又回到原点：所有人都默认信息太多了，直接开口问。就这样又掉回了『什么都得答一遍』的地狱。而我进来，差不多就是在这个节点——我们有一个很棒的移动 App，所有这些资源都在被用，大家也确实在登录，90% 以上的人都在登录，但我们还是从某些特定人群那里收到大量问题，这些人根本没兴趣每周去翻你发给他们的那一大堆信息，因为量实在太大了。而且有些人纯粹是画像不同——有些人爱看，我就是那种会爱看的人；有些人就是『我一个字都不想看，你就在门诊当面告诉我我该知道什么就行』。

[13:36] 主持人

did you know that was going to be what you spend the first four months on or when you took the job?

你当初接这份工作的时候，就知道自己头四个月会花在这上面吗？

[13:42] Chase

No, but I did before I joined. So I took the job and then there was a thing that popped up around a large data um input we were receiving and the previous CTO so I'm I'm over engineering AI and product which is why I'm doing this infrastructure stuff. you know, most product people would not obviously do that. Reached out and I said, you know, just load it into your warehouse. Like, what do we have? Snowflake, what do we have data bricks? And she was like, we don't we don't have any. And I was like, oh, okay. Well, that pretty much lines out my next couple months there, doesn't it? Um because that made me realize, you know, data was in multiple places, not stored in one, you know, in one area. that's probably all in HIPPA compliant uh silos, which is just the classic death nail of AI and healthcare right now. So, I was like, "Yeah, I'm going to spend all these months with vendors convincing them to give me data, usually paying them a bunch of money to get data out because they all have special uh tiers for healthcare to make sure that we get screwed over as much as possible." So, that was, you know, just all negotiations and everything else. And yeah, I did figure it out before I joined, but not whenever they talked about the mission, it didn't matter what my first six months were, right? Like it was just I'm going to join this and tell me how I can help, I'll do it.

不知道，但我在入职之前就搞清楚了。事情是这样：我接了这份工作，然后冒出来一件事——我们要接收一批很大的数据输入。我这人是在 AI 和产品上『过度工程』的，这也是我为什么会去折腾这些基础设施，大多数产品人显然不会这么干。我找到前任 CTO 说：你把它加载进你们的数仓就行，我们有什么？Snowflake？还是 Databricks？她说：我们什么都没有。我当时就『哦，好吧』。这基本上就把我接下来几个月的活儿都勾勒出来了，对吧？因为这让我意识到，数据散在好几个地方，没有集中存在一处，而且大概率全都锁在 HIPAA 合规的孤岛里——这正是当下 AI 在医疗行业的经典死穴。所以我就想：行吧，接下来这几个月我得花在跟供应商打交道上，说服他们把数据给我，通常还得付一大笔钱才能把数据弄出来，因为他们个个都有专门针对医疗行业的『特殊档位』，就是要确保把你狠狠宰一刀。所以那全是谈判之类的破事。是的，我在入职前就把这些看明白了，但当他们跟我谈使命愿景的时候，我头六个月干什么根本不重要——我就是要加入这件事，告诉我怎么能帮上忙，我就去做。

[14:59] 主持人

Yeah. And it sounds like maybe they got even a little bit lucky that you had that experience before. Or were they looking for someone that had that like data warehousing and like infrastructure background because I feel like in a way it was like the perfect storm, but I'm wondering like how intentional they were about that perfect storm or they just got lucky that they found you when they found you. Yeah, I I would say that they had about 70% of it.

对。而且听起来他们可能还有点运气，正好碰上你之前就有这个经验。还是说他们本来就在找有数仓和基础设施背景的人？因为我感觉这在某种程度上像是天时地利人和，但我很好奇他们对这种『完美风暴』是有多刻意去追求，还是纯粹运气好，在那个时间点撞上了你。（Chase）我会说，他们大概想到了 70%。

[15:25] Chase

And that that's

然后那个……

[15:25] Chase

I'd say they had about 70% of it. They actually interviewed. So we were merging with CVS at my old company and they interviewed like three people from the leadership team on the product team and they actually kind of adjusted towards my vision as we talked because they weren't sure they were kind of interested in the AI world and how PMs and tech were merging. Right? That was like one of the hypotheses is it doesn't make as much sense to have like a super hardcore tech leader as your main leader anymore. It might make more sense to have that be a product leader. So they were starting with just pure product and as they went through the interview whether that was from my peers you know and their answers or mine I don't know but they went more towards like a a technical product leader that's someone who's been in architecture and been doing this for 10 years on the technical side and from an engineering type background. So then I think I just sold them on the vision at that point. But yeah,

我会说他们大概想到了 70%。他们其实面试了……当时我上一家公司正在和 CVS 合并，他们面了产品团队里领导层的大概三个人。而且我们聊着聊着，他们其实是朝我的愿景在往那边调整，因为他们自己也没想好，他们对 AI 这个世界、对 PM 和技术怎么融合是有兴趣的。对吧？他们当时的一个假设就是：再养一个超级硬核的技术领导当你的一号位，可能已经没那么合理了，也许让一个产品领导来当更合理。所以他们一开始是奔着纯产品去的，但随着面试推进——不知道是因为我那些同僚（peers）和他们的回答，还是因为我的回答，我也说不清——他们越来越倾向于要一个偏技术的产品领导，就是那种在架构上摸爬滚打、在技术这一侧干了十年、有工程背景的人。所以我觉得到那个点上，我就是靠愿景把他们说服了。但对，就是这样。

[16:20] 主持人

do you I think there's also like something very interesting here about like the rise of the like technical product leader or like CPTO like I'm seeing more and more of that role too. Um, and do you think do you think it's more kind of essential to have that role when in a situation where you have like a very heavily operational business where there's not that capability and but you have to move quickly and build or versus like maybe like a traditional tech SAS company that already has like really good like engineering talent like do you think that's where it makes most sense like do you have a sense of there of like when does it make sense to hire that technical CPO versus maybe someone who's not as technical? Yeah, I this has been a crusade of mine for 5 years, 10 years. So, you know, I went from a world where I thought product should be separate and that lets us um be the the arbiter of all the different groups, right? So, if you're under tech, then tech always wins. You do too much uh you know moving what is it? Moving the deck chairs around on the Titanic as it's sinking, right? just tech dead all the time and you know just for no good reason they don't have to prove themselves right so it was just a huge you know you must keep them split with that I ran into all kinds of issues trying to fight for that and then I kind of led to technical PMs being underneath technical orgs but keeping product out as like more of a growth product marketing type function and then I've kind of switched it and actually just said like this idea of the CTO being the absolute top person is starting to go away because you can hire proper infrastructure people. You can hire them. They're going to be expensive. I'm not suggesting not to do that. You absolutely have to have really good people who understand tech infrastructure and things like that uh in a scaling environment. But you don't need the head of technology and product to be someone who understands every single piece of that. But you don't want them to be someone who doesn't understand anything technical or you're not going to get the right kind of technical leaders is my opinion. So, you're going to end up in a world where they're like, if my boss doesn't know anything about my world and doesn't impress me, right? Your best technical leaders want to be impressed by learning something, right? And if it's just learning how to do Jira tickets, which I'm I'm being a little plas, but um it's they're not going to be interested. So, I I've kind of went full circle to pull the group together, hire a technical product leader. That can often just be a fully engineer person forever who's just always been very product focused. That's fine. But I think it's, you know, somewhere that they're closer to the product because there's a lot more feel, right? So I think we've all seen the the what's his name? Rick uh the guy who does the vibes. Rick Rubin. Yeah. We've all seen the Rick Rubin music, you know, vibes thing.

你觉不觉得这里还有个特别有意思的点，就是所谓『技术型产品领导』、或者叫 CPTO 这种角色的兴起？我也越来越多地看到这种岗位。我想问，你觉得这种角色是不是在这类情况下更不可或缺——就是你面对一个非常重运营的业务，内部又没有这种能力，但你必须快速行动、快速搭东西；而相比之下，一家传统的技术型 SaaS 公司本来就有很好的工程人才。你觉得哪种情况下这个角色最有意义？你对『什么时候该招一个技术型 CPO，什么时候招一个不那么技术的人更合适』有没有一个判断？（Chase）有，这件事我已经念叨了五年、甚至十年了。我最早是从一个信念出发的：我以为产品应该独立出来，这样我们才能当各个部门之间的仲裁者。因为如果你归在技术底下，那永远是技术赢，你就会花太多力气去干那种——怎么说来着？——泰坦尼克号都要沉了你还在甲板上挪躺椅的事，永远是技术说了算，而且往往毫无道理，因为他们根本不需要自证。所以我一度觉得，你必须把它们拆开。抱着这个信念，我在争取它的过程中撞上了各种各样的问题。后来我就走到了：技术型 PM 归在技术组织底下，而把产品单拎出来，更多当成一个增长、产品营销那类的职能。再后来我又把它翻过来了，我实际上是这么想的：CTO 是那个绝对一号位的这套观念，正在开始消退，因为你完全可以去招专门的基础设施人才。你能招到他们，他们会很贵——我不是在建议不要招；在一个要做规模化的环境里，你绝对必须有真正懂技术基础设施这类东西的顶尖人才。但你的技术兼产品负责人，并不需要是那个把每一块细节都吃透的人。可你也不希望他是个对技术一窍不通的人，否则你招不来对的技术领导，这是我的看法。因为你会陷入这样一种局面：你最好的那些技术领导会想——如果我老板对我这个领域一无所知、又没法让我服气……对吧，你最优秀的技术领导是想被『学到东西』这件事折服的。而如果跟着你只能学会怎么开 Jira 工单——我说得有点损，但意思到了——他们是不会有兴趣的。所以我兜了一大圈，又回到把这个组合并起来，去招一个技术型产品领导。这个人完全可以是一个一辈子都是工程师、但一直非常关注产品的人，那也没问题。但我觉得关键是，他得离产品更近一些，因为里面有大量靠『手感』的东西。所以我想我们都见过那个……他叫什么来着？Rick……那个讲『vibes（氛围/手感）』的人。Rick Rubin，对。我们都见过 Rick Rubin 讲音乐、讲手感那套东西。

[19:06] Chase

And that's what's happening in the world is

现在这个世界上正在发生的就是——

[19:08] Chase

everyone is coming with prototypes. Like your marketing team is all of a sudden showing up with a mobile app prototype and you're like, what in the hell are you doing? Like why are you doing this? And I think it's starting to be like okay, we've got a lot more ideas and we've got to we've got a lot more velocity now. So it's not so much about like how can we find the time for the engineers, it's what idea should we work on and I think that's something that you you've got to have a product minded person to help guide that now.

人人都带着原型（prototype）来找你。突然之间，你的市场团队都能拿着一个移动 App 原型冒出来，你就『你们这是在搞什么鬼？为什么要做这个？』。我觉得现在开始变成这样了：好，我们的点子多了太多，我们现在的速度（velocity）也快了太多。所以问题不再是『我们怎么给工程师挤出时间』，而是『我们到底该做哪个点子』。我觉得这就是现在你必须得有一个懂产品的人来帮忙掌舵的地方。

[19:36] 主持人

Yeah. I mean if you look at parallel universes of like let's say they hired someone who was not as technical as you, there's a world in which they start bite coding on top of your infrastructure. they build a bunch of stuff. Maybe they deploy it and then maybe a year later they're like, "Oh this is a mess. We can like that we're spending a lot of money on tokens because our context is messy. It's unreliable." And then you basically just wasted a year of work. H but if you have someone like you who knows kind of like the importance of having a good data warehouse and and moving everything and getting everything organized and kind of also in a way too I think in this case like a good example of like domain expertise is important too, right? like you understand like the ins and outs of HR systems like Epic and kind of what can you do, what cannot do, like yeah, it sounds like you can get to the desired outcome much faster even though maybe the start looks a lot slower and less exciting which is cool. And you're avoiding a bunch of pitfalls, too, like PHI leakage or huge problems down the whenever you partner with a a group that has a full audit team and they're just like absolutely not like none of this is going to work and you've got to start all over. And one of the big things I talk about is institutional barriers in healthcare. And if you go down the path that sets you down one way, it's always hard to reverse it back in healthcare because you start layering on things and these HIPPA requirements and everything else and you add a bunch of risk to go down a different path. So you end up just building Frankensteines on top of the original path and then you just get worse and worse. And that's almost every health care system looks exactly like that. They basically started on sheets of paper. They moved it to Excel and that's how they ran their operations and they built a whole health care system on the Frankenstein of that and it's just not scalable and it's security systems everywhere because they have to be because there's so many holes. So it's so easy to screw up.

对。我是说，如果你去看那些平行宇宙——假设他们招的是一个没你这么技术的人，那就有一个宇宙里，他们开始在你的基础设施之上『vibe coding』，噼里啪啦搭出一堆东西，可能还上线了，然后也许一年之后他们才发现『哦这是一团乱麻，我们在 token 上花了一大笔钱，因为上下文（context）乱七八糟、也不可靠』，那你基本上就白白浪费了一年的工作。但如果你有一个像你这样、懂得『先建好数仓、把一切迁移整理归置』有多重要的人……而且我觉得在这个例子里，还有个很好的点是领域专长（domain expertise）也很重要，对吧——你懂 EHR 系统比如 Epic 的里里外外，知道它能干什么、不能干什么。是的，听起来即使起步看上去慢得多、也没那么让人兴奋，你反而能更快到达想要的结果，这挺酷的。而且你还绕开了一大堆坑，比如 PHI（受保护健康信息）泄露，或者那种——你一旦和一个有完整审计团队的机构合作，他们就会直接『绝对不行，这些统统跑不通，你得从头再来』——那种大麻烦。（Chase）我常讲的一个大话题就是医疗行业里的『制度性壁垒』。在医疗行业，一旦你沿着某条路走下去，之后想掉头回来永远都很难，因为你会不断往上叠东西、叠各种 HIPAA 要求等等，你要换一条路走就得承担一大堆额外风险。所以你最后只能在最初那条路上一层层造出一堆『弗兰肯斯坦式』的怪物，然后越来越糟。几乎每一个医疗系统看起来都恰好是这个样子。他们基本上是从一张张纸质表格起步，然后搬到 Excel 上，就靠这个跑运营，再在这套『弗兰肯斯坦』的地基上搭出一整套医疗系统，根本没法规模化，而且到处都是安全系统——因为不得不这么做，毕竟窟窿太多了，太容易出岔子了。

[21:30] 主持人

Okay. So to bring us back to our our main trail because I'm I'm really curious. I'm really curious about it. So, how long would you say it took you to get to that point where you felt like the basic call it pre-work for or the prelude was complete for you to actually get into what you might consider the more fun the fun stuff? Uh, was that like four four months you said?

好，把话题拉回我们的主线，因为我真的很好奇。我特别想知道：你会说，你花了多长时间才走到那个节点——就是你觉得那些可以叫做前置工作、或者说序曲的部分已经完成，你终于可以进入你眼中更好玩的那部分了？你刚才说是四个月左右？

[21:50] Chase

Yeah, I think it was about four months of vendor and all that fun stuff. Now, while while you were in that initial fourmonth phase, were you starting to have some kind of like thoughts popping up in your head about what the priorities should be once that phase is complete and kind of like aligning with I guess like the CEO about like what the priorities are or did you were you like super heads down for four months and didn't even get much time to think about that until after?

对，我想大概是四个月，都花在跟供应商打交道那些破事上。（主持人）那在你处于最初这四个月阶段的时候，你脑子里是不是已经开始冒出一些想法，关于这个阶段结束后优先级该是什么，并且开始跟 CEO 对齐优先级？还是说你这四个月就是完全埋头猛干，甚至没怎么抽出时间去想这些，一直到之后才顾得上？

[22:17] Chase

Oh, we were absolutely we were chatting about it constantly. I think it's hard to realize what a year ago looked like, right? So I started building kind of an AI agent on a tiny subset of our data in data bricks using their AI and I asked it what's the clinic address and it responded with a madeup address and I was like I mean this is probably March or or April of last year like this is not a long time ago it was I think it was a llama model because we didn't even know that llama 4 was bad at the time and I was like Oh, okay. This is my first experience. I have a bunch of data science experience, but none of the new LLM stuff. And I was like, this is very far away, right? We're not even going to get close. So, I actually had to spend a lot of time working on AI frameworks and figuring out which ones were best. So, you've got I'm using Agno is what it's called. It used to be called FI Data, but you know, you've got your lane chains and all these lane groups and then you've got a couple others that I forget the names at this point, but I went through a bunch of those and I was testing questions and I was doing that in the public. So in our public Slack channel so people could see responses and they were seeing as it was growing and that led to all kinds of ideas. So I was really building where our ops team could see it and our co could see it.

哦，我们绝对是——我们一直在不停地聊这个。我觉得现在很难想象一年前是什么光景。当时我开始在 Databricks 里、用他们的 AI，基于我们数据的一个很小的子集搭了一个 AI agent，我问它『诊所地址是什么』，它给我编了一个假地址出来。那大概是去年三月还是四月，真没多久之前。我记得那是个 llama 模型，因为那会儿我们甚至还不知道 llama 4 很差。我当时就『哦，好吧』。这是我的第一次亲身体验。我有一堆数据科学的经验，但对这些新的 LLM 的东西一窍不通。我就想：这离能用还差得远呢，我们连边都够不着。所以我实际上不得不花大量时间去研究 AI 框架、搞清楚哪个最好。我现在用的叫 Agno，它以前叫 Phi Data；此外你还有 LangChain 那一堆 LangXX 的东西，还有另外几个我现在名字都记不住了。我把这些都过了一遍，一边测试各种问题，而且我是公开地在做——是在我们公开的 Slack 频道里做，这样大家都能看到回答，能看着它一点点变好。这就催生了各种各样的点子。所以我其实是把它建在一个我们的运营团队看得见、我们的 CEO 也看得见的地方。

[23:33] 主持人

Just to interrupt for a sec. So, if I'm if I'm like one of your teammates who's in that Slack channel and I'm and you're like doing testing, am I literally just seeing like question answer question like that's one thread, question answer, that's a new thread and then people can chime in and be like this is wrong, this is wrong, or like this like it that channel starts getting a lot of like side threads going on.

打断一下。假设我是你团队里的一员、也在那个 Slack 频道里，你在做测试——我看到的是不是就是字面意义上的『问题-回答、问题-回答』这样，一个问题一条线程，回答，然后新的问题、新的一条线程，接着大家可以插话进来说『这个错了、那个错了』，或者像那样……那个频道是不是开始挂满了各种旁支的讨论线程？

[23:56] Chase

Yep. And that's how we're doing it to this day.

没错。而且我们到今天都还是这么干的。

[23:58] Chase

It's basically like

基本上就相当于……

[23:59] 主持人

in production. It's basically like eval like an evol like a team eval channel where basically like everyone is like basically grading the output and calibrating based on that.

……在生产环境里。它基本上就像一个 eval——像一个团队评估（team eval）频道，基本上每个人都在给输出打分，并据此做校准。

[24:10] Chase

Yeah. We've got front desk pinging operations operations pinging clinicians saying like hey is this still are we still doing this prenatal test right? And it's like oh no and they're like why is it wrong? and I go and do a search which now I' allowed them to do a search of our vector database and it's in our database somewhere because some old blog on all these resources aren't updated right so this is like another thing you have to talk about is your content management system needs to just be set it once and use it everywhere and you've got to work on these like deep set of evals especially on things you expect to change so that it it flags it and that's just a world that was just we just have the resource and it just goes go stale and you have no idea until someone complains about it because they showed up at the wrong office, right?

对。我们会有前台的人 @ 运营，运营再 @ 临床医生，问：『嘿，这个产检测试我们现在还这么做吗？』结果对方说『哦不，不这么做了』，前台就问『那它为什么答错了？』，于是我就去搜一下——现在我已经开放了让他们可以搜我们的向量数据库——发现它就存在我们数据库里的某个地方，因为某篇旧博客……这些资源里有一堆是没更新的。所以这就引出另一个你必须谈的点：你的内容管理系统（CMS）必须做到『一次设置、处处使用』；而且你得在这些地方下功夫做一整套很深的 eval，尤其是针对那些你预期会变化的内容，让系统能自动把它标出来。否则就会是这样一个世界：你有这份资源，它就那么慢慢过期了，你根本不知道，直到有人跑错了诊所、投诉了，你才发现。

[24:55] 主持人

Yeah. I I mean, you kind of brushed off this part, but I actually think it's really interesting. Um, you basically what you just said is that you basically taught yourself how to run evals and how to like set that the whole system up basically on yourself like you taught yourself that. And as someone who's been meaning to get into evals and just feels intimidating, I'm curious maybe you have any words of wisdom of how um you taught yourself and I know what was your your approach because I think that's pretty unique.

对。我是说，你刚才把这一段轻描淡写地带过去了，但我其实觉得这特别有意思。你刚说的意思基本上就是：你基本是靠自己自学会了怎么跑 eval、怎么把整套系统搭起来——都是你自学的。作为一个一直想入门 eval、又觉得它挺唬人的人，我很好奇你有没有什么心得可以分享——你是怎么自学的，你的路数是什么？因为我觉得这挺独特的。

[25:23] 主持人

Yeah. And and what did you think it was before versus how would what do you think it is now after having kind of done it? But yeah, very curious to learn.

对，还有——你上手之前以为 eval 是怎么回事，真正做过之后又觉得它其实是怎么回事？非常想听听。

[25:29] Chase

Yeah.

嗯。

[25:30] Chase

Yeah. I mean everything in my career is like I just go do it, you know? I would say that like I'm a I'm a builder hacker and I build it and then I build it, right? And it's probably the first build it part is takes a lot longer than building it, right? Because I'm just like spraying. So yeah, I mean eels coming into it I thought were kind of big company BS, right? So it's like, you know, okay, this is just another set of tests, so whatever. And then I first tried them and I was really sure it was big company BS because it was kind of interesting like if you just ask your first couple questions and you get a response. So I'll assume it's a really simple loop, right? You ask your agent a question, you get a response. That response is compared to a response you wrote out. And it doesn't have to be the exact response, right? So, a great example is I had a eval, you know, I go 1 to 10 is how I asked the agent that's the eval. And I was getting a bunch of sevens on appointments, which should just be like almost a one or a zero. And my eval criteria said it should return a list of appointments. and my test account ended up the first appointment passed so there was only one appointment and it was deeing it for it only returned one appointment not a list of appointments so like you learn like oh that looks really dumb right and then you just have to keep working on that so there's a bunch of different questions I have some of which should be closer to 1 to 10 and I'm prompting an agent that says like give it a one or a 10 and here's the criteria and then there's a lot of them that my eval stick right eight and I'm checking between runs and between model changes and everything else to look for drift, but I don't really care that it's at an eight because it just it's always complaining about something, right? Like some form because, you know, again, it's looking at resources that are four pages long on what's a doula, right? Um, which is a a birth helper. And I can't say what the answer is going to be. It's not deterministic. So you kind of just have to give general guidance and then say like if it hits these three important points it's a 7 plus and if it hits these other ones it's an 8 plus and you kind of live with that. So I built out a fully custom eval suite that's looking at timing how long things take to run the tool calls. So I'm running a suite of agents. So it's called a team of agents with a supervisor and there's a bunch of subsp specialcialized agents who all have their own context. Some of those can look up knowledge so they can go look at our vector database. Others can't. So others have access to our APIs for scheduling. So the supervisor routes that person answers supervisor has an output model that is like the chat latest chat model usually and it's formatting it based on their preferences. So the user whenever they're on boarding is getting like do you want concise? Do you want medical terminology? And then as they're talking we're saving those. So the agent actually goes and will save memories and say, "Hey, they didn't like this answer or whatever else and they'll generalize that so their next answer from that output model looks like that." Um, so I have to go and look at that, right? Is it calling the right tools for the kind of question? So in my ev kind of got this huge dictionary that's like here's the question. I actually format the question in like 10 different ways. So I asked Jim and I to give me here's a question give me 10 forms of this question so that it wasn't just like what's the clinic address but you know what's what's the address for my appointment because you know AI goes and looks that up and if they don't look up clinic site they might not find it. So you have to like keep working on that. So it asks 10 it asks the question 10 different ways. It eval based on like a bunch of subset of things and then it's got a bunch of questions on like more um deterministic engineering type things like did it call the right tool? did it return in a certain amount of time, things like that. Or did it try to answer directly without calling a tool, which it could do theoretically as well.

对。我是说，我职业生涯里所有事情基本都是『我直接上手去干』。我会说我是个 builder-hacker（搭建型的黑客），我先把它搭出来，然后再重搭一遍。而且很可能『第一次搭』比后面『真正搭好』花的时间还长，因为我就是一通乱喷、瞎试。所以说到 eval，一开始我以为它是那种大公司才玩的花架子（BS）。就是那种感觉：好吧，这不就是又一套测试嘛，随便啦。然后我第一次上手试的时候，我更确信它是大公司的花架子了，因为它挺有意思的——如果你就问头几个问题、拿到一个回答……我就当它是个很简单的循环吧：你问你的 agent 一个问题，拿到一个回答，这个回答会跟一个你事先写好的标准答案做比对。而且它不必是一字不差的那个答案。举个绝佳的例子：我有一条 eval，我让 agent 从 1 到 10 打分，这就是那条 eval。结果在『预约』这个问题上我老是拿到一堆 7 分，可它本该差不多是 1 分或 0 分。我的 eval 标准写的是：它应该返回一个预约列表。而我的测试账号里，第一个预约其实是过了的（pass），所以只剩一个预约，于是它就因为『只返回了一个预约、而不是一个预约列表』给自己扣分了。你就这样学到：哦，这看起来是真的挺蠢。然后你就得不停地打磨它。所以我有一堆不同的问题，其中一些应该更接近『1 到 10 打分』这种，我给 agent 的提示词就是『给它打个 1 分或 10 分，评分标准如下』；另外还有一大堆，我的 eval 会稳定停在 8 分左右，我会在不同运行之间、不同模型切换之间去检查它，专门盯着有没有漂移（drift），但我并不太在意它是不是 8 分，因为它总归会挑出点什么毛病来抱怨，比如某种格式问题——因为你懂的，它读的可是那种长达四页、讲『doula 是什么』的资源。doula 就是分娩陪护（生产助手）。我没法断定它的答案会是什么，它不是确定性（deterministic）的。所以你只能给一些笼统的指引，然后说：如果它命中了这三个重要点就是 7 分往上，如果它还命中了另外那几个点就是 8 分往上，你就凑合接受这个结果。所以我搭了一整套完全定制的 eval 套件，它还会盯着时延——各种工具调用（tool call）跑起来花多长时间。因为我跑的是一整套 agent。它叫『agent 团队』，有一个 supervisor（主管），下面有一堆各自专精的子 agent，每个都有自己的上下文（context）。其中一些能查知识，也就是能去读我们的向量数据库；另一些不能。还有一些能访问我们的 API 去做排班。所以 supervisor 负责路由，由那个人来作答；supervisor 有一个输出模型，通常就是最新的那种 chat 模型，它会根据用户的偏好来做格式化。用户在 onboarding（引导）的时候，我们会问：你想要简洁的？还是想要医学术语？然后随着他们对话，我们会把这些存下来。所以这个 agent 其实会去存记忆，比如『嘿，他们不喜欢这个答案』之类的，然后把它泛化，让那个输出模型给出的下一个答案就长成他们想要的样子。所以我得去盯着这些看，对吧——它有没有针对那类问题调用对的工具？所以在我的 eval 里，我搞了个巨大的字典，就是『这是问题』。我实际上会把同一个问题写成 10 种不同的问法。我让 GPT 帮我——给我一个问题、生成这个问题的 10 种变体，这样就不只是『诊所地址是什么』，还有『我预约的那个地址是什么』，因为 AI 去查的时候，如果它不去查『诊所地点』这个字段，可能就找不到。所以你得不停地打磨这个。于是它会把同一个问题问 10 种不同的问法，基于一整套子标准来评估；此外还有一大堆更偏确定性的、工程类的问题，比如它有没有调对工具？有没有在一定时间内返回？诸如此类。或者它是不是没调用工具就直接作答了——理论上它也可能这么干。

[29:23] 主持人

Yeah.

对。

[29:23] Chase

So that's EOS.

所以这就是那套 eval（EOS/evals）。

[29:24] 主持人

It's I mean I feel you just went from like level zero to like level 10. Like the level zero being, hey, like where is the address of my clinic to like now like sharing this like full like team of agents with like multiple tool calls and different permissions. No, no. I mean, but it's pretty like it's pretty cool to to just see that evolution and you know, probably in, you know, let's call it like eight months or or whatever, but but it sounds like I want to oversimplify it in a way, but it sounds like the way you learned it was like just start to get a lot of like input output and go through the cycle many times and as you're running into an issue or something that hey, how could I fix this? Oh, maybe I need sub agents because we're hitting too much, you know, we're like filling up our context window too much and that's hurting like accuracy. But it sounds like the way you build the muscle was like just running the cycle over and over again and kind of build like continue to build that intuition and as new problems would come up go out into the world look for a solution test it and keep tinkering.

感觉你一下子从「零级」跳到了「十级」。零级就是「诊所地址在哪」这种问题，而现在你聊的已经是一整套 agent 团队、带多种 tool call、不同权限。看到这个演进过程真的挺酷的，而且这大概也就是八个月左右吧。我可能有点想把它过度简化，但听起来你学会的方式其实就是：先跑大量的输入输出，把这个循环跑很多遍，一旦碰到问题就想「这个怎么解决」，比如「也许我需要 sub agent，因为我们把 context window 填得太满了，这在拖累准确率」。听起来你练出这块「肌肉」的方式，就是把这个循环一遍遍地跑，不断积累直觉，一有新问题就去外面找方案、测试、然后接着折腾。

[30:21] Chase

Yeah. And solutions I would look for um I slowly build out what mattered as well. So for example the length of a response matters a lot. So really um mediocre models have much longer responses because my for things that looked at my knowledge base cuz there was a bunch of useful information. So it would just return a bunch of useful information and it was you know a mix of duplicative information or just kind of repeating information that didn't make sense to add because it was related to something else blah blah blah. So you know I ended up adding that as as a criteria and along with all these different evals medical sentiments. So just understanding like did it respond with something that was medically serious. You know I had an eval based on not just accuracy but also sentiment of the response and did it respond like based on this criteria medical seriousness of the response. So I have all of these happening and now all of these agents that were kind of trained as part of the eval run in real time. So every time someone sends us a slack it sends like new message here's the theme. So, also theme and sub themes are a big thing. I used AI for to build out all of our Zenesk ticket themes. And I had to use AI for about 40 turns to get it to like a list of 12 themes of things we get. So, yeah, each time we get one, it it sends all of that with like medical seriousness. It'll alert us if it's if it's super serious and all of that runs each run basically.

对。而且我在找方案的时候，也是慢慢摸清了哪些东西真正重要。比如说，回复的长度就很关键。那些比较平庸的模型回复会长得多，因为它们去查我的知识库时，里面有一堆有用信息，它就把一大堆都返回回来，混着一些重复内容，或者是那种因为跟别的东西沾点边、其实没必要加进来的信息，等等。所以我后来就把这个也加成了一条评判标准，跟其他各种 eval 一起，比如医疗严重性。就是要判断它的回复是不是涉及了医疗上很严重的情况。我有一个 eval 不只看准确率，还看回复的情绪倾向，以及它有没有按照「医疗严重程度」这条标准来回应。这些我全都在跑，而现在这些当初作为 eval 一部分训练出来的 agent，是实时在运行的。所以每次有人给我们发一条 Slack 消息，它就会发出来「这是新消息，这是它的主题」。对了，主题和子主题也是很重要的一环。我用 AI 把我们所有 Zendesk 工单的主题都梳理出来了，我花了大概 40 轮才让它整理出我们会遇到的 12 类主题。所以每来一条消息，它就会把这些全都带上，包括医疗严重性，如果特别严重它会提醒我们，这一整套基本上每次都会跑一遍。

[31:47] 主持人

So, if you I mean, you you uh you obviously have like a playbook that's running right now. seems to be working well or at least like the system that you built out seems to be serving you well. What has like maybe surprised you about the way that the system is running versus the way you maybe thought it would end up looking when when you were done, you know, when you were getting started? Like what are some things you're like, "Wow, like this is so obvious now that I went through all these iterations, but I had no idea that this was probably like the right way to to to approach this."

所以说，你现在显然已经有一套正在运行的打法了，看起来运转得不错，至少你搭出来的这套系统对你挺管用的。那么这套系统实际跑起来的样子，跟你当初起步、心里预想它最终会长成什么样相比，有没有哪些地方让你觉得意外？比如有没有那种「哇，我折腾了这么多轮之后现在觉得这也太显而易见了，但一开始我压根没想到这大概才是正确的做法」的东西？

[32:22] Chase

Yeah. Um, I mean the first one is the thing we all talk about counseling, which is prompting. Intent is so hard with an LLM and you know, we would give these prompts for, you know, a scheduling agent and I would have to give it these directions and it would keep doing crazy stuff and I'd be like, "What what's going on?" And I'd go in and I'd look and I'd read the words and I'd be like, "Oh, I get how this could be construed in some separate way." Right? So it's like you have to like really look at every single word because you're basically building an engineering system and instead of code which is again fully deterministic it's words and that's just a wild thing because and unless you test it you just don't realize because someone asked something in so many different ways. So like this really I really got down this list because I hooked up our agent. So again this Agno framework kind of hooks up to Slack automatically. It does the databases for me automatically on like my own postgress. nothing hits their server, so it's all very HIPPA compliant. And I hooked it up to Slack so people could ask. And just the way my operations lead would ask, I'd be like, "Oh yeah, this is a totally valid way to ask this question and it's totally failing because of the way I wrote this out." And I mean, and we're only 80% of the way there, right? We will see in real time failures from our patients. And that's why I'm so big on alerting and monitoring. And the app has all kinds of warnings and there's guard rails within the the AI as well that make sure that it always responds with like certain caveats and and you know they have to sign off and all this other stuff. But I'm going to keep seeing that and I'm going to have to keep iterating on these prompts. So I think the biggest thing is just how literal

有。第一个就是我们老生常谈的那个东西——prompt。让 LLM 理解你的意图太难了。我们会给一个调度 agent 写 prompt，我得给它下各种指令，可它老是干出些莫名其妙的事，我就纳闷「这到底怎么回事」。然后我进去把那些文字读一遍，才恍然「哦，我懂了，这句话确实可以被理解成另外一个意思」。所以你真的得逐字逐句去抠，因为你本质上是在搭一套工程系统，只不过用的不是代码——代码是完全确定性的——而是文字，这就很离谱，因为除非你去测，你根本意识不到，毕竟同一件事别人能有无数种问法。所以这一块我真的抠得很细，因为我把我们的 agent 接了起来。这个 Agno 框架能自动接到 Slack，还能自动帮我把数据库建在我自己的 Postgres 上，没有任何东西打到他们的服务器上，所以完全符合 HIPAA 合规。我把它接进 Slack 让大家来提问，然后光是看我的运营负责人怎么问，我就发现「哦对，这问法完全没问题，可它却彻底答错了，纯粹是因为我 prompt 写的方式有问题」。而且我们才走到 80% 而已，对吧？我们会实时看到病人那边冒出来的失败案例。这也是为什么我这么看重告警和监控。App 里有各种各样的警告，AI 内部也有 guardrails，确保它每次回复都会带上某些提醒说明、要用户确认签字之类的。但这种失败我会一直看到，也就得一直迭代这些 prompt。所以我觉得最大的一点，就是

[34:00] Chase

you know a prompt turns into you know almost code right and you just can't be lz fair with it. You can't just say like oh you're a scheduling agent like reschedule. It's like no no you have to like be explain every single step and write out everything really thoroughly.

一条 prompt 几乎就变成了代码，你真的不能对它敷衍了事。你不能只说「你是个调度 agent，去改期吧」，不行不行，你得把每一步都讲清楚，把所有东西都写得特别详尽。

[34:15] 主持人

If you're enjoying this conversation, please check out the links in the show notes to support the podcast. Mark and I do this out of love, but to keep it going, we also need your support. Thanks. And now back to the episode. any anything else that because it it seems like yeah like precision when when it comes to prompting super important and I'm sure you probably become a more precise communicator as a result too

如果你喜欢这期对话，欢迎点开 show notes 里的链接支持我们这档播客。我和 Marc 做这个是出于热爱，但要把它做下去，也需要大家的支持，谢谢。现在回到正题。听起来 prompt 这件事上「精确」确实极其重要，而且我猜你因此大概也变成了一个表达更精确的人，

[34:41] 主持人

because you kind of have to but yeah anything else that that comes to mind

因为你不得不这样。那除此之外，还有没有别的让你印象深刻的？

[34:45] Chase

I think I think the biggest one outside of that was just how capable everything became since gosh I guess GBT5 you know went from a world where we were going be very narrow in our definitions of what AI could do to increasingly expanding and you know if you even look at our clinicians and their use of something like open evidence which is kind of the clinician version of chat GPT right now or the things that they're building in that epic EMR system it's really accelerating so I think it's just a world where the possible just keeps expanding and I think you have to build in the world that there are no limits, right? So, whatever you think a limit is right now, it will probably be gone in the next 6 months. So, whether that's through engineering and saying like we'll never allow a PM to push a feature to production or whether that's, you know, um what your agent can respond to and say, oh, the architecture will always look like this. Just just get rid of that. So, that notion of of knowing what's going to happen is impossible in my mind. Like, this is just accelerating too fast. How do you like h how do you do that like in practice? How do you do that or how do you um implement that that um state of mind that nothing is possible, right? Like how do you how do you plan for that? Is it more just like a speed thing when you see like something that you go go for it or or like how do you live that value?

除了那个之外，我觉得最大的一点就是从 GPT-5 之后，所有东西一下子变得能干太多了。我们从一个得把「AI 能做什么」定义得非常窄的世界，变成了能力在不断扩张的世界。你哪怕看看我们的临床医生，看他们用 Open Evidence 这类东西——就相当于临床医生版的 ChatGPT——或者看他们在 Epic 那套 EMR 系统里搭的东西，真的是在加速。所以我觉得这就是一个「可能性边界不断外扩」的世界，你必须假设根本没有上限地去搭东西。所以不管你现在觉得哪儿是极限，很可能六个月内它就没了。无论是在工程层面说「我们永远不会让 PM 把功能推上生产」，还是你的 agent 能回应什么、你说「架构永远都会长这样」——统统把这种想法扔掉。所以那种「知道接下来会发生什么」的念头，在我看来是不可能的，这东西加速得太快了。（主持人）那你在实操里到底怎么做到的？或者说你怎么把「没有什么是不可能的」这种心态落地？你怎么为此做规划？还是说更多是一种速度上的东西，一看到什么就直接冲？你是怎么践行这个价值观的？

[36:14] Chase

Yeah, it's definitely a mix of things, right? I I think the most important moat is speed. So a lot of focus on the infrastructure that allows you to move things quickly. So an example is like I can't have pieces of my data not accessible. They all have to be in the same place. So that's kind of your classic thing that everyone says they want to do and they they very rarely do. And you know I've got to have PHI protection. So personal health identification identifying information. Um and all of that stuff has to be built so that I can add it wherever. So I think infrastructure for speed is super important. And then I think how you build the team is super important. So we had a mobile app developer who was maintaining the app who got there and I started writing out this huge new AI feature to allow people to chat with basically our our brain of our app. And I spend gosh six weeks I mean it wasn't a huge amount of time right like kind of front to back. And I put out you know a 10,000 line PR or something like that. and he was basically like I I'm not going to review this. Like I I don't feel comfortable with AI in this way. And we had to move on. I was like, oh, cool. Like this is a great learning for us. And I kind of had to push that on a very small team, but four or five people of like this is going to be the world. I, you know, spend all of my time reading amazing engineers from amazing companies and like this is where we're moving. So if you're just someone who's not comfortable reviewing AI PRs, you know, AIdriven PRs, you're going to be in rough shape. So I think if you're a company that was around before LM with a large tech team, those things pop up, right? You've got this amazing leader who's maybe defensive, maybe worried about things and they slow things down everywhere. They ask they find a little a little small thing that is wrong, but you know, you you get rid of a speed up of five or 10x speed for an improvement in in safety. or it's not even safety, it's more of like a a user satisfaction of 1%, right? There's some small bug or something that happen.

对，这肯定是好几样东西的结合。我觉得最重要的 moat 就是速度。所以要花很多精力在那些能让你快速推进的基础设施上。举个例子，我不能让我的数据有一部分是取不到的，它们必须都在同一个地方。这就是那种人人都说想做、但很少真做到的经典事情。而且我还得有 PHI 保护，也就是个人健康身份信息。所有这些东西都得搭好，这样我才能把它加到任何我想加的地方。所以我觉得「为速度而建的基础设施」极其重要。然后我觉得你怎么搭团队也极其重要。我们有一个负责维护 App 的移动端开发，我当时开始写一个巨大的新 AI 功能，让用户基本上能跟我们 App 的「大脑」对话。我大概花了六周吧——其实从头到尾也不算特别久——然后我提了个一万行左右的 PR。他基本上就是说「我不打算 review 这个，我对这样用 AI 感到不舒服」。于是我们只能分道扬镳。我当时想「哦，行，这对我们是个很好的教训」。我不得不在一个很小的团队里推这件事，也就四五个人，去传达「未来就是这样」。我花大量时间读那些来自顶尖公司的顶尖工程师写的东西，趋势就是往这个方向走。所以你要是那种没法接受去 review AI 写的 PR、AI 驱动的 PR 的人，日子会很不好过。所以我觉得，如果你是一家在 LLM 出现之前就存在、还有一支庞大技术团队的公司，这类问题就会冒出来：你有个很牛的技术负责人，可能有点防御心态、可能对很多事担忧，于是到处拖慢进度。他们会揪出一个很小的错误，但你为了那点安全性上的改进——甚至都算不上安全，更像是 1% 的用户满意度提升——就把五倍十倍的提速给牺牲掉了。就为了某个小 bug 之类的东西。

[38:22] 主持人

What do you actually What do you actually think the right answer is to an engineer who is now starting to face a deluge of PRs with like massive numbers of lines of code where it's clearly prompt, it's clearly coming from people who are using AI to code. Like we sure we could tell that engineer they have to get with the times but like what what does that what does that mean to get with the times like what should they be doing in your opinion?

那你实际上觉得，对于一个开始面对海量 PR、每个都是巨量代码行、明显是 prompt 出来的、明显来自那些用 AI 写代码的人的工程师来说，正确答案到底是什么？我们当然可以跟那个工程师说他得「跟上时代」，但「跟上时代」具体是什么意思？在你看来他们到底该怎么做？

[38:48] 主持人

Yeah. Like should they just not review them at all or should they use AI to review them and complement the review process? Like what do you think?

对，比如他们干脆完全不 review 了？还是应该用 AI 来 review、作为 review 流程的补充？你怎么看？

[38:54] Chase

Yeah. I mean we should be in the middle. So although I'm a a very techforward person, right? We should always find a good middle ground here. Um I I can tell you what we're doing. So one I'm breaking up PRs. So, I'm kind of creating like a stacked PR with features. You know, they all merge into a main branch, but there's six kind of separate PRs that all have kind of a more core theme of what they're doing. That's not always testable because you'd have to go back to the old school engineering way of like really phasing things out and spending a lot of time on that so that you could test each approach. And I don't think that's worthwhile, but at least you have the code kind of in one place so they can understand like what's this one doing? they can go look through that, take the main branch and test it. So I think a big one is is testing, right? Use cases, test cases, things like that. We're using a lot more test. So everyone I think talks about this, but you know the amount of test you can write compared to what you used to do are are insane. Looking at those and understanding like do these actually test the functionality is really important. And then uh you know I think the thing that we like to say on my side one right now I'm not I'm being a lot more thorough with anyone who doesn't have an engineering background. So probably smaller features for anyone who doesn't and more of a idea board prototype for anyone who like has a big feature they want to iterate on. We've definitely enabled that but that's not really going to PR, right? That's going to like

对，我觉得应该取个中间地带。虽然我是个非常拥抱技术的人，但我们总该找到一个好的折中点。我可以跟你说说我们在怎么做。第一，我在把 PR 拆开。我会搞成那种带功能的 stacked PR，它们最终都合进主分支，但会拆成大概六个相对独立的 PR，每个都有一个更聚焦的核心主题。这不一定总能测，因为要测的话你就得回到老派工程那套——把东西认真地分阶段、花大量时间在上面，才能逐个方案去测。我觉得那不值得，但至少你能把代码都放在一处，让人能看明白「这个 PR 是干嘛的」，可以进去翻一翻，拿主分支跑一下测。所以我觉得很重要的一点是测试，对吧？用例、测试用例这类东西。我们现在用的测试多多了。这个大家应该都在讲，但你现在能写的测试量跟以前比简直离谱。去看这些测试、搞清楚它们到底有没有真正测到功能，这非常重要。然后，我们这边现在爱说的一句话是——我对任何没有工程背景的人都会更谨慎得多。所以对没工程背景的人，功能一般会更小；而对那种手上有个大功能、想拿去迭代的人，更多是给他一个「想法板」式的原型。这个我们绝对是放开做的，但那其实不会走到 PR，对吧？那更像是

[40:23] 主持人

you mean like a market like a marketing person or like a sales per like like an op. Okay.

你是说像做市场的人、或者做销售的人、或者运营的人？好的。

[40:27] Chase

Yeah. Even product that doesn't have like a engineering background, it's more focused on smaller PRs are perfectly fine. Changing some content, changing the look of something, simple stuff is always fine, right? But for something that's a huge feature, I'm usually leaving that to like this is the sandbox. You're playing in it. You're showing it off. You're getting the idea out there. And then we're we can hand that off, right? the code might be handed off or it might be like we're going to start you know kind of from the scratch and and do it with usually AI but through an engineer just because you can really get off track right so just how our engineering principles are in terms of scalability if you don't know anything about that AI can just start going down crazy paths and you don't realize it because you you're not following the code as you're going so I think our our view is can you say that I didn't write the code but I stand by it so that's like our our big thing.

对。哪怕是那种没有工程背景的产品人，重点也是放在更小的 PR 上，完全没问题。改点文案、改改某个东西的样子、简单的东西，永远都行。但要是那种巨大的功能，我通常会把它留在「这就是沙盒，你在里面玩、你去展示、把想法搞出来」这个层面。然后我们再把它交接出去，对吧？可能是代码交接出去，也可能是我们准备从头再来一遍——通常还是用 AI，但要经过一个工程师，因为你真的很容易跑偏。所以就看我们的工程原则在可扩展性上是怎样的：如果你对那些一无所知，AI 完全可能一路走上一些疯狂的岔路，而你还没察觉，因为你并没有边写边跟着看代码。所以我觉得我们的观点是：你能不能说「这代码不是我写的，但我为它背书」？这就是我们最看重的一点。

[41:23] 主持人

So just to yeah because I think it's a really interesting question is like what is the delta between like that PM that is not very technical and you right and like why you are allowed why why are you comfort comfortable with merging like a thousand 10,000 lines of code to the the main branch and why like it's a terrible idea for that nontechnical PM it is and it sounds like you think you still can probably understand each line of code and how they relate to the feature. Sounds like you might have like an understanding of like the pitfalls of like when you like quoteunquote bite code an app like scalability, security, like those are all things that are very top of- mind of you, but it might be an unknown unknown to a PM who's not technical. Is that more is that more or less the

我插一句，因为我觉得这是个特别有意思的问题：那个不太懂技术的 PM 和你之间的差距到底是什么？为什么你就可以、你就能坦然地把上千行、上万行代码合进主分支，而对那个不懂技术的 PM 来说这就是个糟糕的主意？确实是。听起来你觉得你大概仍然能看懂每一行代码、以及它们跟这个功能的关系。听起来你可能对那些坑很有数——就是当你「氛围编程」一个 App 时会踩到的坑，比如可扩展性、安全性——这些对你来说都是时刻挂在心上的，但对一个不懂技术的 PM 来说可能是「未知的未知」。是这样吗，大致上？

[42:02] Chase

I think it's I think it's grayer than that. I don't think it's a terrible idea for a PM and I think it it's absolutely something that I'm teaching my PMs to get to a spot where we're comfortable with. So it I don't think you have to be fully technical, but I do think there's a I think for me there's the same gate as anyone else. There's a senior engineer in that language, that architecture, that app that understands what we're doing, has been in part of all these long-winded architecture programs or uh back and forths and they're going to review things. Whether that comes from me or from a PM, I actually don't have a problem with. It's more about an expectation set. So for a PM that's like what I don't want is them to be so worried about the feature going to PR ready whenever they're like I don't know anything about that. I actually just think it's like opens their mind more to say like what you're worried about here is what it looks like how it feels the user experience and then taking that and kind of bundling it into what we want to build and here's my prototype. And sometimes that really wildly different experience is actually not complex in code, right? It's a lot of front-end code or moving things around. Totally different experience, but actually the backend stayed the same. There's no security implications. That should pretty much go through. So, it's on the the the engineer at that point that picks it up to say like, is this more of a review or is this more of a rebuild or did they kind of halfass it, right? They demoed it. They stubbed a bunch of data because that's what they wanted to do. They didn't actually build the feature. So I don't think there's a huge gap, but I do think that there's the same wall either way, which is like someone who's worried about our scalability and understands it before it's going to production. So it's so interesting because like I'm think I'm I'm connecting the dots like what we talked about earlier with like the convergence of the um the CTO and CPO into like this the maybe it's a CPO in the future that also oversees engineering or CPTO as a point in time. And what I was just thinking about is there's there is going to be a question at the end of this, but it was about how like as the the roles start to blend at the ground level, at the IC level where like a PM can put up a 10,000 line PR and the engineer still kind of is like technically the person that should review that. And if they're not comfortable reviewing that, there's like um stalemate where like like that's uncomfortable, right? Like that's going to be uncomfortable for basically everyone to to iron out while that that exists. And so my question for you is if someone is listening and they are kind of in your shoes as the product leader who's also overseeing engineering and they want to build a culture where the product managers are empowered to operate the way that you've been describing which is to take take this the initial stab at the prototype or even suggesting some production like pushing a PR to to be reviewed on the path to production. How are you thinking like what's what would your advice be for how to like almost like think about what engineers should be spending their time on to make that possible? Like should engineers start to like spend more time on creating the conditions in which it's like safe for the PMs like push the PRs to production instead of worrying too much about what it means to like review the specific PRs they have to right now like or something else.

我觉得这事儿比那更灰色一些。我不觉得对 PM 来说这就是个糟糕的主意，而且这绝对是我正在教我的 PM 们去达到的一个我们能放心的状态。所以我不觉得你必须是个彻底的技术人，但我确实觉得——对我来说——这里有跟其他任何人一样的一道关卡：在那门语言、那套架构、那个 App 里有一位资深工程师，他懂我们在做什么、参与过所有这些冗长的架构讨论和来回，由他来 review。这道 review 是来自我、还是来自 PM，我其实都无所谓，更重要的是设定好预期。所以对一个 PM 来说，我不希望的是他们特别焦虑于这个功能得「达到可提 PR 的状态」，可他们心里其实「对那个我啥都不懂」。我反而觉得，这更能打开他们的思路，让他们意识到「你在这里该关心的是它长什么样、给人什么感觉、用户体验如何」，然后把这些拿过来、打包进我们想做的东西里，说「这是我的原型」。而且有时候那种天差地别的体验，在代码上其实并不复杂，对吧？很多都是前端代码、或者只是把东西挪来挪去。体验完全不同，但后端其实没变，没有任何安全影响，那这个基本就该放行。所以到那个点上，就该轮到接手的工程师去判断：这更像是一次 review，还是一次重建，或者他们是不是敷衍了事——他们做了个 demo，塞了一堆假数据，因为他们想要的就是这个效果，其实并没有真把功能做出来。所以我不觉得这里有多大的鸿沟，但我确实觉得无论哪条路都有同一堵墙，那就是：有一个关心我们可扩展性、并且在上生产前把它弄明白的人。这就特别有意思，因为我在把点连起来——跟我们前面聊的 CTO 和 CPO 融合是相通的，也许未来是一个同时管工程的 CPO，或者某个时间点上叫 CPTO。而我刚在想的是——这后面会有个问题，但它是关于：当这些角色开始在最基层、在 IC 层面融合时，比如一个 PM 能提一个一万行的 PR，而工程师在技术上仍然算是那个「应该去 review 它」的人。可如果他们不情愿 review，就会出现一种「僵局」，那种局面很别扭，对吧？基本上对所有人来说，在这种状态存在期间去磨合它都会很别扭。所以我给你的问题是：如果有个听众，处境跟你差不多——既是产品负责人又管着工程——他想打造一种文化，让产品经理有权按你描述的方式去干活，也就是先出原型的初稿，甚至去提一个走向生产、待 review 的 PR。你会怎么想？你的建议是什么，关于怎么去想「工程师该把时间花在什么上」才能让这成为可能？比如工程师是不是该开始花更多时间去创造那种条件——让 PM 把 PR 推向生产变得「安全」，而不是太纠结于此刻手上那些具体 PR 该怎么 review？还是说别的什么？

[45:25] Chase

Yep. Yeah. I think that's I think there's a bunch of important conversations there if you're especially if you're taking over something legacy because you can't allow the way things used to be done to to slow you down, right? It's just a new world. There's enough really smart engineers that I really trust that are like this is it's different, right? Since November, it's different. We all see

是的。我觉得那里面有一堆很重要的对话，尤其如果你接手的是一个有历史包袱的东西，因为你不能任由「过去的做法」拖慢你，对吧？这就是个新世界了。有足够多我特别信任的、非常聪明的工程师都在说：这不一样了，对吧？自从去年 11 月起，就不一样了，我们都看得到。

[45:48] 主持人

and just for just for context not to interrupt you but for context like I know multiple product leaders who have come into new organizations and one of the first major observations is wow our engineering team is slow to like adapt to like this new way of working like there so that I am I am hearing this question coming up a lot. Yeah. And I think it's a misnomer to say this, like I don't think this is true, but one of the things that I used to say is like uh if your salary depends on something being true, it's going to be true. And I think there's a lot of people who think that way, right? So I spent my whole life learning to write code in this format with these curly brackets. And now something else does that for me. And that's scary. And I get that. And you know, I'm I'm more of an abundance thinker, right? This is an accelerator for us. It's not something that we need to cut out a bunch of jobs. Although for for companies that aren't accelerating and aren't abundance thinking, they will be able to. There's never been an invention that allows us to build more things that did anything but add more jobs. You know, we didn't just build less. All we did was build more, right? We add test where we didn't used to add test. We add features where we used to not do them. All it does is open up more worlds so companies can do more. So I mean for that I think the original question was just around how do we get sorry say that I don't want to say it wrong. It's basically like or Mark I'm curious what you were hearing in the question you were going to

补充一点背景，不是打断你，只是给点背景：我认识好几位刚进入新组织的产品负责人，他们最先注意到的重大观察之一就是「哇，我们工程团队在适应这种新工作方式上太慢了」。所以这个问题我确实听到很多。（Chase）对，而且我觉得这么说其实是个误称——我不认为它是真的——但我以前常说一句话：如果你的薪水取决于某件事为真，那这件事在你眼里就会是真的。我觉得很多人是这么想的，对吧？我花了一辈子学着用这种格式、用这些花括号写代码，现在有别的东西替我干了，这很吓人，我理解。我更是个「丰盈思维」的人，对吧？这对我们是个加速器，不是那种「我们得砍掉一堆岗位」的东西。虽然对那些没在加速、也不是丰盈思维的公司来说，他们确实会砍。从来没有哪项发明是让我们能造更多东西，结果却不是增加更多岗位的。我们从来不是造得更少，我们做的全是造得更多，对吧？我们在以前不加测试的地方加测试，在以前不做的地方做功能。它做的只是打开更多的世界，让公司能做更多的事。所以关于这个，我觉得最初那个问题是——抱歉，我怕说错，你再说一遍——大概就是……或者 Marc，我很好奇你从那个你要问的问题里听到的是什么。

[47:13] 主持人

um yeah I mean I think the question essentially was like what should the engineers be doing to enable this this world that you described right where PMs are are at the very least you know building prototypes in a sandbox in in an ideal world maybe they're even committing PRs to the main branch right like what like in other words like the question is like what is the highest leverage activity that the engineering team could be doing in this new world.

嗯，对，我觉得那个问题本质上是：工程师们该做些什么，来实现你描述的那个世界——在那个世界里，PM 至少是在沙盒里搭原型，而在理想世界里，也许他们甚至能往主分支提 PR。换句话说，问题是：在这个新世界里，工程团队能做的「杠杆率最高」的活是什么？

[47:40] Chase

Yeah. And I think that's infrastructure to me. So that's that's a mix of like hardcore infrastructure that AI doesn't touch, but also infrastructure of every repo. So how is it set up? How are we explaining it? So, we're actually doing an AI week next week and our engineers are going to spend their entire time kind of looking through all of our repos going through and understanding what do we need to explain, what context does AI need about how we engineer, where our company is headed, our type of partners and their type of connections, right? So, API versus there's a bunch of esoteric things in in healthcare like fire and things like that. And they're just going to go through and explain that world and then you got to do a bunch of setup, right? So what I want to avoid and and a big pitch of mine is like compared to other healthcare companies, they're going to spend two years in vendor relationship to sign one model provider, right? And they're like it's so secure and in reality like I can sign I am literally using Vert.Ex AI on Google fully HIPACO compliant but I can use like 30 different models, right? And I'm just able to switch between them with no vendor arrangement. So I think engineering needs to set us up for that. So the right tools, how's everyone doing it? Set up sandboxes. Make sure that you're able to kind of by code a feature that has no chance of screwing something up in in the real world, right? Make sure that all of your areas that are super sensitive, you're splitting up into like their own kind of functional areas. So you can quickly see, oh, what what are you doing dealing with login data or why are you dealing with patient data here? So a lot of those things I think are super important like the infrastructure of how we code moving it from pure code to to English sometimes is pretty

对，我觉得对我来说那就是基础设施。这是一个组合：既有 AI 碰不到的那种硬核基础设施，也有每个 repo 的基础设施——它是怎么搭的、我们怎么去解释它。我们下周其实就要办一个「AI 周」，我们的工程师会把全部时间花在把我们所有 repo 翻一遍、理解「我们需要解释什么、AI 需要哪些关于我们怎么做工程的 context、我们公司往哪走、我们的合作方是什么类型、他们的连接是什么类型」，比如 API 之类的。医疗领域里还有一堆很小众的东西，像 FHIR 这种。他们就会把那个世界梳理清楚、解释明白，然后你还得做一堆搭建工作。我特别想避免的、也是我一个大主张，就是：跟别的医疗公司比，他们会花两年时间搞供应商关系、就为了签一家模型提供商，对吧？他们觉得「这样特别安全」，可现实里我完全可以直接签——我实实在在就在用 Google 的 Vertex AI，完全 HIPAA 合规，可我能用大概 30 种不同的模型，我能在它们之间随时切换，不需要任何供应商合约。所以我觉得工程要为此把我们的底子搭好：合适的工具、大家都怎么用、把沙盒建起来、确保你能「氛围编程」出一个绝无可能在真实世界里搞砸什么的功能，确保你把所有超敏感的区域都拆分成各自独立的功能模块，这样你就能一眼看到「诶，你在这儿动登录数据干嘛」「你为什么在这里处理病人数据」。所以这类事我觉得都极其重要，比如「我们怎么写代码」这套基础设施——把它有时候从纯代码搬到英文，这挺

[49:21] 主持人

when you say spending time explaining sorry Mark just like a quick clarification what it came to mind is almost like making sure almost every file in the repo has like a readme or something like that like like the engineers making sure that any AI going and navigating the the repo will know like the exact purpose and like context for like the files it might encounter. counter along the way. Is that is that kind of what you're describing?

你说「花时间去解释」——抱歉 Marc，就快速澄清一下——我脑子里冒出来的画面，差不多是要确保 repo 里几乎每个文件都配一个 readme 之类的东西，就是工程师去确保任何来导航这个 repo 的 AI，都能知道它一路上可能碰到的那些文件的确切用途和 context。你说的大概是这个意思吗？

[49:45] Chase

Yeah, AI is getting better. So, you have to do less and less kind of explaining because it's able to fit 17 files in its memory or its context window and then all of a sudden it's like pretty easy to find things. But a lot of these they're searching, right? They're just running like a GP search on your repo on these big repos. And you kind of need to provide almost like a um like an index for them. So, you need to say like here's how we store things. we named them correctly. So all these things that were like kind of silly for a long time. I used to have so many arguments with my engineers about misnaming something and I even had a company where I just fully disagreed that they thought we should make a semantic model of the business world and write the engineering in that way. So like you know the concepts were the same in the operations team as the actual code and I was like we were wasting so much time here and it's kind of funny because I actually think that's really important for AI is like understanding what is going on and not just these weird names or these functions that are like scattered everywhere. you actually kind of have to put things together in one file, point and say where it's going and you know there's just things to make it less errorprone let alone just context right what is this company what are we doing where are we heading what is our status what is our stage how many people are there are we letting PMs code right how should we do comments how should we do PRs all this stuff that you kind of just build a culture around and writing it down was a waste of time because it's we have a good engineering culture, just follow it. You don't have to make an SOP for everything. It's really annoying to do that. And

对，AI 在变强，所以你要做的这种「解释」越来越少了，因为它能把 17 个文件塞进它的记忆、它的 context window，然后一下子找东西就相当容易了。但很多时候它们是在搜索，对吧？它们就是在你的 repo 上跑一个类似 grep 的搜索，在这些大 repo 上。你多少得给它们提供一个类似「索引」的东西。所以你得说清楚「我们东西是这么存的、我们命名是对的」。所以那些长期以来看着有点傻的讲究——我以前跟工程师为「某个东西命名错了」吵过无数次，我甚至待过一家公司，我完全不认同他们的做法，他们觉得我们应该给业务世界建一个语义模型、然后照着那个方式去写工程代码。这样一来，运营团队里的那些概念就和实际代码里的是一致的，我当时觉得「我们在这上面浪费了太多时间」。有点好笑的是，我现在其实觉得这对 AI 恰恰很重要——就是让它搞清楚到底在发生什么，而不是面对一堆怪名字、或者散落得到处都是的函数。你确实多少得把东西归拢到一个文件里、指明它往哪走，这样能少出错，更别提 context 了——这是家什么公司、我们在做什么、往哪走、现状如何、处在什么阶段、有多少人、我们让不让 PM 写代码、注释该怎么写、PR 该怎么弄，所有这些你多少是靠一种文化沉淀下来的东西。以前把它写下来是浪费时间，因为「我们工程文化很好，照着做就行，你不用给每件事都写个 SOP」，那样做真的很烦。而且

[51:22] Chase

use your common sense or whatever, right? But AI has no

用你的常识判断就好，对吧？但 AI 没有

[51:25] Chase

it's like Yeah, exactly. So now it's like you kind of have to do some of these things. So that some of these companies that are super verbose are really well set up for letting non-engineers write code. Um, and I think we're that's where we're moving as well. It's like a lot of time. So, we're probably do a quarterly AI day, which is just like clean up context for AIS, and a yearly AI week, where we're kind of just focused again, it's just and I don't even know what that'll be, right? I cannot predict the future here, but it's just focused on making AI work better for technical and nontechnical folks together.

就像——对，正是如此。所以现在你多少得把这些事做起来了。所以那些特别啰嗦、什么都写清楚的公司，反而为「让非工程师写代码」做好了很好的准备。我觉得我们也正往那个方向走。这挺花时间的。所以我们大概会搞一个「季度 AI 日」，就是专门给 AI 清理 context，再加一个「年度 AI 周」，我们又是专注在——我甚至都不知道那会是什么样，对吧？我在这儿没法预测未来——但就是专注于让 AI 对技术和非技术的人都能更好地一起工作。

[52:00] 主持人

Yeah. I mean I I mean it seems like context engineering is kind of like the key like bottleneck right now that I think a lot of people are engineering with like given like that all these like new powerful models have like smaller context windows and from my like from my personal experience I'm also like struggling like creating like context that's like you know like coding agent agnostic like that's like a whole other thing. So yeah, it's interesting like it seems like the world in which we're moving to is like a lot of like most of the engineers are moving to like become to like work for the platform team and the platform team is basically like work you know developer experience engineering content almost like they're working at a really high level of abstraction.

对。我是说，看起来 context engineering 现在差不多是那个关键瓶颈，我觉得很多人都在围着它做工程，毕竟这些新的强力模型 context window 反而更小了。从我个人经验来说，我也在苦苦挣扎——比如去创造那种「跟具体 coding agent 无关、通用」的 context，那又是另一件完全独立的事。所以挺有意思的，看起来我们要走向的世界是：大部分工程师都在往「去平台团队干活」这个方向走，而平台团队基本上就是做开发者体验、工程内容这类事，几乎是在一个非常高的抽象层级上工作。

[52:39] 主持人

Yeah. And then like PMS and maybe other customerf facing teams are maybe becoming like you know eventually in a dream world would become like the feature teams right that they're actually like building the the front end and and the thing end to end. Um, so I think that's like kind of the analogy that I have in my head. So it's almost like every every like engineer that's technical like is becoming like that platform engineer.

对，然后 PM、也许还有其他面向客户的团队，也许就变成了——在梦想中的世界里，最终会变成「功能团队」，对吧？他们才是真正端到端把前端和整个东西搭出来的人。所以我脑子里的类比大概就是这样。所以几乎是每一个懂技术的工程师，都在变成那种「平台工程师」。

[52:59] Chase

I think it's every engineer who's technical and not interested in the product as much. Like most engineers are just becoming part of that feature team, right? So you've got product people with a a great sense of things and they've hone that and you've got engineers with a great sense of architecture, speed, how things scale and you just throw them together, right? So you're both kind of working on something like it's a wild world where you can have full-on bake offs with a product and two engineers all building the same feature three different ways and like it doesn't take any time. it takes three days to get it to there and then that's how you build your PRD and then you just go off that best one. So you can kind of create a model of you know human in the loop agents. This is what we do with chess right is like chess is human plus AI is stronger than AI alone. So I think a lot of your engineers will absolutely be stubbing out some code and doing features all the time. I I do that as an you know I again as a trained engineer like a lot of my a lot of my feel for my favorite repositories that I I've written from scratch or I'm super involved in I will write out some of the code. So I'll just stub a couple functions in a couple different places and point it to there to kind of get started and say like you know here's how I want this to go and it can understand the flow a lot better whenever I lay it out that way. So you know I read code as well. It's not just all V coded.

我觉得是每一个既懂技术、又对产品本身没那么感兴趣的工程师。大多数工程师都在变成那个功能团队的一部分，对吧？所以你有一群对「事情该怎样」有极好直觉、并且把这直觉磨得很利的产品人，你还有一群对架构、速度、东西怎么扩展有极好直觉的工程师，你把他们凑到一块，对吧？你俩就一起做某个东西。这是个很疯狂的世界，你可以搞一场彻底的「同题竞标」——一个产品加两个工程师，用三种不同的方式做同一个功能，而这几乎不花时间，三天就能做到那个程度，然后你就用这个来写你的 PRD，接着挑最好的那个继续干。所以你多少能造出一种「human in the loop」的 agent 模型。这跟我们下国际象棋是一样的道理——人加 AI 比 AI 单干更强。所以我觉得你很多工程师绝对会一直在打桩式地写代码、一直在做功能。我自己作为一个受过训练的工程师也这么干，我对那些我从零写起、或者深度参与的最爱的 repo，很多的「手感」都在于我会亲手写一部分代码。我会在几个不同的地方打几个函数的桩、把它指向那里当作起点，说「我想让它这么走」，我这样把它铺出来之后，它就能把整个流程理解得好很多。所以我也是会读代码的，并不全是「氛围编程」出来的。

[54:26] 主持人

Yeah. Where my head is going now and Ben feel free to take a different direction is

对。我现在脑子转到的方向是——Ben 你要想换个方向也随意——

[54:32] 主持人

man like how do you prioritize what to hire for? Because like there's so many dimensions that you can optimize for and I I think you're like a weird unicorn in the like most flattering way possible. Like you are technical, you are have insane domain expertise like you've been in this space for a long time and you are

天啊，你到底是怎么排优先级、决定该招什么样的人的？因为可以优化的维度实在太多了。我觉得你是那种——用最恭维的说法——罕见的「独角兽」：你懂技术，你有惊人的领域专长，你在这个领域深耕了很久，而且你还……

[54:52] 主持人

excellent hair. really really good hair.

发型很棒。头发真的真的特别好。

[54:54] 主持人

Really really good hair,

发型也是真的好，

[54:55] 主持人

which is most important.

这才是最重要的。

[54:56] 主持人

Yeah. Vegan, but we won't get we won't get we won't get into that.

对。还是个 vegan，不过这个咱们就不展开了。

[55:01] 主持人

The only downside,

唯一的缺点，

[55:03] 主持人

but also you you are you're a hacker by nature. Like you're you're like you are very quick. You're you like to build things and you optimize for speed. Like you have like kind of that startup vibe to you. And then you also like are like a very curious like self-arning, self-dactic like person, right? Right. And and I those are all like to me like are like wow really important skills to have um in this new world that you're just describing. But I think it's going to be really hard to find other chases. So So what are you optimizing for? Are you optimizing for like the technical plus speed? Are you optimizing for domain expertise? Are you optimizing for like openmindness to learn and use these tools?

但话说回来，你骨子里是个 hacker。就是你反应特别快，喜欢动手做东西，一切以速度为先，身上有那种创业公司的劲儿。而且你还是那种特别好奇、爱自学、自学成才的人，对吧？没错。在你刚才描述的这个新世界里，这些在我看来都是超级重要的能力。但我觉得要再找到几个像 Chase 这样的人会非常难。所以你到底在优先看重什么？你是优先看技术能力加速度？还是看领域专长？还是看那种愿意去学、去用这些工具的开放心态？

[55:39] 主持人

Also as you answer that, can you just talk about how big the team is right now? Just to like set the tone.

另外你回答的时候，能不能顺便讲讲现在团队多大？给大家先垫个底。

[55:43] Chase

Yeah, for sure. So for that last for the well, I'll start with the easy one which is the big the team. So we're small. We have two PMs, one engineer that's fully dedicated outsourced, one head of engineering, and then like a half engineer, and then a me who's I spend 80% of my time as an engineer. So, I'm closer to an FT engineer than I am filling the CTO role. Um, you know, I'm doing I do fundraising decks and vendor calls and all these other things, but in reality, like I spend most of my time as an engineer. So something like three and a half engineers, two PMs.

当然可以。那我先从简单的说起，就是团队规模。我们很小。两个 PM，一个全职外包的工程师，一个工程负责人，再加上算半个工程师，然后就是我，我 80% 的时间都在写代码。所以我更像是一个全职工程师，而不是在扮演 CTO 的角色。当然啦，我也做融资路演的 deck、跟供应商开会、还有各种杂事，但实际上我大部分时间都在当工程师。所以大概是三个半工程师、两个 PM。

[56:19] 主持人

That's a wild ratio, by the way.

顺便说一句，这个比例也太夸张了。

[56:22] Chase

Yeah, it's because our PM team is doing ops. You know, as we raise money next time, what we'll actually do is take a lot of ops work that's touching technology, but it's really just ops work with a, you know, with a SAS type front end. So this is the EMR speaking again, Epic speaking again. We'll put that more into ops and we'll do more product work. Um but yeah uh a lot of the time is spent on ops product ops and it it product ops can stay around as product ops or it can go into ops. I don't have a strong opinion but it's product ops um is where a lot of our team is spending their time even though they're traditional product people. To answer your other question I think the last part you hit on about like openness but I like to call it like misfits. So, I usually just look for people who aren't really big into following the rules and what they've been given because the rules keep getting torn down. And in the world of AI, if you're someone who like really latches on to a way of doing things, it really makes it likely that we're going to fall behind. So, I think it's important each role and what we need to do with it will guide my technical versus product mindset. So, that's just role dependent. I definitely think we will have very few product people who don't have technical chops. So that just means like usually a technical product manager is their previous title. Some will be engineers. Those engineers might fill a product role. They might fill an engineering role, but they'll be expected to write code either way. Um and pretty much all my product will be expected to write code. But in reality, what I really am searching for isn't your technical ability because I think that can go up or down for the role. It's really just someone who's I wouldn't say disagreeable because there's a lot of those too that are just disagreeable on purpose, but it's just someone who's always kind of been like there's got to be a better way to do this. And if you're searching on the internet, you find them a lot faster nowadays.

对，那是因为我们的 PM 团队其实在做运营。等我们下一轮融到钱，真正要做的是把很多牵扯到技术、但本质上就是运营的活儿拆出来——它无非是套了个 SaaS 前端的运营工作。这又要提到 EMR 了，又是 Epic 那套东西。我们会把那部分更多地划到运营去，然后我们就能多做点真正的产品工作。不过没错，现在很多时间都花在运营、产品运营上，而产品运营这块可以继续叫产品运营，也可以并进运营，我对此没什么强烈的看法。但产品运营确实是我们团队大量精力所在，哪怕他们本身是传统意义上的产品人。回到你另一个问题，你刚才提到开放性那部分——我更愿意管这个叫「misfit（异类）」。我通常会找那种不太买账、不甘于照着现成规则办事的人，因为这些规则一直在被推翻。在 AI 这个世界里，如果你是那种特别死磕某一种固定做法的人，那你极有可能落在后面。所以我觉得，每个岗位、以及我们要用它去做什么，会决定我到底更看重技术还是产品思维。这是看岗位定的。我很确定的一点是，我们几乎不会要没有技术底子的产品人。所以一般他们之前的头衔就是技术产品经理。有些人本身就是工程师。这些工程师可能补一个产品岗，也可能补一个工程岗，但不管哪种，都被要求会写代码。而且基本上我所有的产品人都得会写代码。但说实话，我真正在找的其实不是你的技术能力，因为我觉得技术这块可以随岗位需要往上或往下调。我真正要的，是那种——我不会说是「难搞」，因为也有一大堆人纯粹是为了唱反调而唱反调——我要的是那种始终觉得「这事儿肯定有更好的做法」的人。而且现在你上网一搜，能比以前快得多地把这种人找出来。

[58:15] 主持人

As you're just talking, I was asking myself, is this like the best time ever to make the jump from engineer to PM

你在讲的时候，我一直在问自己：现在是不是史上从工程师转 PM 最好的时机——

[58:21] Chase

and and vice versa? Really, I think they're just molding. I think it's technical engineer or it's like product facing engineer and technical PE product and that's like you just put them together. Yeah, it's that's going to move the fastest. That's you know we kind of didn't go into it but we just launched a big AI chatbot and if I didn't build all of it the mobile app, the back end the AI itself,

反过来也是？说真的，我觉得这俩正在融为一体。要么是技术型工程师、或者说面向产品的工程师，要么是懂技术的产品人，你把他们凑到一块就对了。对，这种组合跑得最快。我们刚才其实没细讲，但我们刚上线了一个大的 AI chatbot，如果那整套东西——移动端 App、后端、AI 本身——不是我一个人从头搭起来的，

[58:49] Chase

the handoffs and the time between it would have taken so long. and instead like I got the beginning of the mobile app running. I tested it. I was like that doesn't work the way I want. I fixed that. I fixed the scrolling. I then added like an appointment viewer and I just kept going, right? Which is why I ended up with a 10,000 PR. But I was like this is actually the the MVP that makes sense is something that isn't but if I tried to hand that off in words my first version that I built for myself which I handed off in words to AI

那这中间的各种交接、以及交接之间耗掉的时间，会拖得极其漫长。而现在是这样：我先把移动端 App 的雏形跑起来，自己一测，觉得「这跟我想要的不对劲」，就去改；我修好了滚动，接着又加了个预约查看器，然后就一路往下做，对吧？这也是为什么我最后搞出了一个一万行的 PR。但我当时想的是，真正说得通的 MVP 其实就是这么个东西——它不是那种……但假如我当初试着用文字把需求交接出去，我最早给自己做的那第一版、然后我用文字把它交接给 AI，

[59:17] Chase

was terrible. So it's like, you know, you kind of need these full kind of spectrum people because the handoffs just kill you. So I think no matter what I want, I want engineers who are care about the business, care about the mission are coming to try to solve this mission and not just to build our infrastructure. Although there's a place for them, it just, you know, I'm talking about kind of this generalist person. I think you just got to be on both sides of it and that's going to be the fastest way to do it. The the other thing I picked up on that I want to kind of double click into if you don't mind, Mark. Mis like you call the misfits, people that don't play by the rules, you said something really interesting, which is like if you're the kind of person that needs like rules to follow, you won't do well in this because things are moving so fast that like the rules are being rewritten as we're going. And that kind of led to a couple thoughts or like maybe a couple questions. First one is do you think the people like the people who don't like following rules okay how am I gonna frame this so much of what we do with AI has to be setting the rules for the AI for how to work right so like being really good at explaining how the what the rules are for the AI because we actually do want our AI to be really good at following rules right we don't want our AI to be misfits like so like we want our AI our AI to like be really good at following rules so we want people who are very good at creating and codif ifying the rules and encoding the rules into the AI. So the first question for you is like do you think that the kind of people who are really good at setting rules are like what's the overlap between the people that are really good at setting like creating the rules and people that are like good at following rules in your view? So that's like the first question. And then the second question is how do you evaluate the kind of like misfit vibe of of someone that you might be interviewing and will will still be good to work with and won't be like a like a ridiculous person to have around, but also is someone who's, you know, obviously like not shy about throwing away the playbook and kind of figuring stuff out without rules.

结果糟透了。所以你懂的，你确实需要这种「全栈式」的人，因为光交接就能把你耗死。所以不管怎样，我想要的是那种真正在乎业务、在乎使命的工程师，是冲着解决这个使命来的，而不只是来搭基础设施的。当然，那种纯搭基础设施的人也有他们的位置，只是我现在说的是这种通才型的人。我觉得你就是得两头都站得住，这才是最快的路子。还有一点我特别想再深挖一下，Marc，你不介意的话。就是你说的「misfit」、不按规则出牌的人——你刚才讲了个很有意思的点，就是如果你是那种非得有规则可循才行的人，你在这个环境里会做得很吃力，因为一切变得太快，规则是边走边被重写的。这就引出了我几个想法，或者说几个问题。第一个是：你觉得那些不喜欢遵守规则的人——我该怎么把这话组织好呢——我们跟 AI 打交道，很大一部分工作其实是在给 AI 定规则、告诉它该怎么干活，对吧？所以「特别擅长把规则给 AI 讲清楚」这件事很关键，因为我们其实是希望 AI 非常擅长遵守规则的，对吧？我们可不想让我们的 AI 变成 misfit。我们希望 AI 特别擅长守规矩，所以我们需要那种非常擅长制定规则、把规则编纂出来、并把规则编码进 AI 的人。所以给你的第一个问题是：在你看来，「特别擅长定规则」的人和「特别擅长守规则」的人，这两拨人之间的重叠有多大？这是第一个问题。第二个问题是：你怎么去评估一个你可能在面试的人身上那种「misfit 气质」——既能让他真的好合作、不至于是个让人受不了的家伙，同时又确实是那种敢于把 playbook 扔掉、在没有规则的情况下自己摸索出办法的人？

[1:01:16] Chase

Yeah, it's a really hard line. That first one I I'll hit on it is, you know, I think it it's the same concept as a white hat hacker. So like I think the best people at setting rules are the people who break them all the time and always find ways around things, right? So you know that's not always of course nothing's always true, but some of the people that I've enjoyed working with the most are the ones who are like especially for internal teams because you don't do so much white hat kind of thing for like your product. You expect kind of people to use it. At least in my world, it's always been people that are using it because they want to. And it's usually healthcare. So, it's like it's not someone just randomly from some other country trying to break my product. So, it's usually internal and I'm setting up rules or whatever else. And it's like usually my best people are the people who are like, "Yeah, I'm going to I'm going to try to get around this to move faster." And, you know, that's what AI is doing as well. So, I find a good overlap in terms of setting rules versus people who didn't like following them. Um,

对，这个界限确实很难拿捏。第一个问题我先说——我觉得这跟白帽黑客（white hat hacker）是一个道理。就是我觉得最会定规则的人，恰恰是那些老是在破坏规则、总能找到各种绕过办法的人，对吧？当然，没有什么是绝对的，凡事都有例外，但我合作过最愉快的一些人就是这种，尤其是对内部团队来说——因为你不会对自己的产品做那么多白帽那一套。你会预期用户是想用才用你的产品的。至少在我这个圈子里，一直都是因为想用才用它，而且通常是医疗健康场景。所以不是什么随便哪个别国的人跑来想搞垮我的产品，通常是内部的人，而我在给他们定规则之类的。往往我手下最厉害的那些人就是会说「行，我要想办法绕过这个，好让我干得更快」的人。而你知道吗，AI 干的其实也是同一件事。所以在「定规则」和「不爱守规则的人」这两者之间，我发现有相当好的重叠。嗯，

[1:02:14] 主持人

yeah. And

对。而且——

[1:02:15] 主持人

yeah, um like in a weird way like I feel like people that break rules um they understand the constraints very well and then they work like they're very creative from those constraints, right? And it's it's from my opinion it's way easier to be creative when you have constraint as opposed to when you're expansive. I think they're like really good at like kind of back engineering. Hey, here's how it works. Is that what you do? Given these constraints and these rules that I think I understand how the game works, like let's figure out like a system that allows me to beat these rules to get to the outcome that I want basically. Yep. Yeah. Um the second one there's there's a cheating answer and then a good answer. So the cheating answer is I've got a very long bench of people that I've loved working with that is probably going to be my hires for the next year at least as we scale. Um which you know we'll probably plan on adding two or three or four this year which is again in the previous world would have been seven eight or nine. So we're definitely hiring less to do the more work I think um than what my road map would have looked like. So, I've got a bench of like people that have either mentored me or I've mentored or stayed in touch with from a lot of startups. I think I'm on my fifth startup and I think four of those have been really great experiences. So, you know, I think I have a lot of people I'm interested in working with again. And then the real answer that actually is like scalable after those 15 20 people is probably the thing I look for the most is just that you've built something random as part of your resume. So it's like you just got this job that's like product whatever and it's like a line that I often look for is like my my classic example that kind of got me started down this path was I was hired I was employee number 90 at this company that's now $10 million company and you know I was there for five years but the first thing I had to do like was all the product other product directors were like you're taking over QA and I was like oh that's that sucks I don't want to do QA and I automated it right like so I just went in old school automation Python stuff I was okay at it at the time and it became a core part of our infrastructure much to the chagrin of our CTO who they they were named the chase scripts and you know every engineer hated them because they came such a thorn because they weren't scalable. So I'm usually looking for that. I'm looking for someone you just built something right like whether it's like and it's just something that you didn't you weren't hired to do. So you found a problem and you solved it. So, that's a common question I'll I'll ask is like, "Tell me about a problem you you you fixed." And then I'll follow that up with like, "Tell me about a problem you fixed that had nothing to do with your job." Because they usually have they'll probably usually try to spend that first one has nothing to do with your job. So, it's usually like, "Tell me about your favorite problem you fixed." And then second one's like, "Tell me about another one that has nothing to do with your job." Even better if you answer that first one. It had nothing to do with your job.

对，说来有点反直觉，我觉得那些爱破坏规则的人，其实对约束理解得特别透，然后他们能在这些约束里做得非常有创造力，对吧？在我看来，有约束的时候反而比什么都放开时更容易有创造力。我觉得他们特别擅长那种「逆向工程」——嘿，这套东西是这么运作的。你是不是就这么干的？就是给定这些约束、这些规则，我自认为我摸清了这游戏怎么玩，那咱们就来琢磨出一套系统，让我能钻这些规则的空子、拿到我想要的结果。没错。嗯，第二个问题，有个「作弊式」的答案，还有个「正经」的答案。作弊式答案是：我手里攒了一长串我特别喜欢一起共事的人，未来至少一年随着我们扩张，我招人基本就从这批人里挑。今年我们大概计划加两到三个、或三到四个人，这在过去那个世界里差不多相当于要招七、八、九个。所以我们确实是招得更少、活儿干得更多，比我原本 road map 里预想的人数要少。所以我有一条「板凳」——上面都是要么带过我、要么被我带过、或者从很多创业公司一路保持联系的人。我算了算这是我第五家创业公司了，其中大概有四家的经历都相当棒。所以我有一大批我很想再合作一次的人。然后真正的答案——就是过了这十五、二十个人之后仍然可复制的那个办法——我最看重的，大概就是你的履历里有没有做过什么「不务正业」的东西。就是你拿到了一份职位、写着产品什么什么，然后有那么一行让我特别留意的东西。我最经典、也是把我带上这条路的例子是：我曾经是某家公司的第 90 号员工，那公司现在是家一千万美元的公司了，我在那儿待了五年，但我进去要做的第一件事——当时其他产品总监都跟我说「你来接手 QA 吧」，我心想「啊这也太糟了，我才不想做 QA」，于是我就把它自动化了，对吧？我就用最老派的自动化，Python 那一套，当时我水平也就还行，结果它成了我们基础设施里一块核心的东西，气得我们 CTO 够呛——那些脚本被起名叫「chase scripts」，每个工程师都恨它们，因为它们成了眼中钉，毕竟根本不具备可扩展性。所以我通常就在找这种人。我要找的是那种「你自己搭过点什么」的人——不管是什么，关键是那是件没人派你去做的事。你发现了一个问题，然后你把它解决了。所以我常问的一个问题是：「讲讲你解决过的一个问题。」然后我会追一句：「再讲一个你解决过、但跟你本职工作毫无关系的问题。」因为他们通常……第一个问题他们多半会挑跟本职无关的来讲。所以一般是「讲讲你最得意的一个你解决过的问题」，然后第二个是「再讲一个跟你工作完全无关的」。如果你第一个答案就已经是跟本职工作无关的，那就更好了。

[1:04:58] 主持人

Nice, man. I wish we had another hour. I feel like we're just like scratching the surface. Um, but I think we we probably should start wrapping. And so, two final questions for you, Chase, before we we hit our fun gratitude corner. Um, one is where can people learn more about what you're doing and if you do any writing or anything like that? And then two, like how can people be helpful to you and Millie?

太棒了，兄弟。真希望我们能再聊一个小时，感觉我们才刚触到皮毛。不过我觉得我们大概该收尾了。所以在进入我们有趣的「感恩环节」之前，还有两个最后的问题想问你，Chase。第一，大家可以去哪里更多地了解你在做的事情，你有没有做什么写作之类的？第二，大家可以怎样帮到你和 Millie？

[1:05:21] Chase

Yeah, first one is I am starting to write on LinkedIn more. Not something I've done a lot. It's it's frankly it's a recruiting tool. Recruiting doesn't mean employees only. It means partners and and people I want to work with. So I did write a big set of posts on our AI launch evals, how we thought about it, how we did it in healthcare and launching like fast but safe. Um so that's your best bet. I probably need to do a couple more. Um

好，第一个问题：我开始更多地在 LinkedIn 上写东西了。这不是我以前常做的事。说白了，它是个招募工具。而招募不光指招员工，也包括找合作伙伴、找我想一起共事的人。我确实写了一大组帖子，讲我们那次 AI 上线的 evals——我们是怎么思考的、在医疗健康场景里是怎么做的、以及怎么做到又快又安全地上线。所以那是你了解我最好的入口。我大概还得再多写几篇。嗯——

[1:05:47] Chase

we can link to that. We'll link to that in the show notes. And then um yeah, I think for me the the way people can help is, you know, I just like I said, I mean, I'm not really going to end up hiring a lot of people off the street because I have too many people I've worked with that I I really want to bring back. Yeah. Get the group back together. So, I think it's other partnerships or just people who want to learn about it and stay warm. If we hit the scale we expect, I'll run out of my bench faster than I expect. So, I think it's people that are interested in, you know, women's health. that's an underserved area, especially in VC dollars over the years or maternity and are interested in, you know, being on the forefront of trying to make a difference there, whether that's partnerships or or coming and working with us.

我们可以放那个链接。我们会在 show notes 里放上。然后，嗯，对，对我来说，大家能帮上忙的方式是——就像我说的，我基本上不太会从街上随便招一大堆人，因为我有太多合作过、特别想请回来的人了。对，把老班底重新聚起来。所以我觉得更多是其他合作机会，或者是那些想了解、想保持联系的人。如果我们真达到了预期的规模，我这条「板凳」会比我想象的更快用完。所以我要找的是那些对女性健康感兴趣的人——这些年在 VC 资金上，女性健康、尤其是孕产领域一直是个被严重忽视的方向——那些有兴趣站在最前沿、真心想在这里做出点改变的人，不管是通过合作、还是直接加入我们一起干。

[1:06:32] 主持人

Yeah. And we'll link to Millie in the show notes, too. So, if people are um uh about to enter the

对。我们也会在 show notes 里放上 Millie 的链接。所以如果有人正准备进入——

[1:06:39] 主持人

uh the maternity care space, you know, they should definitely check you.

呃，孕产护理这个领域，那他们绝对应该去了解一下你们。

[1:06:43] Chase

Or if you need a a set of midwives, and I'm not expecting a lot of them to be listening, but if you're, you know, a woman who needs a a gynecologist or a uh a maternity center, then you you can hit us up. We had a we had a birth doula for when my daughter was born and I thought that was a helpful few weeks in preparation for the birth for my wife and I just like ask all of our questions and get that was really helpful. Yeah, we have a um we have a patient that's from a big tech company and she reached out and like gives all kinds of critical feedback and it's so much

或者如果你需要找一组助产士——我倒不指望有很多助产士在听这个节目——但如果你是个需要看妇科、或者需要孕产中心的女性，那你可以来找我们。我女儿出生的时候我们请了一位分娩陪产师（doula），我觉得在临产前的那几周里她帮了大忙，我和我太太就是把我们所有的问题都问她，那真的很有帮助。对了，我们有一位患者是某家大厂出来的，她主动联系我们，会给各种一针见血的反馈，那些反馈比其他所有人的都要好得多，因为——

[1:07:14] 主持人

better than everyone else's because it's like

——好太多了，就因为那感觉像是——

[1:07:17] Chase

great I've got a tech person. So if I can get some startup people to uh

——太好了，我这儿有个懂科技的人。所以要是我能再拉一些创业圈的人——

[1:07:21] Chase

go to our clinic then we'll all learn a lot together and push care.

——来我们诊所，那我们就能一起学到很多，把医护往前推一把。

[1:07:25] 主持人

Okay. And then we have our our final gratitude corner. So the question here for you is as we kind of you know reflected on your journey to this point it can be early career recent but like are is there anyone you can name you know multiple people it doesn't have to be just one but anyone that stands out as someone you really want to thank for the role that they've played in like your your story.

好。接下来是我们最后的「感恩环节」。这里给你的问题是：当我们回顾你走到今天的这段历程——可以是很早期的、也可以是最近的——有没有哪个人你想点名，你知道，可以是好几个人，不必只是一个，但有没有谁特别突出，是你真心想为他在你这段故事里所扮演的角色而致谢的？

[1:07:50] Chase

Yeah. Um

有。嗯——

[1:07:53] Chase

so you you prepped me for this so I was able to like prep but this was the first thing that came to mind. So Una Pippich is is my person. So she was my partner in crime at that startup. I just talked about that 90 person where I started and kind of started automating. And you know I think I came into it worried that I pushed too hard and I was too disagreeable and I was too much of a misfit and I needed to play it closer to the best. Talk a little bit less. Just go up the corporate ladder. And you know she had had probably 10 more years of career. she was two levels above me and was my boss whenever I joined. Um, and I think she really pushed me to be like, "No, you're on the right path." Like, you'll need to hone your edges. Like, you definitely need to hone the edges and like find where you're going way too far, but in general, like, keep going down this path. And I think it made my career at Tippus AI wildly successful in comparison to what would have been her championing me and covering up my faults and yelling at me whenever I uh went too far. And you know, I've kept that spirit. So, I've just never gotten rid of this kind of misfit hacker underdog kind of thing. Even though I'm no longer pretty much any of those things, she just said like, "Keep it alive. Don't follow the rules. Don't, you know, just button up the suit. Like, just keep pushing and keep keep moving things forward or you're going to get bored to death." And, you know, she's one of my one of my favorites.

你们提前跟我预告过这个问题，所以我能准备一下，不过这是我脑子里冒出来的第一个人。Una Pippich 是我要谢的人。她是我在那家创业公司里的「同伙」，就是我刚才讲的那家、我第 90 号员工、开始搞自动化的那家。我记得我进去的时候一直担心自己是不是太爱较劲、太难搞、太像个 misfit，觉得自己需要收着点、跟大家贴近点、少说点话，就顺着公司那套阶梯往上爬。而她大概比我多十年职场经验，比我高两级，我刚进去时她是我的上司。我觉得是她真正推着我，让我意识到「不，你走的路是对的」。她说你需要打磨一下自己的棱角——你确实得磨磨棱角，得找出你哪些地方冲得太过了——但总体上，继续沿着这条路走下去。我觉得正是这一点，让我在那家公司的职业发展相比原本可能的样子成功得离谱，因为她一路力挺我、替我兜住短板、在我冲得太过时会吼我。而你知道吗，我一直把那股劲儿留着。所以我从没丢掉这种 misfit、hacker、underdog（弱者逆袭）的劲头。哪怕如今这些标签基本都不再适用于我了，她当时就是跟我说「让它活着，别守规矩，别把西装扣得那么严实，就一直往前冲、一直把事情往前推，不然你会无聊到死」。她真的是我最珍视的人之一。

[1:09:19] 主持人

Una Pippich. We will we will link to Una in the show notes. Um, that's a great way to wrap. This sounds like an awesome person. When someone that you look up to can tell you to be yourself, obviously to improve and clean up the and tidy up the edges, like for sure I've I have a huge respect for for people that can simultaneously encourage the strengths and tell you to double down on them and not to tone not to mute your strengths while also making you feel good about getting better on the things you're not not as good at or things that could hold you back from letting your strengths shine. So,

Una Pippich。我们会在 show notes 里放上 Una 的链接。嗯，这是个特别好的收尾。听起来她是个了不起的人。当一个你敬仰的人能告诉你「做你自己」——当然同时也要改进、把棱角收拾干净、修整一下——说真的，我特别敬佩那种能同时鼓励你的长处、让你把长处加倍发挥、又不让你去压抑或磨平自己长处的人；与此同时，还能让你在「补上那些你没那么擅长的、或者可能拖累你、让你的长处发不出光的地方」这件事上感觉良好。所以——

[1:09:53] Chase

yep. Every strength can be a weakness, as they say.

对。就像人们常说的，每一个长处都可能是一个短处。

[1:09:56] 主持人

Yeah, there's a shadow. But man, this has been awesome. I'm so glad we did this, Chase. It had been way too long since we hung out and I'm so glad we made the time and really appreciate it.

对，是有那么点意思。不过说真的，今天聊得太爽了。Chase，特别高兴咱们能录这一期。上次见面到现在实在隔太久了，很感谢你抽出时间，真的很感激。

[1:10:06] Chase

Awesome.

太好了。

[1:10:06] 主持人

Yeah. Thanks for coming on.

是啊，谢谢你来上节目。

[1:10:08] Chase

See you.

回头见。

[1:10:09] 主持人

Bye. Bye. Bye. Bye.

拜拜，拜拜，拜拜。