Jerry Liu 和 Robert Yang 谈研究与产品结合的困难

It is really hard to combine research and product. A lot of the traditional wisdom in building pro...

It is really hard to combine research and product. A lot of the traditional wisdom in building product at a startup don't apply for research. Building pure product requires scrappily building MVPs, iterating as quickly as possible on customer feedback, and being able to change direction in order to align within ICP needs. The product that satisfies customer needs doesn't *necessarily* need the latest scientific discovery. Research requires longer-term thinking. It requires dedicated focus time for deep exploration, synthesis, and experimentation in order to iterate on a deep piece of tech. It requires a longer-term bet and can't be disrupted on a whim by customer feedback. To some extent in order to do successful research, you have to basically ignore most/all customer feedback in order to focus on a core set of research goals. You care about general insights, not overfitting to a specific datapoint/bespoke need. In a vacuum where you only do research, the risk of course is that you create a beautiful piece of tech that doesn't have PMF. We've experienced this firsthand at @llama_index , where we *have* to do applied research to drive the frontiers of document understanding. But that means we have to simultaneously balance a wide volume of customer needs with a focused effort on improving the core pareto frontier of cost-accuracy. Robert Yang @GuangyuRobert Sharing the biggest mistake we made building a neo-lab: Confusing research with product It's been 3 years since we started @Fundamental and 1 year since we launched @tryshortcutai . We made lots of common startup mistakes including hiring too fast, too loose w/ the purse, losing focus, raising at the wrong time, etc. But the most profound and painful mistake we made is not understanding the gap & tension between research and product. And we made it repeatedly (1) Our first focus was Minecraft agents, inspired by @DrJimFan 's Voyager paper. We did Project Sid, a 1000 agent society simulation in 2024. It was a very cool research project, but we thought it was a product as well, so we sold these agents directly to Minecraft players. The product was bad: novel but no retention. Millions of views but 10% D1 retention. Thinking this was a product, we didn't open source our code and missed the opportunity for it to be more impactful research. Lose & lose. (2) Then we pivoted to computer use agents late 2024. I led the team to focus on the OS World benchmark and we 2x the SOTA performance in a month. In particular, Peter @BrainsAndTennis led the work on the Spreadsheet category and got superhuman performance (SOTA was ~10% and we got ~70%). OS World evaluates how agents use common Linux softwares. Our agent was performing well by building skills that chain common GUI operations. Cool research. We AGAIN built a product that's too similar to the research: agents taking GUI actions to use your computer. 2 months in, we realized it was a terrible idea. For one, you can't use your computer when the agent takes actions. At this point, we realized that we made the same mistake confusing research with product. But we still didn't know HOW to properly build a product from our research. The atmosphere was grim: we had already raised 3 rounds & $40M pre-revenue, so we felt that if we don't figure out how to build a valuable product from our research, the company will die. In a frantic attempt to save the company, we broke into small teams, each trying to build something worthwhile. Nico @nicochristie and Thariq @trq212 (before he joined Anthropic to work on Claude Code) worked closely together to build a more focused Spreadsheet agent product. Nico had the refined product taste to know what our agents should be positioned as (investment banking analysts), and Thariq had the profound insights to have agents write code against a Spreadsheet API (instead of just tool calling or using GUI). Nico then launched Shortcut @tryshortcutai and to this day it remains the premier spreadsheet agent. (3) If you think we are done making this mistake, we are not. We still constantly struggle with the research-production tension, on what to focus on, how to translate, etc. For a "lab" company, I don't think the tension ever goes away. We just become more aware of it and more proactive about handling it. 🔗 View Quoted Tweet 💬 6 🔄 2 ❤️ 18 👀 3085 📊 8 ⚡