Stop Debating, Build the Experiment Machine
Once AI makes delivery cheap, product teams should spend less time arguing in meetings, run more experiments, and turn the way they run experiments into a machine
Recently, I had a conversation with our engineering lead. He raised a concern many technical teams share: as AI coding capabilities improve and delivery speed keeps increasing, will we one day run out of things to do? My answer: you don’t need to worry at all. This is a classic Jevons Paradox. In the 19th century, steam engines kept getting more efficient with coal, yet total coal consumption went up rather than down, because once coal got cheaper to use, many things that hadn’t been worth doing became worth doing. AI works the same way. When delivery costs fall, demand won’t shrink; it will only grow explosively, and what changes are the rules of the game.
In traditional product development, a huge amount of time goes into discussion. In product meetings, the designer says the button should be bigger, the PM says it should be smaller, the engineer says both are doable, and so the discussion goes on… When should the paywall popup trigger? What’s the pricing strategy? How should the interaction interface work? Relatively detailed questions like these easily devolve into “he said, she said” territory.
You can make an argument, and someone else can make the opposite argument. Human rationality (or more accurately, theory and reasoning) loses too much reference value here. You can say anything: “I think users need this,” “my experience tells me this,” “I feel it should be designed this way”… The result is endless meetings, hours of discussion with no conclusion. Worse still, the vast majority of our hypotheses are actually wrong. According to experiment data from companies like Google and Microsoft, only 10-33% of product hypotheses turn out to be effective, which means 67-90% fail. These hypotheses, built on imagination and reasoning, often don’t match what real users need. The problem is that we’re using reasoning to solve empirical problems, and the only answer to an empirical problem lies with users.
0-to-1 and 1-to-100
But this doesn’t mean every stage should be experiment-driven; different stages of product development need completely different methodologies. In the 0-to-1 stage, you’re searching for Product-Market Fit and still figuring out what users truly need. That calls for gut intuition, rapid iteration, deep user interviews, and doing things that don’t scale. As I discussed in a previous article, chasing A/B testing and growth hacking before PMF is futile. Airbnb’s success had little to do with optimizing button colors; it came from founders Brian Chesky and Joe Gebbia personally photographing NYC listings and crafting 10-star experiences by hand. At this stage, judging product direction depends on taste, that is, on understanding your users’ human nature. If you keep A/B testing better horse carriages, you’ll never invent the car.
In the 1-to-100 stage, you’ve found PMF and have stable user traffic. Now you need an experiment-driven approach, data validation, rapid iteration, and scaled optimization. As Marshall Goldsmith’s widely quoted saying goes: “What got you here won’t get you there.” The methodologies of the two stages are completely different. Confuse them, and you’ll either over-optimize during 0-to-1 or keep making gut calls during 1-to-100.
Experiment-driven development isn’t a panacea either. It has clear prerequisites: PMF is established and the product direction is clear, so you’re no longer exploring the basics; there’s enough traffic to produce statistically significant results; and the metrics are clearly defined, so you know what matters and what doesn’t. It also has traps to watch out for. Many people misunderstand data-driven and become data-centric instead, optimizing metrics for metrics’ sake, chasing vanity metrics, and in the end harming the user experience. The right approach is data-driven but user-centric: data is the means, user value is the end. You can’t sacrifice the overall experience by showing a popup the moment users arrive just to boost conversion by 5%. Brian Chesky is clear on this: if you optimize short-term conversion at the expense of the 10-star experience, you’re killing the golden goose.
Another trap is chasing metrics blindly. If you define the wrong metric, or over-optimize a single metric, that can be worse than ignoring data entirely. A metric should reflect real user value: if your metric is “number of videos generated,” you might optimize your way into a pile of low-quality videos; if it’s “average video quality users can achieve,” then you’re really solving the problem. Pure data-driven approaches also have a classic trap, the conflict between local optimization and global innovation, which can kill disruptive innovation. When the iPhone launched, critics argued that users needed physical keyboards. In Airbnb’s early days, conventional wisdom said no one would stay in a stranger’s home. When Netflix shifted from DVD to streaming, the short-term data was indeed negative. So during a paradigm shift, you need human judgment and taste; during the optimization phase, you need experiments and data.
Momentum Is the Moat
As I discussed in a previous piece on experiment culture, besides raising productivity, AI lets us turn “I think” into “I tried.” When experiment costs drop dramatically, the need for debate disappears. A typical experiment cycle goes like this: define a clear hypothesis and success metrics; build an MVP in a few days with AI’s help; roll it out gradually, to a subset of users first, without rushing to a 100% rollout; see the data change the next day; then decide quickly whether to continue, adjust, or kill it. As long as your traffic is sufficient and PMF is confirmed, there’s no need to argue for hours in a conference room. You can ship the feature and let the data speak.
The key to this cycle is speed: the faster you get from hypothesis to validation, the better, and that’s why momentum matters so much in the AI era. If you can ship and test 5 product features per day or per week in a data-driven way, and use data to verify whether they work or not, then compared with teams still debating in meetings and writing PRDs, that’s a generational gap. It’s like you’re already using nuclear weapons while they’re still fighting with cold steel. In this growth battle, they don’t stand a chance. So momentum itself is the moat. HeyGen’s product development handbook has a passage on this: “Competitors ship one feature per month; we ship five experiments. We learn five times faster. That learning compounds into superior products.” The value of speed is that the learning loop turns faster; what you’re really delivering quickly is learning.
Iterating this fast naturally makes people worry about technical debt and user experience consistency. My view is that technical debt is a lot like financial debt: it’s a form of leverage, and there’s no need to steer clear of it. You trade technical debt for product growth velocity, and in this momentum-defined AI era, that’s a reasonable trade-off. Only when technical debt accumulates faster than you grow is it time to stop and deal with it.
As for user experience consistency, rapid experimentation does create problems: run multiple experiments at once, and different users might see completely different product interfaces, which causes confusion and raises support costs. But this is a feature gate and gradual rollout problem. You don’t need to roll out every experiment to 100% of users. You can test on new users first, on only 10% of users, by region, or by scenario. The key is making sure each individual user sees a consistent experience; don’t let anyone see a button on the left today and on the right tomorrow.
The Machine That Builds the Machine
An AI Native company designs its entire product development process from scratch to be AI-first, and what separates it from a traditional company lies in organizational structure and mindset. HeyGen systematically lays out this philosophy in their product development handbook: in the AI era, we operate without a stable technology foundation; every few months AI technology evolves dramatically, and model capabilities are unknown and changing fast. Traditional software development assumes a stable foundation, but in the AI era, this foundation changes every 2-3 months. That instability is actually an opportunity. The thing to do is ride the wave, not fight the current.
Traditional-era thinking is to build on stable foundations, optimize for longevity, plan 12-18 months ahead, and polish before shipping; in the AI era, you ship to learn and run experiments in parallel. There are a few principles. Distinguish what changes from what stays constant: models change and capabilities change, but users’ core problems and workflows don’t, so build systems around what stays the same while riding the wave of model improvements. Design products that improve on their own: when GPT-5 arrives, your product should automatically get better without needing a refactor, which takes abstraction layers that let the product experience ride on top of AI advancement. Keep the architecture flexible: expect change, version everything, and make systems replaceable. Set planning cycles at 2 months, long enough to build meaningful things and short enough to adjust quickly, in step with AI model upgrade cycles. And although realistic planning runs on 2 months, strategically you need to predict capabilities 6-12 months out and position early.
Elon Musk once said: “It’s important to build the machine that builds the machine.” I now have a deeper understanding of this statement. Pursuing extreme attention to product detail is reasonable, and you must have that pursuit. But it can’t be the subjective “it’s good if I think it’s good”; it needs an objective standard of measurement, and that standard is whether users love it. Whether users love it is largely evidenced by the metrics you define and the data behind them. So building your product optimization workflow well matters far more than building any single product well. That’s what I mean by “build the machine”: building a better assembly line.
This machine is a complete, self-evolving product development system that keeps producing better products, and it has three layers. One is technical infrastructure: a feature flag system for toggling features quickly and rolling them out gradually; an A/B testing platform that supports multivariate experiments and statistical analysis; real-time monitoring and analytics tools for spotting problems and opportunities quickly; and the abstraction layers for model upgrades mentioned above. Another is organizational capability: an experiment culture where data does the talking; a fast decision-making mechanism, with two-way door decisions made the same day so you don’t fall into the consensus trap; 2-month planning cycles synchronized with model upgrades; and disagree and commit, putting speed first and correcting quickly when wrong. The last is strategic judgment, which we’ve already covered: how to define metrics so they reflect real user value, how to stay data-driven but user-centric, when to rely on taste and when on data, and what changes (AI capabilities) and what stays the same (user needs).
This is a meta-capability: what you’re optimizing is the “optimization capability” itself. Most startups lack the capability to build this machine, which is why 0-to-1 and 1-to-100 are completely different problems. But in the AI era, this is a machine you have to build.
Moving fast and pursuing excellence don’t contradict each other; in fact, moving fast is the prerequisite for building better products over the long term. Moving fast doesn’t mean shipping features quickly. It means delivering customer value quickly (and learning quickly). Speed serves the ultimate goal: being the absolute best. HeyGen’s quality bar is clear: for creative tools like video content, quality is non-negotiable, and users love products that solve their problems with exceptional quality. Their success metric is the one mentioned earlier: the average video quality any user can achieve on the platform. I think this is the right north star.
Back to the engineering lead’s worry. I don’t think AI will leave us with nothing to do, but what we do will change completely. Once delivery gets cheaper, there will only be more demand, and what separates teams is whether they’re still handling the new rules of the game the old way or have already built the experiment machine. So he doesn’t need to worry about running out of work. There won’t be less of it; we’ll just spend much less time arguing in meeting rooms, and the time we save can go into running experiments.