Experiments Are a Durable Moat
I've been watching my current and former colleagues ship faster, write more tooling, and get more done with the help of AI. I've also seen more product iterations, more new features, and more decisions. The speedup is real.
But speed without learning is just faster guessing. Experimentation is the mechanism that converts speed into learning. The companies that wire experimentation into their AI-accelerated development will compound knowledge. The ones that don't will compound errors. And after a year or two, that gap is uncatchable.
I know this because I've been inside the machines that do it well. I spent four years at Google Ads where I personally analyzed thousands of experiment variants that contributed to growing search ad revenue by over 10% over 4 years. I've done two tours at Meta supporting over a dozen different teams shipping very fast. Despite very different cultures, philosophies, and experiment frameworks, both companies use experiments to determine truth, set a shared mental model of their products, and enable learning for better product iteration. Experimentation works as a compounding knowledge engine. It produces consistently better decisions than any individual's intuition could.
The Real Value of an Experiment Is the Why
Most teams treat an A/B test as a gate: metric goes up, ship it. That's the least interesting thing an experiment can tell you.
Google Ads constantly ran learning experiments — especially early in a product iteration. These were informed by what had been done before. There were docs that synthesized a decade's worth of experiments and product insights with a mental model for why the experiments produced the results they did, and we leveraged them. We used them to explore the product space, understand how different dimensions impacted users and advertisers, and align with long-term Google goals. We understood tradeoffs, validated mental models, and constructed a bundle of changes from all the learnings before launch. It was a flywheel that produced consistent impact for over a decade.
I was on the team that launched favicons on Google Search — the little site icons next to each result. The metrics were positive. If we'd stopped there, the story would have been "favicons good, ship it" and we'd have learned nothing.
But we dug into why. What we found was that favicons increased the scannability and parsability of the results page. Users could visually distinguish between results faster, which meant they made better click decisions. The favicon wasn't decoration. It was a cognitive aid that made the page more efficient as an information-selection interface.
That insight — visual distinctiveness improves user decision quality — changed how we thought about every subsequent layout decision. It was a principle, not a data point. And we needed it. When we launched favicons on desktop, The Verge covered it and the press was brutal. We stepped the launch back, but the experiments continued to show stronger user benefits the longer they ran. The causal understanding — not just a p-value — gave us the confidence to push through and eventually launch it across every Google surface.
Data points don't compound. Insights and learnings compound. When the team understands the mechanism, designers use it, PMs use it, engineers use it. Everyone makes better tradeoffs, faster, because everyone is optimizing for the same verified model of how users actually experience the product. That's not data science. That's product empathy built on evidence instead of projection.
What Happens Without This
I love Annie Duke's "resulting" — evaluating decisions by their outcomes instead of their quality. We all know a good decision can have bad luck and a bad decision can get lucky. We also share an intuition that if we play the infinite game, decision quality will become readily apparent. But most organizations evaluate decisions by results. It's easier than evaluating the process.
My two tours at Meta highlight this. The company is an experiment powerhouse, but I also saw resulting everywhere. Meta has structures to fight it — long-term holdouts and strong performance incentives.
I watched dozens of teams at Meta ship fast with small, underpowered experiments, watching a few metrics. They hit their goals. Then approaching the end of the half, the long-term holdout shows major regressions in other teams' metrics. Investigation happens. It happens again next cycle. The teams that have the holdout mechanism catch it and fix it. The performance system enforces it. This allows Meta to move fast while maintaining product quality — it's a superpower. But I've been at companies without those mechanisms that also ship fast, and the errors just compound silently.
AI Makes This More Urgent, Not Less
AI collapses the cost of building. It does not collapse the cost of being wrong. Going faster can mean being wrong more often. It can mean shipping more cracks that degrade the product.
More changes ship per unit of time. More decisions get made. If the quality of each decision stays flat — or degrades because people are pattern-matching faster instead of thinking harder — you're compounding uncertainty at a higher rate. Duke's poker analogy works here: if someone gave you a tool to play five times as many hands per hour, would that make you a better player? Only if you're learning from each hand. Otherwise you're just losing faster.
AI also makes the analysis infrastructure cheaper. The instrumentation, the log parsing, the causal decomposition — these are exactly the workflows AI accelerates. The cost of understanding why is dropping fast, which makes the ROI of building an experiment culture and robust mental models enormous. We're in a moment where the cost of experimenting and the cost of understanding results are both falling, while the opportunity cost of not doing either is rising because decisions are happening faster.
The teams that invest in learning — fast and well — are going to outcompete teams that don't. Execution speed is increasing at the same time that user expectations and product landscapes are changing the fastest in our lifetime. It was a game changer for Facebook when they discovered the news feed. Despite the bad press and pushback, the data said this is the future — and news feeds became the default interface for social products. No one has found the news feed of AI yet, but someone will, and it will be a team with phenomenal product insight.
This Is a Moat
Nicolas Bustamante recently argued that most vertical software moats — learned interfaces, custom workflows, searchable public data — are getting destroyed by AI. But one category gets stronger: proprietary data that can't be scraped, synthesized, or licensed.
A rigorous experimentation program produces exactly this. The insight that favicons improved Google Search by reducing cognitive load during information selection isn't something a competitor could derive from public data. It came from running a controlled experiment and user research, in our product, and doing the work to understand the mechanism. That insight was ours. Everything built on it compounded the advantage.
Now imagine two companies building AI products. One runs experiments systematically and invests in understanding the why. Each experiment produces a verified insight. Those insights accumulate into a growing map of what users actually want, how they behave, what drives retention. Every new bet is informed by that map. The map is proprietary data. It's a moat.
The other company ships just as fast. Maybe faster. But each decision is a fresh guess. They don't accumulate verified knowledge. Their map doesn't get better. A year from now, both have shipped hundreds of changes, but one has a compounding advantage in decision quality that the other can never catch up to — because they never built the infrastructure to learn.
A culture that shares a mental model of the problem space — one that people agree on and that's also true — makes better product decisions. The teams and companies doing this will far outpace those that aren't. This is one of the best ways to develop product empathy and insight at scale. Not empathy as a buzzword — empathy as verified, shared, growing understanding of the humans using your product. That understanding compounds.
That's my current mission — deep empathy for users of AI products, built through the discipline of understanding why things work. The window to build this advantage is open right now. It won't stay open forever.
Final Thought
I'll leave you with the YoY revenue growth by quarter for Meta and Google as a reminder of what great experiment processes and cultures can reap:
