Why Growth Hacking and A/B Testing Are Bullshit Before PMF
Before PMF, growth hacks and A/B tests won’t help you much. What you should do is use unscalable methods to build a ‘10-star’ experience that users rave about.
Startup founders love growth hacks. They A/B test everything and think about scale from day one. Build an MVP, get some users, then keep tweaking: feature A lifts retention X%, feature B boosts conversion Y%. It looks like methodical progress. But before product-market fit (PMF), this rush to optimize may be a dangerous distraction.
Before PMF, growth hacks mostly don’t work, and even after PMF, your real job is still the user experience. Airbnb got started by doing things that don’t scale. When their New York listings weren’t getting booked, Joe Gebbia and Brian Chesky rented a camera and went door to door photographing hosts’ apartments. Within a week, revenue in New York doubled.
At that stage you barely know your users or their real problem, so you don’t even know what you’re optimizing. Growth hacking then is like polishing a car with no engine: it looks busy but goes nowhere. Plenty of startups are led astray by three assumptions: that small, data-backed tweaks are always the best choice, that “data-driven” automatically means scientific, and that scalability is a day-one requirement. Airbnb’s story shows all three are wrong. They put their effort into the user experience instead of optimizing early, and found a higher ceiling.
Do Things That Don’t Scale
Paul Graham said: “Do things that don’t scale.” Many people quote it, but few really get it. Early on, this unscalable grunt work is the main road: talk to users, onboard them by hand, be their concierge. The more you talk to users, the more raw, unfiltered feedback you get. You see their pain, and you see the “aha!” moments that make their eyes light up. Dashboards don’t show any of this.
Startups can also offer an “insanely great experience” that big companies can’t. Tim Cook won’t send you a handwritten thank-you note, but you can. That kind of attentive, even over-the-top personal service helps you understand a niche deeply and win it, and what you get back is fierce loyalty as well as feedback. Even if you only serve a handful of people well, a great early experience turns them into your sales force, and that works better than any early ad spending. (Data matters too, of course. It’s just that before PMF, you need to do the right kind of learning first.)
The 10-Star Experience
Airbnb didn’t start by A/B testing button colors. From the beginning, they obsessed over the user’s whole journey. Brian Chesky had his team design a “10-star experience” by imagining what each star level would look like. At 5 stars, you arrive, the host lets you in, and everything is as described, which is what you’d expect. At 7 stars, the host welcomes you in person, your favorite snacks are in the fridge, and a surfboard is ready for you, which is delightful. At 10 stars, you get a Beatles-style arrival, with 5,000 fans cheering for you at the airport. That’s absurd, but it opens up your thinking. At 11 stars, Elon Musk personally flies you to space, which is beyond absurd. Nobody actually planned to build an 11-star experience. The point was to break “good enough” thinking and find what makes users rave: dream big, then work back to a 6-, 7-, or 8-star version that is both amazing and doable.
To make this real, Airbnb brought in a Pixar animator and storyboarded the entire journey (the guest’s, the host’s, even the employees’) in 45 frames, each showing the context, emotion, and motivation of that moment. These “Snow White” storyboards made a big difference. They made everyone see the product as one whole experience, so for PMs it was no longer just a pile of features to A/B test. They pinpointed the key emotional moments (check-in, first contact, building trust), which is where users come to love a product. And they gave the whole company a shared “movie” script, so everyone’s expectations lined up. With these storyboards, the team knew which moments to aim at for 10 stars.
This is where “doing things that don’t scale” met the 10-star dream. Airbnb first prototyped these ideal experiences by hand for a few users, to prove they were worth scaling. The photos I mentioned at the start are one example. It was 2009, Airbnb’s New York listings were doing terribly, and they suspected bad photos were the problem. Neither Gebbia nor Chesky was a professional photographer, but they knew good photos mattered. None of it could scale, but it proved that professional photos convert, and only then did they build a scalable photography program.
The founders also stayed in hosts’ homes and took notes as they went, and features like user profiles and two-way reviews came out of that deep, unscalable empathy. The team also planned a 10-star trip to San Francisco for a test guest named Ricardo, taking care of everything from the airport pickup to curated activities, and Ricardo cried with joy. The experiment showed how strong human connection can be, and the scalable product that later came out of it was Airbnb Experiences. The pattern is clear: define the extreme, storyboard it, hand-craft it for a few users, and see whether it’s magic. If it is, find a way to scale its value, without copying the manual work you did at the start.
Don’t Hand the Decision to the User
Chesky said, “A/B testing is abdicating your responsibility to the users.” He wasn’t against all A/B tests. What he was against was misusing them for big decisions, or getting stuck on tiny improvements while missing the big picture. If Ford had only A/B tested faster horses, he never would have invented the car. Airbnb’s usual approach was to start with a strong, experience-driven hypothesis (for example, “pro photos build trust”) and test it qualitatively, on a small and unscalable scale. If it worked, they put resources into scaling it and measured it with long-term metrics such as retention and referrals. Only then did they use A/B tests to fine-tune the scaled version, for example, how to offer the photo service to hosts.
Once you start obsessing over user experience while searching for PMF, it never stops. After PMF, piling growth hacks on top of a clunky experience won’t get you far either; users can tell a carefully designed journey from a patchwork of optimizations. Early growth is often unscalable. It takes manual work, deep empathy, and a constant effort to delight users. Do things that don’t scale to find what’s truly valuable, and only then will you have something worth scaling.
So before you jump on the latest growth hack or fight over a fraction of a percentage point of conversion, ask yourself: do we really understand our users? Is there any part of the experience we’ve built, even a small one, that makes them rave?