Conducting More Informative Survey Experiments - by Rudy Venguswamy
As part of the Arrow Capital team, I have had the privilege of meeting tons of amazing early stage founders in the UC ecosystem. A common process that I see founders experience, especially in the very early stages of building their startup, is attempting to find product-market fit. In fact, this is perhaps the most crucial learning process that a team undergoes, pivoting off of ideas and nudging their product and team as they learn more about their customers and industry. Finding the right pain point, the right product, and the right target customer who will pay for the solution all requires great hunches, industry experience, and lots of learning along the way. The process of learning and converging on product-market fit is messy and contains quite a few pitfalls.
In this piece, I'm going to elaborate on a major issue I see with a popular method used to evidence and validate product-market fit: the online survey. I’ll outline some alternatives rooted in both design-thinking and research design, which admittedly take slightly more effort, but provide more convincing data to validate your startup hypothesis. When the Arrow team and I look to invest, clear evidence of a need that is being met with a startup’s innovation is vital to our excitement about a company.
Nearly every founder conducts user interviews as part of the process of finding product-market fit. This stage is often accompanied by commitments to design-thinking philosophy and customer validation interviews. These evaluations are compelling and when done right, provide great insight into both the team and the kind of challenges they are tackling. However, things can take a turn for the worse when it comes to validating with a larger audience, which requires posing these important customer validation questions in a way that scales. In about 40% of pitch decks I’ve seen, I've noticed the appearance of a slide like this, aimed at showcasing a large pain point:
Fashionable, Red T-Shirt Company
The kinds of surveys that generate these types of charts come from bad survey designs and do not yield the results that founders hope to get from expending the effort needed to gather broad customer validation data. I'll illustrate this by elaborating on the example chart above:
Let's say for this hypothetical company, they came up with an idea through the design-thinking framework, are convinced that not enough people have red fashionable wear, and now begin asking themselves whether their idea is appealing to a broader group of people, beyond those that they initially interviewed who were all obviously willing to try the product out. To do this, they land on a simple online google forms survey. Though clearly conducting 100+ more in person interviews would be ideal, the time cost for doing so would be unreasonable. It makes sense at this point after at least 20+ user interviews to spread a wider net with a generalized survey.
They keep it simple and ask on their survey, "If you didn't have red in your wardrobe, would you consider buying our t-shirt?" and "Would you dress more fashionably if it were cheaper (on a scale of 1- 5)?" They post their questions, incentivizing responses through offering a raffle for a $50 Amazon gift card, and invite everyone on their Facebook network to join. The more thoughtful founders consider the sample bias and take an extra step to seek out and send their survey to a more diverse set of participants who might match their target audience.
The team, after about two weeks, checks in on the results and is relieved to find that 85% of respondents would buy the shirt! And, quite a few respondents (72%) rated 4+ on wanting to dress better! With those great results, they go full-steam ahead to start selling. They convince friends and family to give it a go and pitch at a few competitions. Finally, when it comes time to sell, they create a landing page, start talking with manufacturers, and pay for ads online. However, to their surprise, the click through rates on the ads are abysmal and nearly every customer on their website is friends with a member of the team, not a new face. From here, they may choose to pivot or persevere, but it took them a full production run and a lot of spent social capital to realize there was an issue.
The plan of action this founding team took seems rational, and in fact, a larger survey definitely can follow appropriately after the design-thinking stage, since it's easier to get positive replies face-to-face rather than online. The issue, though, lies in their survey questions. The kinds of questions the team posed in their survey differ from good research design practice and exemplify a trap founders fall into when not wanting to find bad results. By asking hedged questions like "If you did not have red in your wardrobe..." or questions like "Do you wish you dressed more fashionably for cheap?" which almost beget a positive reply (who doesn't wish they dressed better for less?), founding teams confirm their own startup idea hypothesis without creating room to disprove it as well. Framing the wording ever so slightly can make it difficult to get a "no" and bias responses. In other words, the founders asked “scared” questions. This is not uncommon for early-stage founders who understandably want to set their product up for success. Ultimately though, scared survey questions lead to overconfidence and in fact don't accurately reveal whether or not a product has potential.
Instead, a better solution is to err on the side of disproving your hypothesis with tough constraints. Ask harder questions you might expect a decent amount of people to say “no” to: Are you willing to sign up and pay for the product right now? What if my product was not cheaper than the competition? What is the reason you haven't bought (similar product) yet if you felt (pain point)? Set up realistic scenarios for your product such as a landing page where people can sign up for a trial. Rather than say, "If our product could do x,y and z, would you buy it?", say "Our product does x, y, and z and will be shipping shortly. Are you willing to preorder it?"In the real world, it's very easy for consumers to say "no." Think about how your surveys can more accurately reflect this reality.
In the best case scenario, you're pleasantly surprised to find that the product has customers willing to pay and has strong data to validate the startup idea! On the other hand if you find people are not willing to bite, the disappointing results don't have to be terminal. Relax the tough constraints and replace them with easier ones (i.e. free trials/samples, surveys over the phone where people feel more inclined since they know a human is on the other end, educational blurbs that persuade). If it turns out that getting a customer to bite takes an in-person conversation that provides lots of education about the product, that insight likely saved your team months (high contact sales exist, but usually demand much higher ticket prices, which might suggest a change in the target customer or business model).
At first, it may seem counterintuitive to work backwards from tougher to easier in an effort to get customers to agree with questions/requests on online surveys, but this methodology actually allows a team to move quickly and effectively pinpoint the things that sell the product and the things that don't. Rather than hide the fact that the product isn't a wonder-product that self-sells without any founder-driven momentum (word-of-mouth and virality take lots of initial customers and don't happen without a founder push), teams should have a tempered and realistic idea about the kind of message and product scope that motivates customers while assessing and internalizing the aspects to avoid along the way during a scale out.
Undeniably, this new approach takes more effort than blasting a survey to friends on the founding team’s part. But, the upshot of asking tough questions in surveys is the exponential increase in the speed of customer discovery, insight, ideation, and ultimately, idea validation. Nothing changes a product for the better than finding out the least resource intensive path to a customer. In the long run, this technique will save the team hours of work later down the line.
Knowing the sales and pain point thresholds will elucidate what your sales process would need to look like and showcases founder pragmatism and empathy. When brought to the table, this insight can serve as a competitive advantage as well. The results of your acuity will be apparent and impressive to any investor.