Quantitative Market Research: How to find out what people aren’t saying

📈 A reminder as a bit of grounding

As much as quantitative research is often based in numbers – Vaughan opened his presentation with a useful reminder: we are not dealing with data, we are dealing with humans.

You may think that as obvious once it’s said – but how many of us are guilty of not thinking of it that way when designing surveys? I’ll leave that hanging!

👀 So what’s the problem with a lot of quantitative research?

The sins of quantitative research cumulate into respondents not necessarily sharing the information you really want – or the things they really mean.

Vaughan suggested three places a lot of research goes wrong:

1. Setting the wrong tone – The context for many surveys is the respondent will often feel like they are doing you a favour when filling it in. If this is the case, don’t then repay their kindness by declaring ‘x questions are MANDATORY’. All of a sudden, this moves away from a human interaction into a non-human, forced space. We’re dealing with humans, not data – set the tone with your copy appropriately.
2. Being a bore – If every question on your survey is asking folks to rate things on a scale of 1 to 10 – that’s pretty boring. Make the experience of interacting with your survey as interesting and varied as appropriate!
3. Outstaying your welcome – Don’t be too long! Vaughan suggested 15 mins is about the max he’s seen anyone have any success with – but clearly, make it as short as possible and only ask the questions you really need answering. Avoid the ‘can we just ask…’ if you don’t really think you’ll need that question answered.

🤭 So, how do you help people to share what they’re thinking?

Vaughan shared two things you might want to consider…

• There is more to quantitative research than just numbers – Think of the question ‘On a scale of 1 to 7, how challenging is it to keep on top of your work?’
  • We could all answer this with a rough estimate. But… what does a 5 out of 7 really mean?
  • Instead, could you ask something where the scale is presented as words? Instead of a ‘1 out of 7’ – the answer could be ‘I could do it in my sleep’ – ‘7 out of 7’ is ‘I’m under far too much pressure and everything in-between.
  • These answers could give a more relatable, actionable insight into how folks are feeling
  • The same theory applies to pictures… could you use emojis to represent faces, rather than numbers?
  • The possibilities are endless! 🙂
• Don’t forget some people can be shy or reticent to share the whole truth, or, are unable to express what they really mean – in this case – go tangential or visual.
  • Vaughan shared the example where when answering the question about workplace culture (23.00 minutes into the video), they used pictures of trees to get a visual idea of how the team viewed the culture, rather than asking people outright.
  • A similar theory applies for feelings that are inexpressible using words – can you ask people to try a more visual exercise to give you the answer you need (26.23 into the video)

🏠 And two final bits of advice to take you home…

1. Let people say ‘I don’t know’ on questions! It may well be that they very simply don’t have an opinion on what you’re asking them.
2. Open ended questions can give you a window into the mind of respondents. They have a place in quantitative research. BUT, make them as specific as possible to get more meaningful, more readily analysed answers.

Q and A on quantitative market research

[35:15] Q: Is there a right or a wrong way to go about incentivizing respondents (ex. entering respondents into a raffle)?

Yes, incentives work. What we’ve found is that giving out four £100 prizes beats one £400 prize. A complete no-no is to use your product as the incentive because that crosses the barrier into self-promotion.

[39:44] Q: Is there a “golden number” to the amount of survey questions you found to get the best amount/quality of responses?

This is a “how long is a piece of string?” question. A survey about a utility will need to be much more succinct than one about motorcars. Typically, however, alarm bells will start to ring for us if a survey takes more than 15 minutes with 40+ questions to finish. There has to be a good reason to make it that long. You also want to tell them upfront at the beginning of the survey approximately how long it will take to finish it. Set expectations.

[43:44] Q: Is there a good way to ask the “segmentation” questions, or is it just a necessary evil that you may need to split your results?

If you make the questions binary, you lose virtually nothing. In fact, you gain a lot by doing so. More often than not, binary beats a scalar response.

[47:45] Q: Data people like bigger scales for regression analysis, but I feel humans prefer smaller scales wherein each answer is more distinct. Do you have a rule of thumb?

This is a difficult point to be dogmatic on because it depends so much on what it is you want to explore. Note that the gap between a 2 and a 6 is nothing compared to the gap between an 8 and a 9.

[52:15] Q: How do you create the narrative of the story in the first place?

Look at the world from the broadest possible lens rather than getting right into your particular concerns. Take them on a journey. Lead them down a “funnel”, getting more and more specific with your questions as they go along.

[55:57] Q: Are there better ways to motivate people to answer your surveys?

If you look at Steve Jobs, Jeff Bezos, and other wealthy people, you learn that you get results by making things as simple as possible for people. Get people to your door, whether it’s, say, via a QR code. Then, once they’re at your door, guide them in, sit them down, and make them a cup of tea. Get them relaxed and motivated to stick around.