Category Archives: Market research

Evidence: How much is enough?

I recently tried to identify why the back brake on my bike was not working well. After taking the brake system apart, I discovered that the lever at the handle was badly bent out of shape. Reason for malfunction found! Could not be bent back, brake system replaced, done.

When you are running an organization, your problems are much larger. Which technology should a company invest in to gain a strategic advantage? Which product benefits should be highlighted to consumers? Which new features will users want in a year from now?

One way of making these decisions is going by your gut, or your industry expertise. But it is probably good to collect feedback from the world outside your building and do some (market / customer / user) research.

How much evidence do you need to address your business problem? Will a few user interviews do the trick? Do you need a survey or conduct some ethnographic research? Or is the information already available somewhere, in your in-house data bases?

Let’s start here: The research you conduct should be proportionate to your problem. If you are pondering a major change in strategic direction, you should probably take your time, allocate a considerable research budget and think deeply about which data you want to collect and how it will help you make your decision. In contrast, if you want to tweak a minor feature, a few quick user interviews or a small scale A/B test may do the trick.

Also, research to support business decisions is not an exact science. There are no standards for how much research a particular problem warrants. There is always more to be learned, always some doubt remaining about whether the evidence was sufficient. How do you deal with that?

It’s simple! Time and money are the key factors in determining where to draw the line in your research efforts. Business decisions need to be made in a timely manner, and if the strategy session with the CEO is in six weeks from now, your research needs to be done by then. Likewise, if you have a research budget of $30k you cannot commission a project that costs $100k. Business research is conducted in support of business goals and business decisions, and these are constricted by time and resources.

But…

Sometimes, you want to recommend that the company push back its strategy session by a couple of weeks to allow for collecting more comprehensive information about the customer. Or you lobby for an expansion of the research budget so that the size of the research better fits the size of the problem.

If you are unsure if the evidence will be enough, work from the concept of understanding. After collecting your evidence, do you feel you understand the customer enough to move ahead with your business decision? Or is there something missing?

If you feel your understanding is insufficient to make your business decision with confidence, go back, collect more evidence. If there is no time / no budget, frame your research results with caution signs, so that their limitations are clear to all.

Often, organizations develop a sense over time of what constitutes an acceptable amount of evidence. Some are happy with 4-6 interviews to explore fairly major problems, others want a large-scale study. Some don’t trust qualitative insights at all and feel that a quantitative survey must be conducted to support important decisions.

Usually, research results are not perfect. They do not meet all your insights needs. They do not include all possible stakeholders. But they provide valuable real-world guidance from users and customers that counter-balance gut-based decision making. Now, gut and industry experience and business sense still play an important role and insights stemming from research are only a part of the bigger picture.

Plus, when research results come back, their relationship to your business problem is usually not one-to-one: brake lever bent = brake not working = install new brake. Rather, they are subject to interpretation and often beg further analysis.

Research is no replacement for business sense, vision and leadership. These must have the last word and take responsibility for the future path of the product or organization.

Advertisement

Post Disruption: Market research, what next?

Based on my personal experiences

Around 2011 the owner of the company that I then worked for came to Toronto and challenged the employees to go on social media. At that point, few had considered how important this was about to become for our jobs.

In the pre-historic days of market research, consumer data were collected by walking around the city, knocking on doors and interviewing people face to face. I caught the tail end of this when I started in market research in Russia (which is another story). Then, call centres became industry standard and most survey research was conducted on the phone. However, qualitative discussions were still held in person at focus group facilities.

In the 1990ies, online surveys took over as the main means of eliciting consumer data. Rather than finding consumers through random digit dialing, companies now started to build online panels of people who were willing to answer surveys. There was some concern about the representativeness of these panels at first, but it was soon put to rest when the considerable cost savings of this approach became apparent. Initially, research companies needed specific expertise, expensive software and sufficient server capacity to program, host and run these online panels and surveys. Quite a few even tried to develop their own software.

The 2000s saw a rise in self-serve online research solutions. The idea of software as a service (SaaS), and ‘freemium’ products such as SurveyMonkey made it possible for anyone to collect the data they wanted. It seemed no longer necessary to have any particular expertise or resources. At that time, I felt that many established market researchers were reluctant to open their eyes to the new reality. A few companies were blazing ahead, but the majority sat back and hoped it wouldn’t get too bad.

I realized myself around 2012 that things would never be the same again. When I had the opportunity to start my own business in 2014, I jumped with both feet into the world of SaaS, social media and startups, looking for ways to innovate and carve out a niche for myself in the insights community. I read Clayton Christensen and Eric Ries. I experimented with different tools and methodologies. I immersed myself into WeAreWearables Toronto, then a monthly event at the MaRS Discovery District, an innovation hub that offers services for startups and scaleups, and various other incubators and accelerators around the city.

Innovation was the buzzword, and also agile, real-time, lean and design thinking. The word disruption became popular in my neck of the woods a little later. Hard to believe that this was only six years ago. Now the word already seems a little tired. I feel that the dust of disruption has settled in market research, and firms have made adjustments.

Those who started off with the new technologies – for example Qualtrics, or the Canadian company Vision Critical – have probably done very well (not that I know their books). The traditional firms have incorporated various technologies into their offering. Many have social media analysis products. Some run large-scale customer feedback platforms for their clients. Some use virtual reality goggles for concept testing or shopper studies. Some offer online communities or hold virtual discussion groups. And to cut cost, many move business functions into the cloud, and outsourcing is widely used.

Whether this is sufficient to keep traditional market research organizations profitable, I don’t know. What I have learned in my own business is that I am still selling ATUs, segmentations, concept tests and one-on-one interviews. After the frenzy to be innovative and different, what clients seem to appreciate is my expertise. They trust me to know how market research is done properly, to execute the project for them, and to deliver them results that reflect a thorough understanding of their business questions, and in a form that adheres to their internal processes and requirements.

That’s why I stopped following the hype in the last couple of years. Disruption happened, but that was in the past. No amount of research automation can substitute understanding. And understand the customer we must.

But…I recently started renting a co-working desk at a business centre. I am with ease twenty years older than the majority of the other tenants there. So now I am back in a startup environment, and I see that while the hype has decreased, startups are here to stay for the foreseeable future. Businesses who are trying to disrupt various fields are numerous and, with the myriad applications of AI, far from done.

It will remain important to understand how new technologies fit into and change existing businesses. For example, I am intrigued by the anatomy of the Cloud, where servers are located, how data are moved around and what implications different factors have on data loading speeds, data security etc. When some people that I recruit for surveys complain about it taking longer than they thought, is that because their Internet is slow, or because the survey hosting company has switched from using their own servers to the cloud?

Data governance also interests me. With many market research companies using subcontractors, it is almost impossible to see all the way down the supply chain where data may be stored, processed or transferred to. The business risk related to data governance has increased exponentially for market research firms. Good times for lawyers and insurance companies.

But understanding AI, Cloud and computing in general is and will be immensely important for anyone interested in the affairs of this world. I think this is a topic that they should teach kids about at school, so that future citizens can make informed decisions about it. I want to learn more. So far, I have read three books on AI – it’s a start.

Barbara’s AI reading list:

  • Kartik Hosanagar: A Human’s Guide to Machine Intelligence
  • Ajay Agrawal, Joshua Gans, and Avi Goldfarb: Prediction Machines
  • Virginia Eubank: Automating Inequality

Update, also… Janelle Shane: You look like a thing and I love you

Why Startups don’t do market research (but should)

Startups are great at coming up with innovative ideas, but why don’t they ever do market research?

“A startup is a human institution designed to create a new product or service under conditions of extreme uncertainty”, said Eric Ries in The Lean Startup. Extreme uncertainty is a defining feature of the startup experience. Uncertainty about the product, the supply chain, the customer, the business model. Two key success factors are experimentation and customer orientation. They enable the startup to achieve product-market fit and to creative a product that can eventually be commercialized.

Getting input from target customers is essential in developing a great offering. Yet, when I talk to startups about conducting market research, many dismiss the idea.

Often, startup founders believe market research is only for large multi-nationals, or that it comes later in the evolution of a company, when a company is established and has multiple successful product lines. Some think it is too expensive or they don’t fully understand the ROI. In addition, many founders tell me that they already have all the customer information that they want.

Let’s look at how startups typically gain information about their market, and what they are missing out on by not conducting market research:

1. At the early stage, founders’ market research typically consists of talking to ‘friendly individuals’ who are willing to give some feedback on the startup idea. The problem here is that the selection of individuals is very likely biased towards people who have a positive affinity to the founder and/or to the product to begin with. It is simply easiest to reach out to your existing network. How objective is the feedback of these individuals going to be?

In addition, we tend to connect with people who are similar to us – similar in age, in gender, in ethnic background, in social status – as it lets us stay in our social comfort zone. How representative are these people for potential users of your product? Particularly if a startup is developing a health-related product or service, potential users are likely very different from the twenty-something healthy males who tend to populate the startup community.

Plus, the ecosystem within which founders operate tends to be highly tech savvy. Getting feedback only within their own ecosystem gives founders a rather poor idea of how regular people would see their product. Granted, tech savviness is going to increase in the future. But do you really have enough time (and cash reserves) to wait for fifteen or twenty years until the majority of the population has caught up with you?

Opportunity missed: Conducting market research early on may help a startup discard an idea more quickly that won’t be popular outside their immediate circle of friends and like-minded acquaintances (‘fail early, fail fast’). Real market research means soliciting feedback from individuals that are not personally connected to you, and who belong to the intended market segment for your product.

Many startups use Kickstarter/Indiegogo as a means to assess if there is really a market out there for their product. While it is comforting to know that two thousand people are backing your campaign and are willing to pay good money now for something great to materialize in the future, this is not enough to build a successful company on. Also, let me tell you a little secret as a market researcher and anthropologist who is in touch with the ‘common man/woman’: Many people have never heard of Kickstarter or Indiegogo.

2. As startups mature, products typically go through many iterations and user experience is incorporated in a continuous feedback loop in the product development process. The UX lead will bring in users, expose them to the latest version of the product and propose adjustments accordingly. In addition, the product itself, assuming that it has a digital component, sends back usage data, which allows the startup to assess how well certain features are performing and conduct A/B tests.

Having a mountain of data that streams back through the device or app makes founders confident in their ability to optimize the product. While this is certainly the case, at times it is hard to ‘see the forest for the trees’. As startup guru Steve Blank said, you need to ‘get out of the building’ to understand your potential market and to learn about your potential customers.

Usability research and user experience research are essential for product development from a technical standpoint and from a design standpoint. In contrast, market research helps you understand the broader picture, beyond your immediate product to prospective customers and their evolving needs and pain points for which your product (along with many others) may provide a solution.

Some methods employed for market research are similar to user research, and some are different. Market research consists of the full gamut of large-scale, quantitative surveys, mobile quick check-ins, trade-off exercises, ad tests, focus groups, location-based research, ethnography, online discussion boards, co-creation labs, in-depth one-on-one interviews and many more.

Opportunity missed: User research looks at, well, users. Market research looks beyond. Only input from a wider target audience will help your startup understand where your product is missing the mark in a fundamental way.

You should also engage in market research as a ‘disaster check’ before going to market. While your product may be delightful and great in terms of functionality, perhaps you are not communicating something that you think is trivial, but that consumers want to know. Safety in wearable devices is a good example. It may be obvious to you that devices have to undergo rigorous safety testing before you are allowed to sell them to people, but the regular person may not be aware of this. If you launch a new wearable without a footnote on safety in your communications, you may be impacting your sales in a significant way. While safety may not be the reason people want to buy your device (unless it is a safety device) they want to be reassured that it is not going to harm them.

Then, after a successful product launch, your startup will only be able to continue expanding its market if you find out what non-users need and want, and if you understand their perceptions of the competition. A successful product fulfills a need better than other products. To find out what target customers need and how your product stacks up against others in addressing that need, you have to engage in market research.

Market research enables startups to step out of their comfort zone, go beyond speaking to the ‘enthusiasts’ and ‘evangelists’ and build an offering that resonates with a larger audience. Do not miss out on this opportunity to critically examine the appeal of your idea and to lay the foundation for exponential growth.

Market research for startups has clear ROI in the following areas:

  • Establish the size of your potential market (ROI: A key talking point when you approach investors at your next funding round)
  • Determine size of different segments within your market and prioritize them in terms of their potential for your product (ROI: Don’t waste money and effort on marketing to a segment that you have personal affinity to but that is tiny or less likely to buy your product)
  • Uncover needs and pain points of potential customers (ROI: Allows you to pivot early, and conserve precious early funding, if your product misses the mark)
  • Validate and refine user personas (ROI: Supports better product-market fit and sets you on a growth trajectory beyond current users to potential future users)
  • Optimize marketing and communications to focus on the product benefits that potential customers most care about (ROI: More bang for your advertising buck)
  • Get an understanding of competitive offerings, how your product stacks up and how to position your product for success (ROI: Stay ahead of the competition and get a bigger share of the pie)
  • …and last but not least: Obtain unbiased feedback on what people really think about your product (priceless 🙂 )

Some questions that startups may have about market research: Is it expensive? Can I do it myself? How do I go about it? There are many reasonably priced ways to conduct market research. Some components can easily be done by the startup itself. However, it is important to get expert advice on methodology and on the design of the research questions, to ensure your research is representative and unbiased.

If you’d like to get more information on what market research can do for your startup, please contact me. I am happy to give you an initial consultation free of charge. You can reach me at: barbara@creativeresearchdesigns.com

Also published on Medium

Our online and offline worlds

I am a student of human behaviour. When the Internet first became a thing, we used to compare online and offline behaviour. As if we were one person while we sit at our computer and another person in real life. Maybe it was like that in those days. I am talking ten, fifteen years ago.

The world of communication was segmented into different channels: television, radio, print etc. Talking like that does not even make sense any more. Today a typical user experience includes interacting with content and with people across a number of platforms, in a more or less fluid fashion.

Vinu George, Market Intelligence and Customer Insight Manager at Microsoft, recently described this in an article in VUE magazine as follows: We are now moving to a five- screen world…large-screen TVs, gaming consoles, laptops/PCs, tablets and smartphones. Content is now consumed and created across these screens. We are moving from one screen to the next to the next, reading, watching, posting, commenting, sharing online, sharing online offline (Look, mom, have you seen this video?).

Up until recently, I have not been a technology junky at all. But with four out of the five interfaces at my disposal, and discovering the infinite possibilities of social media, I find it more and more difficult to differentiate between online and real life, it is all just life.

Having school-aged children also gives me a privileged view into the future of online immersion. Many parenting experts advise parents on limiting screen time for their kids. Which I agree with. The trouble is, there is not just brain-dead consumption of junk going on, there are lives lived, and they are lived in part through electronic platforms.

As a market researcher, I wonder if our methodologies really address this level of immersion in the online world and the fluidity with which online and offline experiences are intertwined. Rather than focusing on one interviewing medium and throwing in a bit of social media analysis or a few ethnographic observations for good measure, how much richer and more insightful could a truly integrated multi-media exploration of behaviours and attitudes be?

Mom and baby

Sea Change in Market Research

The art and skill of market research lies in asking the right questions and drawing the right conclusions from the answers. I know how to ask questions.

I know how to ask them online, on the phone, in person, in a fashion that makes responses quantifiable, in a fashion that allow us to publish the results, in a fashion that elicits emotions, in a fashion that minimizes bias, in a fashion that entertains my clients. I know how to ask questions to old people, to young people, to people with illnesses, to people with children, to CEOs, to large donors, to physicians, to nurses, etc. etc.

In a house, with a mouse, in a box, with a fox, here and there, I can ask questions anywhere…

What if market research is no longer about asking direct questions to real, live people? Why are we asking questions anyway? Our clients want to know what people think and feel, and what they will do, based on their thoughts and feelings. How they will vote, who they will support, what they will buy.

Much of this can be elicited from data that is produced without asking questions. I recently read an article on how you can predict someone’s age, gender, sexual orientation, level of education and the emotional state he or she is in relatively accurately from the pattern of likes they leave across the Internet. Predictive modelling is the name of the game. How can you link likes, content of posts, tweets and comments to action, online and offline? The best people who develop these algorithms sit no longer in traditional market research companies.

They sit in large IT companies. Or they sit in smaller digital shops, where they specialize in a particular thing. And probably in some large financial institutions. And government think tanks.

What do they understand about people? What do they not understand about people? What do my clients need me for? Sure, I know my clients business. I consult. I interpret. I put things in context. At the end of the day, it is still all about making the right connections. So you know what pattern of online behaviour precedes a purchase. Now what? What information do you really need, and how do you use this information to your advantage? That is where the consultant comes in.

To do the job right, however, the consultant needs to understand what kind of information is out there, what is technically possible, what is practical and what is economically feasible in terms of analysis. And to stay on top of that is becoming more and more time consuming with the data explosion in which we are currently caught up…