So, Big Data. The market research industry continues to struggle with the concept. It was one of the buzzwords of 2013. Some have come up with a big data offering. Some are searching for a point of view on it. Some counter with small data. Many still have only a vague sense of what we are talking about.

I have asked many colleagues and clients what this concept means to them, in the hopes of developing a brilliant solution that would make me wildly successful. Well, this seems to be taking some time, but anyway, I’d like to share with you what I have learned so far. As I am working in the healthcare sector, this is my focus below.

1. Big Data (in Pharma) is IMS data

For some of my pharmaceutical clients, all they can think of when asked about large data sets is IMS data. IMS captures and sells information about the prescribing behaviour of physicians at the pharmacy level. Through this data, pharmaceutical companies track the sales of their products.

2. Big Data (in Hospitals) is Patient Records and Interaction Statistics

Healthcare providers, particularly hospitals and other large organizations, capture myriads of data on patients flowing through the system. The analysis of this data is largely off the radar screen of traditional market research, and falls under the discipline of health informatics.

3. Big Data is Social Media data

This is a view that many market researchers adopted when social media first appeared on our professional horizon as another form of human expression. Last year’s MRIA NET Gain conference, dedicated to big data, featured a number of presentations in this area.

For those who do not want to develop their own proprietary solutions, subscription-based social media analysis tools are available and used by both end clients and market research vendors.

4. Big Data analysis is a different way of saying Data Mining

Some sectors have worked with large data sets for some time. I am thinking of scanner data in retail, and loyalty programs (Air Miles, Petro Points etc.). Fifteen or so years ago, the statistical techniques used to sift through such data sets were called ‘data mining’.

This practice is still ongoing, and the size of data sets ever increasing with more and more customer touch points being added. Some think of this type of analysis, when hearing the words big data.

5. Big Data is Data that is created by Machines

This type of big data is rarely mentioned and obviously not in the forefront of a market researcher’s mind. However, it has grown exponentially and is increasingly viewed and used as a source of customer information.

For market researchers, the question (and the fear) is to what extent human analysts are still needed, and to what extent ‘the machine’ can do it on its own. And how we can make sure we are still needed.

We say: “You need an expert to interpret what your data means.” We say: “A consultant is needed to guide the analysis process.” We say: “Meaningful data analysis is the development and testing of hypotheses, and only people can come up with those.”

And we are right.

How many times have I looked at the results of a statistical analysis and said, “This does not make any sense.” And then we discarded the analysis and started fresh, because results need to make sense. To another human. To your client. They need to lead to actionable insights and recommendations.

So we are still needed. But…

But many, many processes are now automated, from data analysis over producing charts even to highlighting key insights in charts. Far fewer people are needed to work with the data then before. Take a look at www.beyondcore.com – will get you thinking.

Some companies function with very little market research, in the sense of interactions between real live researchers with real live respondents. Machine-generated user data that streams back from devices guides the refinement of these products. New Apps are developed by split testing and seeing how early users interact with certain features, following the logic of clicks. Service companies integrate their customer interface and CRM software with their enterprise management system. Automated cues let managers at different levels know how they are performing, and notify them if there is a problem.

Technical skills are essential for survival. How can you tell a successful agency these days? If you look at their ‘careers’ page, most of the open positions are for developers (i.e. IT people). Those who win in providing business intelligence are either companies who are focused on the digitalization and automation of data collection and analysis, or companies who make intelligent use of available software products and platforms within the research process.

Do you know what Hadoop is? A wireframe? CSS? If not, perhaps it is time to google it right now…