“There is a world market for maybe five computers,” confided IBM chairman Thomas Watson in 1943, while – almost four decades later – a young Bill Gates predicted that “640k of memory ought to be enough for anybody”.

Over the past 70 years, an avalanche of disruptive technology has transformed the research industry, so why should the future be any different? Can we predict what we cannot imagine?

Passive data is flooding our personal and business worlds: wearables measure our bodies; telematics control our cars; there are more mobile devices than people in the world, their numbers growing five-times faster than the human population.

Gartner forecasts that more than 26 billion objects (excluding computers, smartphones or tablets) will be connected to the internet by 2020 – a 30-fold increase on 2009’s Internet of Things. As homes, wearables, vehicles and personal data exchanges become more connected, the devices will become smarter; shifting from reacting to predicting.

As our interactions with each other, our suppliers and customers, and the state increasingly take place via a device, the trillions of recorded daily interchanges become the new market research data. Recipients of insight will become blind to the sources of information, concerned only with their utility and ability to deliver competitive advantage.

If big data is today’s disruptive revolution, our task is to find within it that which adds value to business decisions – a process of winnowing or sculpting rather than sampling; a new mindset of data selection, not just data collection. This means customers, employees and stakeholders will be connecting with researchers in a new context – and this is where behavioural economics, organisational psychology, ethnography, anthropology, and a host of other scientific frameworks for understanding people, can add value. If we can use these to pull practitioners out of those grooves, let’s embrace it.

All of this will help break down the barrier between insight and action. Market research will no longer be things collected some weeks ago from a subset of people answering only the questions we thought to ask. It will be a constant interaction with the real world and in real time, helping us to better understand what is actually going on, to predict the impact of actions more fully, and to make improved business decisions.

As an industry we are the interface between business and the world. We will have to consider how we are organised to deal with the scale and speed of information. Data visualisation and storytelling are only a fraction of the future. Old measures of quality and rigour will have to be rewritten. 

The relationship between research agency, insight department and business stakeholders will need to evolve. Creativity will come into data selection and not just data use. We will need to save time in the gathering phase so that we can spend more time driving value from the insight and deciding how to act. Businesses that neglect the new market research world will be forever stuck in their old grooves.

Honor Mallon is partner of PwC, research to insight (r2i) and she is leading a collaboration with MRS to evaluate the UK research sector, in a report due in spring 2016