Where traditional surveys prove to be too expensive, too slow or unreliable, innovative researchers have been discovering the power of transactional data to tell the story instead. In 2013, for example, Thoralf Gutierrez and Vincent Blondel, of Louvain University, and Gautier Krings, of Real Impact Analytics, used data from a mobile phone company in Cote d’Ivoire to produce a detailed map of poverty. “Many developing countries do not have up-to-date statistical information about the state of their population,” they wrote. “Surveys are onerous; they are not conducted very often… Even when they are conducted, their reliability is questionable.”

Their insight was the behavioural observation that poor people topped up their phones with smaller amounts of credit, even if it meant they wasted time and ran out of credit more often. By mapping the top-up amount per transaction, using geo-location data, the trio measured purchase averages in each region (an indication of average disposable income), the variation in those averages (income diversity) and calculated the Gini Index (to measure inequality). The results were far more detailed, more up to date, and made more intuitive sense than census results. They used pre-existing data, and were quick and cheap to produce.

Finding new truths in passively acquired data can’t fail to capture the imagination. The idea that this type of analysis might be all the insight you need is one theme of Big Data, the book by Kenneth Cukier and Viktor Mayer-Shönberger that has become a sort of Freakonomics of this data revolution. Much of the book, and the many TED talks and interviews the authors have given, make uncomfortable reading and listening for market researchers. “In many instances, we will need to give up our quest to discover the cause of things in return for accepting correlations,” they write. “Big data helps answer what, not why – and, often, that’s good enough.”

Cukier and Mayer-Shönberger assert that it is sufficient in many cases to know that C occurs when A and B happen, whether or not we research why A and B might cause C. At the same time, the cost of capturing A and B from transactional data is falling, creating a wave of creativity as analysts are empowered to play with data. 

For example, techniques to establish causality by measuring what people do are moving from econometric textbooks to practical reality. In a 2014 paper in Business Economics, Professor Hal Varian – who wrote the textbooks for a generation of economics students, and who is now chief economist at Google – wrote of the potential for transactional data to establish the effect of treatments using automated experimentation.

“Google runs about 10,000 experiments a year in search and ads. There are about 1,000 running at any one time and, when you access Google, you are in dozens of experiments,” Varian wrote. 

“Now we have ‘computer kaizen’, where the experimentation can be entirely automated… continuous experimentation will improve the way we optimise business processes in our organisations.”

While few data scientists would claim that data entirely replaces market research (Cukier and Mayer-Schönberger only claim “many instances”, and Gutierrez et al needed previous research evidence to create their hypothesis), some researchers worry that it will be crowded out by admiration for the scientists’ boast that ‘N = all’.

Ray Poynter, managing director of The Future Place and chair of NewMR, warns that this is already happening. “Big data looks easy to clients – the claims made by some vendors would make a second-hand car salesman blush,” he says. “And there is a risk that the market research option will be ignored in the rush to get the latest, shiny, big-data toy.”

Better evidence

For many researchers and organisations, however, market research and data analysis are not in competition with each other. The hunger to make decisions based on better evidence can be a benefit to both, especially for those applications that synthesise research with transactional data analysis. Poynter picks out Sky’s insight department as an example of where research and transactional data combine, with “one showing where to dig and the other doing the digging”. 

This is the ‘sweet spot’: where each discipline benefits from the other. To identify and exploit this will require a profound shift in emphasis for many research agencies. Researchers who invest in a data specialisation or partner with other agencies that provide analytics – or become ‘data shepherds’ for multiple data sources – will more often be involved in long-term partnerships with clients. They will be expected to build insight delivery into the client’s decision-making tools. Many will have to cut back on their investment in surveys in the face of superior data sources. It’s a high-stakes proposition: if they are successful, they might achieve a more strategic role for their clients. However, this comes with risk; it might even become possible to pay them according to how much value their insights create.

The trouble with transactions

When data analytics is credited with spotting disease or predicting crime, is doesn’t claim to explain the choices and decisions that people made. However, for commercial applications, it’s necessary to know why something happened. This is often the foundation of the pitch for market research – that data can tell us what, but research tells us why.

Notwithstanding Google’s success at establishing causality using experiments, this has some truth: transactional data only captures what can be ‘datafied’ as part of the customer relationship. At a basic level, this means even the most advanced loyalty programme can’t track customers when they choose to shop at a rival. For example, the causes of what Tesco’s Clubcard analysts used to call a “hole in the basket” are obscure: shoppers defect for price, service or convenience, and usually for more than one motivation. They might even have stopped buying the product. 

Detailed data analysis might be able to tell us more about the nature of the hole, but, usually, the most convenient way to discover why people do something is to ask them. If our recollection of why we acted is imperfect, or tainted by well-known biases, then at least research can provide a working hypothesis to test, using an experiment or a promotion.

Colin Strong, managing director of Verve Ventures and author of Humanizing Big Data, says responsibility for the what-why distinction has always been blurred, and is becoming more so. The industry can improve the quality of a ‘what’ analysis, he explains, by creating environments in which people feel there is an advantage to voluntarily submitting data that isn’t part of a transaction. He points to midata, the UK government’s initiative to allow people to see the data that is held about them. “The research industry can be part of the ‘what’ without necessarily doing surveys,” he says.

It’s perhaps more accurate to say that research can improve the quality of the ‘what’ and introduce a ‘why’ that transactional data can resolve through experiments or analysis. However, to do this, organisations will have to mitigate the weaknesses of their transactional data. Researchers know how to do this, because solving these problems – or, at least, warning about them – has always been a fundamental part of the survey process.

No data is raw

We are constrained by what can be measured and what has historically been measured. In the book Raw data is an oxymoron, Lisa Getelman explores the problem that data is not discovered – it is generated. By choosing what to measure, how to measure it and who gets measured, transactional data is ‘cooked’ even before it is analysed in detail. 

“The phrase raw data – like jumbo shrimp – has understandable appeal,” she writes, but adds that this involves the “assumption that data [is] transparent, that information is self-evident, the fundamental stuff of truth itself”.

Even the assumption that ‘N = all’ – that a large data set must be representative because it’s big – isn’t often true. Claire Emes, chief innovation officer at Ipsos Mori, says: “These data sets tend to contain systematic biases. For example, we did some real-time analysis of Twitter during the [General] Election, but we were constantly reminding people that only one in five people is on Twitter, and half of them are under 35. You can’t really use it to draw conclusions about the public mood, even if there are hundreds of thousands of people – I’d still rather have a representative sample of a thousand.”

Behavioural data alone is inconclusive

Measuring more – even accurately – isn’t always measuring better. Capturing many variables doesn’t necessarily lead us to a conclusion, especially when we are trying to model complex decision-making or choice processes. For example, covariant variables may ‘steal’ importance from each other. Are we happy because we are rich and have friends, or do we have friends because we are rich and happy – and how relatively important are wealth and friendship? Because all three variables move together, slightly different statistical models may conclude that money is more important than friendship, or that friends are all that matter, using the same underlying data.

At worst, analysis – even on a representative data set – reaches false conclusions because it uses the wrong model. “I was talking to a head of analytics at a media company,” Strong says. “She was saying: ‘I’ve got mathematical geniuses sitting in my office, but they often bring me stuff that I look at and say, “that makes no sense. Consumers would not do that”.’”

Transactional data reflects the world as it is

Is this a weakness? It is if the priority is to change that world. Transactional data may make clients more efficient, more aware of their customers, or more profitable: these can be instrumental goals – or even symbolic ones, justifying a decision that has already been made. By definition, transactional data simply cannot deal with counterfactuals. It may uncover problems that are the starting point for big-picture innovation but, after that, it is of limited use for conceptual research.

Google’s technique of performing automated A-B experiments can investigate ‘what ifs’, but this will rarely lead to disruptive innovation. When economist and political scientist Joseph Schumpeter described the process of innovation as “creative destruction”, he wasn’t thinking about changing the location of a button on a web page.

Guy Champniss, associate professor of marketing at Henley Business School, and the founder of Meltwater Consulting, experiences this problem: “Some people are sacrificing foresight for accuracy. All the data set can do is drive efficiency. I’m not convinced that the focus on data can improve effectiveness. It doesn’t allow us to think how we can redesign the experience of our customer. It allows you to tweak it, but it’s all incremental.”

Even this incremental efficiency might not be useful in the long run, or for all customers. Champniss has also conducted research into how we react to being targeted (often retargeted) using transactional data. The research shows that what he terms “hyper-personalisation” lessens brand attachment in many cases. Put simply: it freaks us out. 

His research shows that trust in a brand – not captured in transactional data – may determine how you respond. “Instinctively, there are certain brands in each category for the consumer where you would give them the benefit of the doubt and you’d be less likely to be creeped out. Many of those brands are proven to be trustworthy and reliable. Take Amazon. Fantastic customer service gives it extraordinary licence to use the data that I generate through transacting with it to make a recommendation.”

These limitations demonstrate that using transactional data alone is a partial solution to the big question for businesses: what do we do now? Despite the hype around data, surveys, trackers and other tools are still part of strategic decision-making. If we could integrate the two disciplines – as Nectar (see box, ‘Nectar’s one-eyed stick man’, p35 ) has done – there is potential to exploit the best of both. Which leads to another problem: insight generated from transactions, and insight generated from traditional research, often live in different locations and follow different paths through an organisation.

Emma Macdonald, research director of the Cranfield Customer Management Forum – a club of companies that meets regularly at Cranfield University – has studied this problem in detail. In a 2015 paper, ‘How organisations generate and use customer insight’, Macdonald and her co-authors researched four companies from different sectors that faced this problem. “Organisations are not structured for data to flow in the way it should. Organisations are often hierarchical and top-down when it comes to using data, and so information gets blocked by the hierarchy,” she says. “They need someone with a skill to bring that knowledge together.”

The consequences are familiar: research that is often not assimilated outside the insight function, or that is not useful as a decision-support tool because it is too detailed, too hard to understand, or simply not available; research that runs parallel to transactional data, but is never formally integrated with it.

The paper explains there is a need for “insight providers [who] can also beneficially look further along the demand chain to the research function’s internal customers, not just to support dissemination of insight but to align the value these users require with insight form, as well as insight content.”

Or, in Macdonald’s words, the internal or external research department’s job is “not providing reports, but being a partner”. 

The experience of organisations such as O2 and Sky show that this is possible. By integrating traditional research with large-scale data analysis, a hybrid discipline holds out the possibility of a mutually beneficial ‘sweet spot’. 

What’s the use of research?

Locating this sweet spot will, says Emes, require a change in emphasis for some agencies, away from monolithic projects to embedded, continuous feedback that links to operational performance. “If you set up a huge ‘use and attitude’ study across multiple markets – and it takes you 12 months from when it’s commissioned to when you do your final presentation – the people who asked the business question at the outset are on to something else. We need to pull together the best insights we possibly can, and make sure they’re relevant and timely.

“We are moving from a single source of data – a big project – to multiple data sources; moving from standard analytical techniques based on that one data set to more predictive modelling; from doing research into a big strategic decision to more immediate decisions, made quicker. We do fewer big presentations and more ongoing feedback to our clients. 

“This shift to incremental, rather than monumental, research is a trend that we’ll continue to see, and it’s fostering a closer partnership with our clients.”

Group CEO at Quadrangle, Nick Baker, explains how working with O2, the Inland Revenue, and others has changed the direction of the agency. “This is about connecting data, and connecting it to the business. There’s not a lot of people that will do this – and there’s certainly not very many people that do it very well,” he says.

To be relevant, agencies will need to invest in the skills to understand and model data at scale. The clock is ticking for researchers, Baker warns: “If you don’t start doing some of the right things now, you are going to get left behind, and it will take you a long time to catch up. You start three years late, you end up 10 years behind.”

So what might the right things be? There are many opportunities to mitigate the weaknesses of transactional data, and build projects that exploit the best of both. In this, market research has skills that are in short supply.

Audit data

Given that transactional data, acquired passively, is often biased, incomplete, ‘noisy’ or just hard to understand, the long-established skills of a market research company are more valuable than ever. 

“We are in a very good position to understand the data provenance – to be able to understand, fundamentally, how representative data is. We have all this stuff imbued in us from the day we start in our companies,” Strong says. 

“There are numerous researchers who both understand consumers and also understand where the numbers come from. We should be having a point of view on this.”

Emes adds: “As long as we’re aware of these systematic biases and skews, then we can adjust for them, be aware of them, or use other data and market research with representative samples to help us bring in some of the rigour.”

The ‘datafication’ of behaviour is an example of this. Using technology to automate life-logging, for web tracking and for following multi-device customer journeys, can radically improve the quality of transactional data by quantifying consumer behaviour in a larger sense. However, it needs to be manageable and rigorous.

The definition of useful data depends on what it is being used for. While much of the excitement around large data sets is how much we can integrate – transactions from sales, performance from operations, satisfaction from marketing – it would be rare to use the full data set for any effective research. So, in deciding what sliver of the data to take, we need a well-structured question or set of objectives. 

Champniss says that having lots of data can divert an organisation into setting easily measurable goals, which may have little  – or undefined – business meaning. 

“Market research still has an incredibly important part to play here; it is working out how any of this is ultimately going to add value for the consumer. That link is sometimes lost: we talk to the client and the client has the objective of 100,000 Facebook likes, for example. What does that mean? More to the point, how are you going to work out what it means?”

This can also affect segmentation analysis. As Rachel Kennedy points out (see box, ‘When significant isn’t significant’, p32 ), analysing a transactional data set can produce statistically significant segments that make little intuitive or operational sense to the rest of the business. Without knowing more about the customers’ motivations, the best analysis will not help managers to make better decisions.

Creating a win-win

Knowing more about the consumer means understanding what the consumer sees as valuable, and balancing that with operational effectiveness. As Champniss pointed out, this is different for every brand. Research can build in long-termism, resisting the temptation to play consumers like a pinball machine. 

For long-term success, Strong argues, companies need to demonstrate and articulate why consumers benefit from big data. Research can uncover attitudes to subjects such as privacy, but it also needs to explain the mutual benefit of cooperation. “If what we do is simply extract data from consumers to provide value to brands, which then extract more money from us, that’s clearly a dead end… Fundamentally, the industry needs to be able to demonstrate the value of what we are doing to the general public.” This helps avoid the creepy feelings that Champniss measured, but also helps to ensure long-term participation – for example, encouraging the use of loyalty cards.

Deploying high-value people

The skills to create the hybrid of transactional and research data are in short supply. As regular reports about the lack of data scientists imply, many organisations feel they are light in expertise. 

Emes argues that by investing in those skills as centres of expertise, agencies can deploy them during the project-design phase. Ipsos Mori has created a behavioural data centre based in the US, and a data science team in the UK, and uses those skills for clients when they are needed. This re-uses their experience and allows smaller projects to access the best expertise.

Strong believes that, in the long run, a powerful ‘sweet spot’ will be the ability to pool transactional data outside the boundaries of the organisations that generated it. By creating an environment in which the individual feels in charge of their data, he argues, that person will be willing to give trustworthy organisations access to it. 

By discovering and defining a socially useful outcome for a particular group (better health, better financial management) – and explaining and demonstrating the benefit – the long-term future is for the combination of research and big data. Baker, at Quadrangle, agrees: the liberalisation of access to automatically generated data will shift the emphasis from the generation and curation of private data to the creation of useful knowledge by combining open sources. The most effective analysts will be ‘data shepherds’ – finding out what’s available, and using their talent and knowledge to combine sources.

Making big data better

This is even a requirement inside the organisation, as Macdonald’s research shows. She uses the same language to describe how to create value from multiple sources of data. “Research agencies may need to take on a ‘shepherding’ role to help insight make its way around the organisation to overcome the obstacles of hierarchy and silos,” Macdonald says.

To discover the sweet spot, a final attribute that the research community may need to invest in is humility about the long-term value of some of its quant work. For all the flaws in transactional data, it trumps the quality of much survey data because it records what happened, rather than what people remember or believe happened. 

“Market research is awful at telling you what people actually do, but it’s very good at giving you answers – because people give you answers if you ask them questions,” Baker jokes. Instead, there’s the opportunity to enhance transactional data; this type of research can focus on what transactional data can’t do. 

An example, which Macdonald has used for research, is real-time experience tracking, a technique created by agency Mesh Experience so that consumers can report their emotion at every touchpoint, including media and peer-to-peer conversations. It finds that in-store positivity is much more influential than word of mouth or advertising, a result that is both beyond the scope of transactional data, and adds depth to it. 

“Emotional responses are difficult to capture through retrospective surveys and missing from behavioural data – yet we’ve found they are a significant predictor of the impact of a touchpoint on brand consideration,” Macdonald explains.

Whether they act as shepherds, integrators or co-creators, research agencies and departments that invest in the right skills have an opportunity to enhance transactional data, rather than be crowded out by it. “We’re all engaged in a big enterprise here. Let’s embark on really sharing some of the brilliant insights and understanding that we’re getting,” says Strong.

However, he warns, “it’s going to be a long sell.”