Data quality has been a significant challenge for market researchers in recent years. From survey fraud to poor respondent experiences and the looming threat (and promise) of AI and synthetic data, the insights industry faces a crucial period to tackle the data quality issue and demonstrate its importance to clients.

At a roundtable event in London last month with senior representatives from research agencies, Research Live was able to listen to some of the thoughts, ideas and concerns of senior people within the market research industry on the issue of data quality. This MRS data quality roundtable was co-hosted by sponsors Ayda and ReDem.

The event was held under the Chatham House rule, which means the discussion can be reported on but participants cannot be identified.

Data quality has become more of a pressing issue in the insights industry, leading to work over the past couple of years by the Global Data Quality Partnership, a coordinated effort to address risks to data quality in the market research, consumer insights and analytics industry. The initiative was launched in March 2023 by the Market Research Society (MRS), Esomar, the Insights Association and SampleCon, with The Research Society in Australia, the Canadian Research Insights Council and the Association of Market Research Austria (VMÖ) joining later as members.

An initial survey of roundtable attendees, carried out prior to the event, provided some indication of the scale of the data quality challenge. Most attendees thought that the number of interviews removed due to “insufficient quality” in their organisations was between 5% and 30%. The breakdown of that figure was that 30% of attendees believed 10% to 20% of interviews were removed, with 20% each opting for one of three categories: less than 5%, 5% to 10%, and 20% to 30% removed. The remaining 10% of attendees thought 40% to 50% of interviews were removed.

The survey also found that 80% of attendees thought the amount of bad data they had received in the past year had increased, with the rest stating that it had stayed about the same. Half of attendees thought 5% to 10% of interviews could be affected by fraud and should be removed, with 30% putting the number of interviews that needed removing at 30% to 40%. The remaining 20% were equally split between less than 5% of interviews and 20% to 30%.

In terms of how actively organisations are addressing AI-powered survey fraud, 40% said it was a key topic at their business and that robust measures were in place to tackle the issue. In addition, 40% of attendees said their organisation knew AI survey fraud was an issue and had started to implement new measures to tackle the problem, while 20% said that the organisation knew it was an issue but had yet to act.

Surveying the problem
Data quality challenges can be sorted into two camps: one being survey fraud related, and the second stemming from bad survey participant experiences. On the latter, attendees said that “there has been a reduction in incentive value” for completing market research surveys and more of a clamour from panel members for instant gratification. Attention spans are down, and panel companies need to be honest about what panellists get from taking part, the roundtable heard. Therefore, there is a need to decide “how to get money into the hands of participants as soon as possible”.

Attendees said many surveys are also too long, although it was pointed out that engagement is not necessarily tied to length of survey. But improving the participant experience was seen as vital, with attendees pointing out that many surveys have too many open-ended questions and can take too long to reach the content the participant initially signed up to complete. The industry has “got to be realistic about expectations” on participants, the roundtable heard.

In the age of AI, keeping data quality high was seen as a particular challenge – for example, are open-ended answers being generated by ChatGPT? Quality checks are struggling to pick up fraud, attendees suggested, as “fraud has evolved” with AI agents the “next big thing”. AI is exacerbating the data quality problem, attendees felt, alongside other issues such as a poor economy making research fraud a more attractive proposal and a high churn rates for panellists affecting panels’ viability.

AI could also be a solution, respondents said, as could synthetic data, and a means of identifying fraudsters more effectively, but attendees said that “market research needs to step up” and the industry should “take ownership of AI” within brands to boost insights’ role within organisations.

Grasping for solutions
Whose responsibility is it to get data quality sorted? Is it panel recruiters? The roundtable heard that panel users want large panels and low costs, which incentivises the recruitment of poor-quality respondents.

There is a danger that everyone collectively ignores the challenge, attendees said. Are data quality issues always brought to the attention of clients? Stakeholders are often working agile and iterative, the roundtable heard, and need data quickly to draw insights, but stakeholders also need to be shown the importance of investing properly in high-quality research and surveys to be persuaded away from low-cost and low-quality options.

The importance of data quality must be demonstrated, attendees said, with insight professionals within brands needing to own the conversation and manage messaging internally. Clients needed to know that “you get what you pay for” – the challenge is getting an equal equilibrium between price, speed and quality in surveys.

One solution put forward was that categories of quality for surveys – for example, bronze, silver and gold tiers representing the quality of the panel – could help make clients more aware of what they were buying. In that circumstance, it would then be up to the client to decide how much quality they were willing to pay for, and hopefully they would therefore be more aware of the drawbacks in choosing cheaper prices.

Is regulation the way forward? Would benchmarking data help? Could clients force change on the industry through procurement? There is work ongoing from MRS to get research fraud designated as a specific crime, which attendees agreed should act as a better deterrent to prevent individuals from taking part. This policy change could also help to force social media companies to close survey fraud groups and remove posts that encourage illegal behaviour. Roundtable participants also praised the work of the Global Data Quality Partnership, with the initiative working to improve standards in the industry.

Other solutions were suggested. For example, could a common way of verifying identity online be introduced? Estonia, for example, has a digital ID for citizens, but such a system would be hard to implement in numerous nations. Could the market research industry agree on common standards, similar to an ISO? Australia has had some success in this, with the government having pushed companies to address data leaks and raise standards.

Measuring the impact of research fraud on GDP, for example, could also help to focus minds in government and other bodies to tackle the problem, as well as persuade clients of the need to invest in better data quality. “Buyers will always drive the market. How do you demonstrate the value of not doing anything?”

Inevitably, with a problem as complicated as data quality, there is no single solution, the roundtable heard. But there are some options – for example, key stroke analysis – that could make a positive impact. Could large language models help to analyse responses, and weed out those that are fraudulent? Could statistical rules help? The reality is that many different measures are needed simultaneously to address data quality issues, with no “golden bullet” available to solve the problem overnight. For attendees, the game is getting “marginal improvements” in place.  

• In 2024, MRS issued new guidance on data quality and integrity in partnership with Austrian research association VMÖ.