We are in the midst of a digital revolution that fundamentally changes the world around us – and with it our research industry. We are experiencing an exponential growth of data and new technologies.  And a global community of developers is sharing new models and algorithms through open source projects, giving us access to cutting-edge technology at no cost.

This brave new world requires more than the traditional ‘marketing science’ skillset of analysing and modelling neatly-structured consumer data. We regularly use a handful of software packages and have, over time, picked up some basics of coding. Sometimes we bring together data from different sources, or even use statistical fusion to create integrated databases. But we need much more than this to harness and use all the new data and open source technologies now available to us.

Data scientists – rock stars of the information age

This is why data scientists – a hybrid species made of software developers, statisticians and data engineers – are highly sought-after professionals. They know how to use technologies to process large volumes of unstructured data. They can analyse and model this data using the latest open source algorithms and cutting-edge models. And they can build end-to-end applications that deliver analytical outputs programmatically in form of data feeds through APIs or web applications. 

Good data scientists combine these technical skills with domain knowledge and communication skills. They have a hacker mind-set, i.e. they are curious about data and are keen to identify and solve problems – but, crucially, are always driven by the business application. And they can explain complex data and models to decision makers using intuitive visualisations.

The demand for professionals with this profile is rocketing.  In Forbes’s recent list of the most promising jobs in 2016, data scientist came first.  McKinsey projects that by 2018, there will be a 50% gap between demand for and supply of data scientists, and that this shortage of talent will soon become one of the key challenges for our economies. No wonder so many statisticians, marketing scientists and research analysts now claim to be a data scientist.

Do I need to hire data scientists? Should I become one myself?

In order to keep up with technological progress and the growing competition from outside the industry, MR companies need to get these new skills on board, via recruitment or by encouraging existing staff members to learn and adopt. Either way, it is important to keep the business application in mind. It is unlikely that you need someone to implement a full-blown Big Data infrastructure in your business. After all, we are MR companies, not online retailers or car insurers.

A typical Big Data application will predict the likely behaviour or preferences of every single customer, based on past transactions. This requires processing and scoring of large volumes of data in real time. But market research is fundamentally about creating insights that help us see the bigger picture, about understanding the principal mechanisms rather than predicting here and now what a specific customer is likely to buy.

And if we needed to use customer records to create these insights, then who would be better placed than us to do this based on samples from the Big Data pool? What we, as an industry, need are people with strong statistical and mathematical skills, who actively follow new developments in the fields of data and analytics, who enjoy coding and are excited about trying out new systems and approaches.

So, as research professionals in analytics roles, should we all become data scientists?  If the hype is right, we have never had greater career opportunities. But beyond this, I would argue that anyone working in analytics should be curious about all things related to crunching data, and should learn how to use relevant technologies and programming languages. After all, we are analysts because we are passionate about data analysis, and this passion should fuel our desire to continually develop and to explore the world of data science.

Frank Hedler is director, advance analytics at Simpson Carpenter