Supercrunch Blog

Dr. Ralph Wirth & Norbert Wirth April 13, 2017 Big Data

Big Data – Broken Promise? Our learnings on how to avoid Big Data frustration

Part 1: Introduction

This is the first post in a series where we’ll share our experience in how to avoid common mistakes and thus frustration in Big Data projects. We won’t get too technical, so even if you’re not a Data Scientist and have just been tasked to manage this super important new Data Lake project that your Management Board has signed off – be fearless, read on – you’ll probably find some of this really helpful.

Despite all the hype, there is no doubt – we really live in the “Big” Data era! The amount of data generated by connected consumers and devices grows exponentially. Retailers and manufacturers gather huge amounts of transactional and CRM data along the whole purchase journey. Market Research, data brokers and other players with commercially relevant data assets are celebrating the new Data Economy, and it seems like companies across virtually all industries should embrace the opportunities this data provides. Looking at the area of analytics and data-driven decision support in Marketing, high hopes have been linked to the “new oil”: Data. Initial steps have been taken to fully automate certain marketing processes (e.g., in programmatic advertising) utilizing Big Data and the respective technologies.

However, we have also seen many companies struggle. Huge investments have been made to create Data Lakes, trying to integrate “all” data that conveniently slumbered in silos before, well protected by its so called owners. Dedicated Data Science and IT teams have been set up. Numerous Big Data conferences have been visited, papers on tools with unpronounceable names have been read – just to end up with the feeling that everyone else seems to be far ahead. And in the end, in spite of all these efforts, the critical voices are getting louder. First stakeholders start questioning the value of the Data Lake, which by the way has become a data dump in the meantime. The Data Scientists get frustrated because none of their fancy ideas gets ever implemented. And even the true innovators can’t avoid thinking: “So what?”

Throughout the upcoming series of blog posts, we will publish our learnings and experiences on how to address the main challenges when approaching big data projects, or trying to become a data-driven company. We will focus on:

  • Use case first: The key foundation for data-driven companies
  • Data meets use case – how to analyze which use cases to focus on
  • Elephants and human experts: Infrastructure, tools & data governance
  • Don’t lose focus: Why Big Data requires even more critical thinking

We are looking forward to discussing our findings with you!