The pressure on big data projects to deliver is huge. On the one hand, there is a high demand for data analytics tools for better decision making. On the other hand, projects often struggle with technical issues when implementing and operating the large variety of software and hardware in the big data stack. As a result the time to value is often too slow.
In most cases there is too much emphasis on the technology alone. Therefore it takes time to integrate and run the relatively flat operating environment that most companies have today. Different technologies must be considered for specific purposes and the various layers of the big data stack; for example data warehousing and distribution, security, storage, processing, analytics and visualization. In software engineering there is a high level of uncertainty when deciding what the best practices, tools, abstractions and systems to implement are.
Furthermore, the big data stack evolves rapidly and innovation can easily lead to project delay. Firstly, because shifting to a new framework requires thousands of programming hours to re-write the existing code. Secondly, programmers need additional time to learn the new technology before being able to apply and maintain it. Thirdly, new technologies extend the overall time for implementation, as innovation often comes hand-in-hand with increased complexity. For example, cloud technology challenges IT engineers about how to seamlessly connect data in the cloud with on-premise systems.
So how can businesses overcome these challenges and make their big data analytics project a success?
By shifting the focus from technology to adding business value. But how? Open-source data analytics tools can help by fostering fast and effective project delivery. They make analyzing the growing volumes of data manageable; no matter if the specific use case requires batch or stream data processing. Furthermore, processes which used to take days can be executed in hours or even in real-time, thus increasing speed and efficiency.
When the deployment stage is reached, DevOps tools for automation deployment increase IT agility, by eliminating manual work and making software deployments faster and more reliable. Businesses can apply a continuous delivery of working software to ensure software increments can be developed, validated and improved at a much faster pace.
How are you dealing with the reality of tight project resources and dynamic digital change? Have you been in the situation of spending months developing a new application without delivering a shippable product? Do you think that the business problem you are trying to solve is still relevant? Or while addressing technology issues the use case was neglected.
Share your experiences with us.