Supercrunch Blog

Kiran Kumar November 20, 2017 DevOps, Software engineering

Crawl… walk… running your application with Continuous Delivery Pipeline

Did you know that from the time you login to the Amazon website until the time you complete your order, tens of patches are deployed to their website without any disruption to service? Top Internet companies like Amazon, Google, Netflix and Facebook have embraced continuous deployment practices; whereby incremental software updates are moved into production automatically with tens, hundreds or even thousands of deployments per day. However getting a Continuous Delivery Pipeline setup for your product is not a straight forward task – it involves a lot of decision making at different levels and implementation with continuous feedback, until you get a pipeline that works for you. Moreover, the practice is interpreted and implemented with great diversity across organizations and there is no one “right way” of running the delivery pipeline. Here at SUPERCRUNCH by GfK we’re also embracing Continuous Delivery Pipeline practices and principles, which I’ll share with you. But first more about Continuous Delivery Pipeline.

What is Continuous Delivery Pipeline?

Over the last decade, with a paradigm shift away from traditional software development practices to a more agile development approach, new jargon such as Continuous Integration (CI), Continuous Delivery (CD), DevOps etc. started popping up. As more and more development teams got structured around GIT principles, adaptation of continuous integration/continuous delivery pipeline became an intrinsic part of their Application Lifecycle Management (ALM) workflow. In simple terms continuous delivery can be explained as applying automation to ALM.

Just to explain, let’s imagine that you and your team is responsible for making a movie by continuously adding scenes to it. Each member of the team is assigned a part of the storyline to shoot and direct.  Once the movie frames are ready, it is added to the main storyline and a check is made if the whole thing makes sense. Every time a new frame is added, this process is repeated. Finally once all the frames are ready and the story line is in order, the movie is released to the audience. Similarly, in a software development teams each developer works on different features. Every time a new feature is completed, it is added to the existing code and tested if the whole application works as expected. In this process, instead of building the features separately and putting them together at the end, the application is in a constant state of expansion with continuous builds and integration. Moreover, following such a process results in immediate feedback if the new piece of code didn’t fit and thereby minimizing errors in the application.

In a nutshell, CI is a software development process, where developers integrate their code early and often, to reduce the workload and errors involved with larger, independent integrations. Extending the process to the point where a constant flow of changes to the software passes all tests and is deployed to production in a completely automated fashion can be termed as Continuous Delivery Pipeline.

What does Continuous Delivery Pipeline involve?

Continuous Delivery Pipeline reacts to any code changes and provides feedback. An automatic build and verification process takes place for each code change as part of CI. Furthermore an automated step is added for software delivery. Then, the verified software is prepared for deployment to an actual production environment, to instantly provide value to end users. There is nothing like a “standard” Continuous Delivery Pipeline. However, standard industry practices have led to the belief that a typical CD pipeline must involve build automation, continuous integration, regression and test automation, deployment automation, code quality report and a feedback system.

 

The figure depicts the various stages of a typical Continuous Delivery Pipeline. The double sided arrows illustrate the feedback mechanism at each stage of the pipeline.

 

Build Automation

The pipeline starts by reacting to any code change made to the repository and building the executables/deliverables. Usually build automation tools also handle dependency management.  The build automation provides feedback on any compilation errors.

Continuous Integration

Any new feature implemented is integrated into a single source code repository, built, unit and integration tested. This is the most direct feedback cycle that informs the development team about the health of their application code. This is followed by a stage of acceptance testing performed on software verifying functionality, security, performance and compliance. This stage is usually a hybrid of automated and manual activities.

Continuous Delivery

Output of the Continuous Delivery will be a new verified release version/ deliverable of the software that brings some added value to the end-user.

Continuous Deployment

This stage involves automated deployment of a new software version to production. It can be argued that deploying each and every version code change may not be desirable to all projects. However, every change should result in a potentially higher quality deployment candidate.

 

What does Continuous Delivery Pipeline bring to the table?

To quote Martin Fowler –“Continuous Delivery doesn’t get rid of bugs, but it does make them dramatically easier to find and remove”. The benefits of adopting CI/CD pipeline are multifold. Debugging errors are much easier, as errors can be associated with small increments in code. Integration risks at the final phase of the project are minimized, as the state of the software is verified constantly. Continuous Delivery Pipeline is a continuous feedback mechanism; thereby averting any major software failures. Build and test automations can make testers’ lives easier by contributing to increased quality and productivity. Moreover, it also helps to isolate and track bugs with different versions and builds. CD facilitates the automated deployment of new versions, thereby reducing waiting times for updates/upgrades. Overall it increases the confidence of the team by creating a foolproof environment which eliminates the risk of easily breaking software.

What are the challenges?

As flashy and sexy as it sounds, adopting Continuous Delivery Pipeline as a practice and process to a team is neither easy nor free of cost. Test automation is a crucial stage of CD pipeline. However, implementing an acceptable quality of with automation testing requires considerable efforts. Most cases involve a bit of manual testing, which may slow down the verification process for every increment. Communication becomes an essential part of the process where the build results feedback have to be communicated to the developers to take responsibility.

Often implementing CD pipeline takes long time with a lot of iterations and amendments to the process. Obtaining management support for CD pipeline adoption and convincing them about the long term perks becomes a challenge. “Old habits die hard” – there is hesitation among developers to put their code to scrutiny in its unfinished state.

Adoption of CD may involve software architectural changes. Choosing the right tools for building pipeline is always up for argument. With a lot of tools available in the market, making a strategic decision to get the right set of tools to enable the process requires some amount of maturity and understanding. These are just a few but the list goes on and may vary with respect to specific cases.

The SUPERCRUNCH by GfK mantra for Continuous Delivery Pipeline

Continuous Delivery Pipeline is so much more than just implementing a process and enabling tools to assist the process. The pipeline should be strongly backed by practices and principles within the team. Here are the practices and principles we follow at SUPERCRUNCH by GfK:

  • Every developer is also a DevOps specialist – if you write a code, you should also think about running it
  • If you do the same thing twice, automate it!
  • Maintain a single source repository and leverage the power of GitFlow for parallel feature development, release management and better collaboration within the team
  • Make each build self-testing. Implement tests as part of development and automate it with a build process
  • Have different types of environments for integration, staging, testing, production, production-clone etc.
  • Test in multiple environments before moving to production
  • Semantic versioning is your friend
  • Integrate code quality and code coverage reports as part of the pipeline
  • Ensure everyone can see what is happening and keep a close eye on the software health with every increment
  • Automate deployment
  • Communication is key – leverage notification features of the tools you’re working with to get constant feedback at each step of the pipeline. Use multiple channels for notification – email, slack etc.
  • Frequent code check-ins accompanied with pull requests
  • Practice small incremental code merges to avoid failure with huge merges
  • React to notifications from the CI system as soon as possible!
  • Perform a smoke test on an automated deployment

 

Just like the Continuous Delivery Pipeline is continuously evolving, we as a team are also on the learning path and are continuously improving our process by strictly following our practices and principles. Adopting CI and CD as a part of our product development process has not only reduced risks and bugs, but has also helped us to move rapidly towards a working software. With the help of a continuously improving Continuous Delivery Pipeline, we can quickly adapt to business requirements and client needs; turning our release process into a tactical advantage.