On continuous integration and delivery

One of the early adopters and more vocal organisational proponents of Continuous Integration, ThoughtWorks, defines CI as:

They go on to define Continuous Delivery:

Even given these definitions too understand how to operate CI/CD now it’s first worth understanding how CI/CD came to be.

Though the iterative model of software development can be traced back to the 1950s^[3] it was previously more common for software development to go through distinct phases, not unlike a construction or other major project:

This is referred to as the "waterfall" pattern and it makes sense, presuming:

Unfortunately neither are generally true of software project and the approach quickly ran into issues. The issue most prescient to this article is called "integration pain", and denotes the period of time in which developer work is reconciled and tested together before release. Martin Fowler describes the story:

In addition to this integration pain the challenges associated with shifting requirements and shifting consumer expectations mean that modern software projects are rarely "done", and instead burdened with attempting to meet a constantly shifting target. Software released in the "waterfall" pattern is difficult to maintain in a reasonable way following its release and bugs that are discovered later in the process are up to 15 times more expensive to fix than those discovered earlier^[5].

As a mechanism to address these and other issues with the SDLC Kent Beck developed the Extreme Programming (XP) approach, the precursor to the "Agile" software development approach^[6]. Among other things XP endorsed the use of continuous integration as a mechanism to ensure that integration pain was much reduced, bugs were caught early and the software could be adjusted to new requirements. Kent used this approach with various projects including the "Chrysler Comprehensive Compensation" project with Martin Fowler^[4], who would later go on to publish and popularize this approach.

While not a requirement for continuous integration^[4], CI servers became a popular method of implementing this approach. ThoughtWorks were among the first with their "Cruise Control" serer, released in 2001^[7]. Later Sun Microsystems developed "Hudson", released in 2015^[8], the precursor to the infinitely popular and still widely used Jenkins CI. Jenkins was developed as a result of disputes with Suns purchasers Oracle^[8] and released in 2011^[9].

Jenkins later introduced the concept of "Workflows", which were builds that could be checked in to version control instead of configured in a GUI^[10] and marking the shift to CI systems entirely driven by VCS.

It is difficult to map when containers started being used as environments in which to run CI jobs as the history of containers goes far further back than one might imagine^[11] however the earliest development of the Docker plugin for Jenkins goes back to 2014^[12]. The rise of the popularity of Docker and its excellent ephemeral environment tooling mean that many modern CI systems consume Docker containers as their only build environment^{[13],[14],[15]}.

This brings us to the current era where the majority of CI/CD tools are built on top of:

In this case this post will be talking primarily about the Drone build server which is an excellent open source implementation of this technology.

While the historical context of how CI/CD was implemented is super useful for providing understanding of the pressures that generated CI understanding how it fits into the software development life cycle (SDLC) first requires an understanding of how the SDLC is currently architected. One of the most common patterns of development is called the 'feature branch' workflow^[16].

The work begins with the description of requirements in a ticket, usually in the form of a "user story" or "bug report". These are tickets that either a new application behaviour or some circumstance in which the application does not behave as expected. I would then review the ticket, asking questions for clarification or to ensure that the feature is being used as expected before adding it to the work queue.

Implementation starts by updating the local copy of version control. The canonical branch in all version control systems I use is called master and is the branch that is either deployed on production systems directly or used as a base to create a "release branch" to be deployed. It is the version of the code on which all future work should be based:

Immediately after ensuring my local branch is up to date I switch to a new "feature" branch. This branch is temporary and allows me to make changes to the codebase in a way that is not going to affect other people:

From here, I do development:

Once development has been completed and tests have been written I commit the changes and push the branch up to the origin to save my work

I will then create a pull request^[17]:

And we see the first CI checks. In the image above we can see "Some checks have not completed yet", with a note that Drone is running a "PR" job. It is here that CI is responsible for enforcing code safety checks such as unit, integration and smoke tests:

Following the successful completion of these tests the PR is sent to a colleague for their review. If that colleague is happy with the changes they will merge the work into the mainline.

Once the work is merged in to the master branch, CI is responsible for deploying the work to a production system:

Around 3 minutes after I (or my colleague) has merged code in the work is in production and facing users.

That’s it!

Given the above development workflow we can see our CI/CD system has several different responsibilities, split into two areas of focus:

This work is usually broken up into a staged process. The following is an example of the "Pull request" process:

These sets of processes are usually referred to as a "Pipeline", named after the computing definition:

There are several different components required to create such a CI/CD pipeline^{[19],[20],[21]}:

These tools are not specific to CI/CD but are rather general tools that are designed to track, ensure correctness and deploy the software. Accordingly they should be set up to be executed both in and outside the CI/CD environment with the environment itself being as "dumb" as possible. This can be done by putting the logic to run these tests behind a task runner and invoking that task runner both in the development and the CI/CD environment:

Practically speaking this means instead of configuring pipelines with large sets of commands:

The pipeline should be invoked only with a single command with some argument indicating which task to run:

This separation of responsibilities into the task runner, deployment manager and CI pipeline allows the reuse and debugging of the majority of the pipeline locally, switching between CI/CD services without undue cost and makes the pipelines as simple and predictable as possible.

As discussed, the pipeline consists of a set of steps that need to be executed prior to verifying the codebase is still correct or deploying the software to some environment. Further, these steps should be executable both on the local machine and in the build pipeline in exactly the same way. The way to implement this is via a task runner, such as Robo:

We then add our tasks in our task runners domain specific language; in this case, by writing a public function in PHP:

That allows us to test the task locally:

And adjust the check (or the code) until it works as expected:

The task should work successfully before being committed:

After which the step can be committed to version control and is ready to be consumed in the CI/CD pipeline. Steps can be written for any number of tasks:

However, they should all work and be useful locally before they’re consumed in the pipeline.

Continuous integration & delivery has become a fairly essential part of software development. It augments the software delivery life cycle with additional correctness checking to find bugs earlier and dramatically reduces the cost and risk associated with pushing changes to production. There are many CI/CD servers however Drone implements all required features for the vast majority of software projects and is open source and trivially installable. With Drone we constructed a multi-step build cycle stubbing out lints, tests and other correctness checks common software, as well as stubbed out a workflow for continuous delivery.

That’s it! This post got a bit long, and I did not get to cover all of the things I would like to. However if you made it this far and you enjoyed it be sure to voice yourself on GitHub, and I’ll keep writing this stuff up.

<3