Concourse was dreamt up and built while working on the Cloud Foundry project and experiencing the interesting CI and CD challenges that it has. Cloud Foundry projects and their tests needed to run on different architectures, different platforms, against varying underlying IaaSs before being tested that they could be deployed fresh while also remaining compatible with upgrades from previous versions. Finally, these projects need to be packaged up in various ways to create multiple derivatives end products.
We went through various different generations of our CI infrastructure and were never completely happy with one (or in some cases many) aspects of each attempt.
We started with Jenkins: the big and ubiquitous player in the CI world. Each team would maintain their own deployment and add builds as required. Standard.
After a while cracks began to show.
Jenkins servers become snowflakes. More and more important shell script gets pasted into those little textboxes, plugins get installed, and dependencies are installed on the slaves. It's possible to try and store this state but Jenkins servers are extremely resistant to being declaratively configured.
Concourse does not allow any configuration to enter the server that cannot be easily saved away in version control. Deleting your Concourse server and bringing it back somewhere else is not and should not be a big deal. Builds run every step in their own containers, with their own dependencies, so workers never need to be configured in the first place.
Jenkins has no first class support for pipelines. There are plugins that add some of this functionality but they never seemed to work perfectly. You can set jobs to run before or after others but this becomes error prone and confusing to perform more complex sequence changes.
Jenkins 2.0 tries to address this by promoting a Pipeline plugin (plus another plugin to visualize it), but it kind of misses the point. It, like the plugins before it, allows you to define a flow of scripts and plugins to execute, but the real heart of a pipeline in the Concourse world is resources, not the fact that it can run jobs one after another.
Concourse is built on the assumption that you will be building pipelines. However, instead of making you explicitly say the order in which jobs should be run, Concourse just asks that you say which are valid inputs for each of the jobs in your pipeline and uses this to calculate the pipeline itself.
Everything you care about in Jenkins is accomplished by plugins. Concourse aims to never have a plugin architecture, and instead determine the generic features and user experience patterns that allow it to remain nimble, stable, and consistent.
Rather than scouring the earth for a still-maintained plugin to accomplish your goals, Concourse encourages modeling what you need in terms of the few concepts that Concourse requires you to understand.
Trying to find the build output log for a failed build in Jenkins can be up to 3 clicks from the home page.
The Concourse UI is built and designed around making that sort of information as quick to get to as possible. Getting to the most recent build log is a single click.
We began using Travis CI in order to open up our builds and their results to the open source community. Travis gets a lot of things right. They don't let you have any snowflake configuration outside of your
.travis.yml file. Builds are run inside a container to avoid pollution. It also has great support for building your GitHub pull requests. This is something that Concourse cannot yet do but we have plans to allow the same workflow in the future.
Unfortunately it still doesn't have support for pipelines and only very simple builds are possible. It also brings the frustration that if something doesn't pass in CI you normally need to send up lots of little debugging commits to work out why it's behaving differently. Concourse lets you submit your local working directory up to the server to be run in exactly the same manner as CI.
We were excited for the release of GoCD. Finally, someone was bringing pipelines as a first class citizen to CI! CF decided to switch and unify the disparate CI approaches to form a unified pipeline.
Unfortunately there were some problems with some of the GoCD design decisions that, in our opinion, make it very difficult to use.
Finding how to configure something is very difficult. There are two distinct sides of the application: watching jobs run and configuring them. You cannot be viewing a job run and easily get to the configuration of the same job. Everything is hidden in endless menus.
As mentioned above, Concourse optimises for getting to the information you want quickly and keeps all configuration in a separate version-able file.
The execution hierarchy is deep and complex. Pipelines have stages which have jobs which have tasks. Some of these run in parallel and some of them run in serial. Artifacts from some are available in others. These concepts make sense eventually but ramping new developers up on how to use it was very difficult.
What may be multiple pipelines in GoCD often conceptually maps to a single pipeline in Concourse. A Concourse pipeline just has isolated jobs that you can look at without needing to understand the entire pipeline. The flow of resources inside jobs in Concourse is explicit and user-defined. There are no surprises about when or where you'll have a resource available.
Another frustration with GoCD was that - by the time it was released publicly - it didn't bring anything new to the world of CI (first-class pipelines non-withstanding (which are great)).
If you'll indulge us tooting our own horn a second: we feel that Concourse brings some interesting new ideas to the table:
Pluggable Resource Interface
Running builds in Containers Natively
Zero Snowflake-able Configuration
Submitting builds from the local file system up to run in CI