Clusters with BOSH

Deploying Concourse with BOSH provides a scalable cluster with health management and rolling upgrades.

If you're not yet familiar with BOSH, learning it will be a bit of an investment, but it should pay off in spades. There are a lot of parallels between the philosophy of BOSH and Concourse.

Learning BOSH

If you've never used BOSH before, you may want to first go through its introductory material:

To tie the terminology together: Concourse is packaged as a release, whose source repository lives at concourse/concourse on GitHub.

Releases and stemcells (operating system images) are put together to form a deployment manifest (a YAML file, similar to a Concourse pipeline, which describes what jobs to run where, and with what properties).

A BOSH director is given a cloud config, which is another YAML file describing IaaS-specific things like network names and VM types. This file is changed a lot less often than your deployment manifests.

The BOSH director is then used to deploy the deployment manifest, which is what brings the "abstract" (releases, deployment manifests, a bunch of YAML) to the "concrete" (software running on VMs running on an IaaS of your choice). The director is then used to maintain the deployment, allowing you to e.g. SSH into VMs, re-create them, perform a rolling upgrade, manage persistent state, recover from IaaS instability, etc. etc.

Deploying Concourse with BOSH

The quickest and easiest way to get going is to use our concourse-deployment "cluster" recipes. This repository contains a base manifest, some example cloud configs, and useful Operations files for deploying and maintaining the latest version of Concourse with BOSH. In the future, it will also include useful deployments to provide things like credential management and metrics.

The concourse-deployment repo covers the following steps:

  • Setting up your cloud config: there's a cloud_configs directory containing various examples. Some of them may work verbatim. If none of them apply, though, you're on your own; you should consult bosh.io for IaaS-specific information.

  • Creating the manifest: that's kind of the point of the repo. Any required property changes will be made to the repository as soon as they're needed, so when a new Concourse comes out you should be able to just re-deploy.

  • Uploading the releases: the manifest will include them by URL, version, and sha1, allowing the director to download them for you.

  • Configuring Configuring Auth and other fancier setups should be available as Operations files under the operations/ tree.

However, it does not cover the following:

These steps are general to anyone using BOSH, and so are not replicated in our documentation.

Another handy reference is the release documentation on bosh.io, which makesit easy to peruse the available properties for each job. We intentionally keep all documentation and examples within the release itself to ensure there is no out-of-date documentation.

You may also want to consult Configuring Auth for configuring something other than basic auth for the main team.

Reaching the web UI

This really depends on your infrastructure. If you're deploying to AWS you may want to configure the web VM type to register with an ELB, mapping port 80 to 8080, 443 to 4443 (if you've configured TLS), and 2222 to 2222.

Otherwise you may want to configure static_ips for the web instance group and just reach the web UI directly.

Upgrading & maintaining Concourse

With BOSH, the deployment manifest is the source of truth. This is very similar to Concourse's own philosophy, where all pipeline configuration is defined in a single declarative document.

So, to add more workers or web nodes, just change the instances value for the instance group and re-run bosh deploy.

To upgrade, just upload the new releases and re-run bosh deploy.

Supporting external workers

If you need workers that run outside of your BOSH managed deployment (e.g. for testing with iOS or in some special network), you'll need to make some tweaks to the default configuration of the tsa job.

The TSA is the entryway for workers to join the cluster. For every new worker key pair, the TSA will be told to authorize its public key, and the workers must also know the TSA's public key ahead of time, so they know who they're connecting to.

Authorizing worker keys

Workers are authorized into the cluster by listing their public key in the authorized_keys property on the tsa job, and configuring the worker's tsa.worker_key property on the groundcrew job to register with the corresponding private key.

To do so, set the following properties:

authorized_keys on the tsa job

the array of public keys to authorize

tsa.worker_key on the groundcrew job

the private key for the worker to use when accessing the TSA

tsa.host and tsa.host_public_key on the groundcrew job

if the worker is in a separate deployment, these must be configured to reach the TSA

garden.forward_address on the groundcrew job

if the worker is in a separate deployment, this must be the locally-reachable Garden address to forward through the TSA; e.g.: 127.0.0.1:7777

baggageclaim.forward_address on the groundcrew job

if the worker is in a separate deployment, this must be the locally-reachable Baggageclaim address to forward through the TSA; e.g.: 127.0.0.1:7788

After setting these properties run bosh deploy to make the changes take place.

Making the TSA reachable

Typically the TSA and ATC will both be colocated in the same instance group. This way a single load balancer can be used with the following scheme:

  • expose port 443 to 8080 (ATC's HTTP port) via SSL

  • expose port 2222 to 2222 (TSA's SSH port) via TCP

Be sure to update any relevant security group rules (or equivalent in non-AWS environments) to permit both access from the outside world to port 2222 on your load balancer, and access from the load balancer to port 2222 on your TSA + ATC instances.

The BOSH deployment manifest would then colocate both jobs together, like so:

instance_groups:
- name: web
  vm_type: web_lb
  jobs:
  - name: atc
    release: concourse
    # ...
  - name: tsa
    release: concourse
    # ...

In AWS, the web_lb VM type would then configure cloud_properties.elbs to auto-register instances of web with an ELB. See the AWS CPI docs for more information.