Flight School

Welcome, Cadet.

This walkthrough assumes you've already got a Concourse set up and ready to go. If you don't, the most important thing is that you don't panic and go through the installation guide before coming back here.

Getting Set Up

First, let's get you a copy of your training materials.

Fork this repository on GitHub into your own account. This will give you somewhere to keep your work.

Next, clone your fork down to your local machine by running this command:

$ git clone https://github.com/(your username)/flight-school

This is a simple Ruby project that will let you get a feel for flying. Let's check everything is working by running the tests locally.

$ cd flight-school
$ bundle install
$ bundle exec rspec

If everything is working then you should see a few green dots on the screen to let you know that the tests are passing. If something failed then make sure to fix it and get the tests green before continuing. For example, if you're missing the bundle executable then you'll need to run gem install bundler.

First Steps

Ok, we've got a project. We want a pipeline. It's important to build a pipeline up gradually rather than attempting to do the whole thing at once. Let's start at the start by running those unit tests we just ran in Concourse.

Download fly from your Concourse. You can find a download link on your Concourse installation main page. They will either be at the bottom right or in the middle if you don't have any pipelines yet.

$ mkdir -p $HOME/bin
$ install $HOME/Downloads/fly $HOME/bin

Make sure that $HOME/bin is in your path.

You should now be able to run fly --help to see if everything is working.

Ok, let's target and log in to our Concourse.

$ fly -t ci login -c (your concourse URL)

The -t flag is the name we'll use to refer to this instance in the future. The -c flag is the concourse URL that we'd like to target.

Depending on the authentication setup of your Concourse it may prompt you for various credentials to prove you are who you say you are.

Right, let's try running the current project in Concourse.

$ fly -t ci execute
error: the required flag '-c, --config' was not specified

Huh. Well then, let's give it that flag with a file that it wants.

$ fly -t ci execute -c build.yml
error: invalid argument for flag '-c, --config' (expected flaghelpers.PathFlag): path 'build.yml' does not exist

Alright, so we need a file called build.yml. Let's create it under the flight-school directory.

$ touch build.yml
$ fly -t ci execute -c build.yml
error: invalid task configuration:
  missing 'platform'
  missing path to executable to run

Surprise, surprise - looks like we can't just give it an empty file. We need to write a task definition. A task definition describes a unit of work to Concourse so that it can execute it.

platform: linux

image_resource:
  type: docker-image
  source: {repository: busybox}

inputs:
- name: flight-school

run:
  path: ./flight-school/ci/test.sh

Let's go through this line by line:

  • The platform simply states that we would like this task to run on a Linux worker.

  • The image_resource section declares the image to use for the task's container. It is defined as a resource configuration.

  • The inputs section defines a set of things that we need in order for our task to run. In this case, we need the flight-school source code in order to run the tests on it.

  • The final run section describes how Concourse should run the task. By default Concourse will run your script with a current working directory containing all of your inputs as subdirectories.

Ok, let's save that in build.yml and run our command again.

$ fly -t ci execute -c build.yml
executing build 107401
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 40960    0 40960    0     0   662k      0 --:--:-- --:--:-- --:--:--  800k
initializing
running flight-school/ci/test.sh
exec failed: exec: "./flight-school/ci/test.sh": stat ./flight-school/ci/test.sh: no such file or directory
failed

Ok, so what happened here. We started a build, uploaded the flight-school input, and then tried to run the flight-school/ci/test.sh script. Which isn't there. Oops! Let's write that.

#!/bin/bash

set -e -x

pushd flight-school
  bundle install
  bundle exec rspec
popd

This is basically the same commands we ran above in order to run the tests locally. The new bits at the start set up a few things. The #!/bin/bash is a shebang line that tells the operating system that when we execute this file we should run is using the /bin/bash interpreter. The set -e -x line is setting a few bash options. Namely, -e make it so the entire script fails if a single command fails (which is generally desirable in CI). By default, a script will keep executing if something fails. The -x means that each command should be printed as it's run (also desirable in CI).

Let's give this new script a whirl.

$ fly -t ci execute -c build.yml
executing build 107401
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 40960    0 40960    0     0   662k      0 --:--:-- --:--:-- --:--:--  800k
initializing
running flight-school/ci/test.sh
exec failed: exec: "./flight-school/ci/test.sh": permission denied
failed

This error message means that the script we told it to run is not executable. Let's fix that.

$ chmod +x ci/test.sh

Running again gives us:

$ fly -t ci execute -c build.yml
executing build 107401
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 40960    0 40960    0     0   662k      0 --:--:-- --:--:-- --:--:--  800k
initializing
running flight-school/ci/test.sh
exec failed: no such file or directory
failed

This message is a little obscure. It's complaining that the shebang (/bin/bash) can't find the interpreter. In our task config, we specified the busybox Docker image, which is a tiny, un-opinionated operating system image that doesn't contain bash. This isn't very useful for running builds so let's pick one that is.

Docker maintains a collection of Docker images for common languages. Let's use the ruby image at version 2.4.1. We can specify that the task should run with this image by updating the image_resource block in our build.yml like so:

image_resource:
  type: docker-image
  source:
    repository: ruby
    tag: 2.4.1

Let's try running that.

$ fly -t ci execute -c build.yml
executing build 107418
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 40960    0 40960    0     0   7657      0 --:--:--  0:00:05 --:--:--  9884
initializing
Pulling ruby@sha256:f7cb2fda8271b272f9adb5f396937c09499113a2e5299e871efb04195dabfc96...
sha256:f7cb2fda8271b272f9adb5f396937c09499113a2e5299e871efb04195dabfc96: Pulling from library/ruby
10a267c67f42: Pulling fs layer
[... docker output ...]
b8b6e6204a6b: Pull complete
Digest: sha256:f7cb2fda8271b272f9adb5f396937c09499113a2e5299e871efb04195dabfc96
Status: Downloaded newer image for ruby@sha256:f7cb2fda8271b272f9adb5f396937c09499113a2e5299e871efb04195dabfc96

Successfully pulled ruby@sha256:f7cb2fda8271b272f9adb5f396937c09499113a2e5299e871efb04195dabfc96.

running ./flight-school/ci/test.sh
+ pushd flight-school
/tmp/build/e55deab7/flight-school /tmp/build/e55deab7
+ bundle install
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/.
Installing public_suffix 2.0.5
Installing backports 3.7.0
Installing safe_yaml 1.0.4
Installing diff-lcs 1.3
Installing hashdiff 0.3.2
Installing multi_json 1.12.1
Installing rack 1.6.5
Installing rspec-support 3.5.0
Installing tilt 2.0.7
Using bundler 1.14.6
Installing addressable 2.5.1
Installing crack 0.4.3
Installing rack-protection 1.5.3
Installing rack-test 0.6.3
Installing rspec-core 3.5.4
Installing rspec-expectations 3.5.0
Installing rspec-mocks 3.5.0
Installing webmock 2.3.2
Installing sinatra 1.4.8
Installing rspec 3.5.0
Installing sinatra-contrib 1.4.7
Bundle complete! 5 Gemfile dependencies, 21 gems now installed.
Bundled gems are installed into /usr/local/bundle.
+ bundle exec rspec

Randomized with seed 17037
.........

Finished in 1.01 seconds (files took 0.35998 seconds to load)
9 examples, 0 failures

Randomized with seed 17037

+ popd
/tmp/build/e55deab7
succeeded

Woohoo! We've run our unit tests inside Concourse. Now is a good time to commit and push.

In general, try and think in terms of small reusable tasks that perform a simple action with the inputs that they're given. If a task ends up having too many inputs then it may be a smell that your task is doing too much. Similar to if a function in a program you were writing had a lot of parameters. In fact, that's a good way to think about tasks: they're functions that take inputs as parameters. Keeping them small and simple allows you to easily run them from your local machine as above.

Please excuse the long-winded iterative process we used to get to the final result. You'll end up writing enough tasks that you can skip directly to the end. We felt it was important to go through all the possible hurdles you may encounter on your journey.

Starting a Pipeline

Ok, so. We have a task we can run. How about we run that every time the code changes so that we can check to see when anything breaks. Enter: pipelines.

Pipelines are built up from resources and jobs. Resources are external, versioned things such as Git repositories or S3 buckets and jobs are a grouping of resources and tasks that actually do the work in the system.

resources:
- name: flight-school
  type: git
  source:
    uri: https://github.com/(your username)/flight-school
    branch: master

jobs:
- name: test-app
  plan:
  - get: flight-school
  - task: tests
    file: flight-school/build.yml

Uploading that:

fly -t ci set-pipeline -p flight-school -c ci/pipeline.yml
...
pipeline created!
you can view your pipeline here: https://(your concourse url)/pipelines/flight-school

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command
  - click play next to the pipeline in the web ui

Follow the instructions to unpause the pipeline.

Click the job. Then click run.

It runs!

Running Continuously

Add trigger: true to the get. Any new versions will trigger the job.

jobs:
- name: test-app
  plan:
  - get: flight-school
    trigger: true
  - task: tests
    file: flight-school/build.yml

Try pushing a commit to the repository. For extra credit push a commit that breaks the build and then another one that fixes it again.

Extending the Pipeline

What you've seen so far is the very essence of the way pipelines work in Concourse: a new version of a resource appears and then a job runs to do something with that new version. This is the very start of a pipeline. Let's keep adding to it and have it do more things for us.

Deploying your Application

We've shown ourselves that we have a working application. Let's show the rest of the world by deploying the application to a PaaS (Platform as a Service).

The author of this walkthrough is most familiar with Pivotal Web Services (PWS). So that's what this guide is going to use. Feel free to use whichever deployment environment you'd like to use (such as Heroku, etc.). The mechanics and placement in the pipeline will be similar for all of them.

Follow this guide to get yourself set up with a PWS account and push some sample applications. There's a 60 day free trial so don't worry about this costing you anything.

Now, that you're familiar with how to deploy an application to PWS, let's have the pipeline do all the hard work for us. First, let's add a resource for the pipeline to interact with.

- name: staging-app
  type: cf
  source:
    api: https://api.run.pivotal.io
    username: ((your cf username))
    password: ((your cf password))
    organization: ((your cf organization))
    space: ((your cf space))

Then, in our job, we can push our application.

- name: test-app
  plan:
  - get: flight-school
    trigger: true
  - task: tests
    file: flight-school/build.yml
  - put: staging-app
    params:
      manifest: flight-school/manifest.yml

Now, if we run the job again we'll see that after the tests have passed that Concourse deploys our application for the world to see. Try pushing some commits to the repository and seeing that they are automatically tested and pushed. (For example, try adding or removing a U.S. airport code from the list).

Adding and Integrating Second Component

Coming soon!

Next Steps

By now you have a good grasp on the practical side of using Concourse. For a more in-depth explanation of how things work you should read the two documents on the Pipelines and Anatomy of a running task.

The rest of the documentation on the site provides reference material for Concourse. If you'd like to find out more about what attributes you can attach to a step in a plan or all the options that you can pass to fly then the links on the left hand side are the best place to start.

And, as always, if you can't find what you're looking for in the documentation then we're around on Slack during work hours and should be able to help you if you have further questions.