Tasks

The smallest configurable unit in a Concourse pipeline is a single task. A task can be thought of as a function from inputs to outputs that can either succeed or fail.

Going a bit further, ideally tasks are pure functions: given the same set of inputs, it should either always succeed with the same outputs or always fail. This is entirely up to your script's level of discipline, however. Flaky tests or dependencies on the internet are the most common source of impurity.

Once you have a running Concourse deployment, you can start configuring your tasks and executing them interactively from your terminal with the Fly commandline tool.

Once you've figured out your tasks's configuration, you can reuse it for a Job in your Pipeline.

Configuring a Task

Conventionally a task's configuration is placed in the same repository as the code it's testing, possibly under some ci directory. For a simple Ruby app with unit tests it may be called ci/unit.yml, and look something like:

task.yml

---
platform: linux

image_resource:
  type: docker-image
  source:
    repository: ruby
    tag: '2.1'

inputs:
- name: my-app

run:
  path: my-app/scripts/test

This configuration specifies that the task must run with the ruby:2.1 Docker image with a my-app input, and when the task is executed it will run the scripts/test script in the same repo.

A task's configuration specifies the following:

Required. The platform the task should run on. By convention, windows, linux, or darwin are specified. This determines the pool of workers that the task can run against. The base deployment provides Linux workers.

Optional. The base image of the container. This style of specifying the base image has the same effect as image: above but uses Concourse resources to download the image. The contents of this field should be the same as a resource configuration in your pipeline (without the name).

The following example configures the task to use the golang:1.6 Docker image:

image_resource:
  type: docker-image
  source: {repository: golang, tag: "1.6"}

...and the following example uses an insecure private Docker registry with a username and password:

image_resource:
  type: docker-image
  source:
    repository: my.local.registry:8080/my/image
    insecure_registries: ["my.local.registry:8080"]
    username: myuser
    password: mypass
    email: x@x.com

You can use any resource that returns a filesystem in the correct format (a /rootfs directory and a metadata.json file in the top level) but normally this will be the Docker Image resource. If you'd like to make a resource of your own that supports this please use that as a reference implementation for now.

If you want to use an artifact source within the plan containing an image, you must set the image in the plan step instead.

Optional. A string specifying the rootfs of the container, as interpreted by your worker's Garden backend.

You should only use this if you cannot use image_resource for some reason, and you know what you're doing.

Required. The expected set of inputs for the task.

This determines which artifacts will propagate into the task, as the build plan executes. If any specified inputs are not present, the task will end with an error, without running.

Each input has the following attributes:

Required. The logical name of the input.

Optional. The path where the input will be placed. If not specified, the input's name is used.

Optional. The artifacts produced by the task.

Each output configures a directory to make available to later steps in the build plan. The directory will be automatically created before the task runs, and the task should place any artifacts it wants to export in the directory.

Each output has the following attributes:

Required. The logical name of the output. The contents under path will be made available to the rest of the plan under this name.

Optional. The path to a directory where the output will be taken from. If not specified, the output's name is used.

Note that this value must not overlap with any other inputs or outputs. Each output results in a new empty directory that your task should place artifacts in; if the path overlaps it'll clobber whatever files used to be there.

For example, the following task and script would be used to propagate a built binary to later steps:

---
platform: linux

image_resource: # ...

inputs:
- name: project-src

outputs:
- name: built-project

run:
  path: project-src/ci/build

...assuming project-src/ci/build looks something like:

#!/bin/bash

set -e -u -x

export GOPATH=$PWD/project-src

go build -o built-project/my-project github.com/concourse/my-project

...this task could then be used in a build plan like so:

plan:
- get: project-src
- task: build-bin
  file: project-src/ci/build.yml
- put: project-bin
  params: file: built-project/my-project

Required. The command to execute in the container.

Required. The command to execute, relative to the task's working directory. For a script living in a resource's repo, you must specify the full path to the resource, i.e. my-resource/scripts/test.

Optional. Arguments to pass to the command. Note that when executed with Fly, any arguments passed to Fly are appended to this array.

Optional. A directory, relative to the initial working directory, to set as the working directory when running the script.

Optional. Explicitly set the user to run as. If not specified, this defaults to the user configured by the task's image. If not specified there, it's up to the Garden backend, and may be e.g. root on Linux.

Note that this is not provided as a script blob, but explicit path and args values; this allows fly to forward arguments to the script, and forces your config .yml to stay fairly small.

Optional. A key-value mapping of values that are exposed to the task via environment variables.

Use this to provide things like credentials, not to set up the task's Bash environment (they do not support interpolation).

Anatomy of a running task

A task runs in a new container every time, using the image provided by image_resource as its base filesystem (i.e. /).

The command specified by run will be executed in a working directory containing each of the inputs. If any inputs are missing the task will not run (and the container will not even be created).

The working directory will also contain empty directories for each of the outputs. The task must place artifacts in the output directories for them to be exported. This meshes well with build tools with configurable destination paths.

If your build tools don't support output paths you'll have to copy bits around. If it's a git repo that you're modifying you can do a local git clone ./input ./output, which is much more efficient than cp, and then work out of ./output.

Any params configured will be set in the environment for the task's command, along with any environment variables provided by the task's image (i.e. ENV rules from your Dockerfile).

The user the command runs as is determined by the image. If you're using the Docker Image resource, this will be the user set by a USER rule in your Dockerfile, or root if not specified.

Another relevant bit of configuration is privileged, which determines whether the user the task runs as will have full privileges (primarily when running as root). This is intentionally not configurable by the task itself, to prevent privilege escalation by way of pull requests to repositories containing task configs.

Putting all this together, the following task config:

---
platform: linux

image_resource:
  type: docker-image
  source:
    repository: golang
    tag: '1.6'

params:
  SOME_PARAM: some-default-value

inputs:
- name: some-input
- name: some-input-with-custom-path
  path: some/custom/path

outputs:
- name: some-output

run:
  path: sh
  args:
  - -exc
  - |
    whoami
    env
    go version
    find .
    touch some-output/my-built-artifact

...will produce the following output:

+ whoami
root
+ env
USER=root
HOME=/root
GOLANG_DOWNLOAD_SHA256=5470eac05d273c74ff8bac7bef5bad0b5abbd1c4052efbdbc8db45332e836b0b
PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GOPATH=/go
PWD=/tmp/build/e55deab7
GOLANG_DOWNLOAD_URL=https://golang.org/dl/go1.6.linux-amd64.tar.gz
GOLANG_VERSION=1.6
SOME_PARAM=some-default-value
+ go version
go version go1.6 linux/amd64
+ find .
.
./some-input
./some-input/foo
./some
./some/custom
./some/custom/path
./some/custom/path/bar
./some-output
+ touch some-output/my-built-artifact

...and propagate my-built-artifact to any later tasks or puts that reference the some-output artifact, in the same way that this task had some-input as an input.

Running tasks with fly

Fly is a command-line tool that can be used to execute a task configuration against a Concourse deployment. This provides a fast feedback loop for iterating on the task configuration and your code.

For more information, see execute.