set-pipeline: Configuring Pipelines

To submit a pipeline configuration to Concourse from a file on your local disk you can use the -c or --config flag, like so:

$ fly -t example set-pipeline --config pipeline.yml --pipeline my-pipeline

This will present a diff of the changes and ask you to confirm the changes. If you accept then Concourse's pipeline configuration will switch to the pipeline definition in the YAML file specified.

Using {{parameters}}

The pipeline configuration can contain template variable in the form of {{foo-bar}}. They will be replaced with string values populated by repeated --var or --load-vars-from flags.

This allows for credentials to be extracted from a pipeline config, making it safe to check in to a public repository or pass around.

For example, if you have a pipeline.yml as follows:

resources:
- name: private-repo
  type: git
  source:
    uri: git@...
    branch: master
    private_key: {{private-repo-key}}

...you could then configure this pipeline like so:

$ fly -t example set-pipeline --pipeline my-pipeline --config pipeline.yml --var "private-repo-key=$(cat id_rsa)"

Or, if you had a credentials.yml as follows:

private-repo-key: |
  -----BEGIN RSA PRIVATE KEY-----
  ...
  -----END RSA PRIVATE KEY-----

...you could configure it like so:

$ fly -t example set-pipeline --pipeline my-pipeline --config pipeline.yml --load-vars-from credentials.yml

If both --var and --load-vars-from are specified, the --var flags take precedence.

Note that only strings are supported for these values, so that interpolating them into the template is trivial. Other types of values, and features like concatenation, are not supported.

Combining with get-pipeline

You can use process substitution to quickly replicate a pipeline by composing get-pipeline and set-pipeline together like so:

$ fly -t new set-pipeline -c <(fly -t old get-pipeline -p foo) -p foo

While this replication method may be useful for migrating pipelines between concourse instances, note the following caveats:

get-pipeline fetches the current pipeline configuration with variables already interpolated. So you are working around any previous parameterization.

Replicating a pipeline, unmodified, risks resource contention. For example, you could end up with two pipelines pushing to the same S3 bucket.

If you're starting a new pipeline, we recommend you try to work through the above examples before using this one-step clone.