Database Schema

Pipeline/API Objects

The following tables manage objects exposed directly to users, either via the API or by configuring pipelines.

The pipelines table contains details about a pipeline; its name, which team it belongs to, whether it is paused, and whether it is public.

The jobs table tracks the details of all jobs in a Concourse deployment. Jobs reference the pipeline they belong to, their Job configuration, and their name in the pipeline.

There are also a number of attributes which are used by the runtime:

paused: boolean

The build scheduler does not schedule builds for jobs which are paused.

The build scheduler does not schedule builds for jobs which have not yet had their inputs_determined.

The build scheduler does not schedule builds for jobs that have had their build max_in_flight_reached.

interruptible: boolean

Workers trying to land will wait until all builds are finished, unless the build is for an interruptible job.

The builds table tracks the details of every run of every job or one-off build, including its name, the status of the build, whether the build has been scheduled or completed, and information about the start and end times of the build.

An entry in the resources table represents a resource defined in a pipeline. The resources table has columns for configuration, name, type, and details about resource version checking.

This table contains versions of resources discovered by checking. Versioned resources have attributes describing their version, metadata, resource type, and whether the version is enabled in the pipeline.

An entry in the resource types table represents a custom resource type defined in a pipleine, and all of the information neccessary to pull the image bits for a custom resource type. This includes the resource type used to fetch the resource type's image (eiher a base_resource_type or another custom resource) , the version of the image to fetch using that resource, and the configuration of the custom resource type.

Abstract Objects

Concourse's database manages the abstractions around resources, resource configuration, resource versioning, and resource types. These tables help make the Resources concept operate as expected, scale across many pipelines, and to many different types.

A resource config is a distinct configuration of a resource type, not specific to any pipeline resource.

A resource config is used for discovering versions. Unlike pipeline resources, they have no name, and are identified entirely by their type and source. Resource configs are shared across pipelines and teams, since identical resource configurations can be reused.

A resource config's type is either a base resource type or, in the case of a custom resource type, the resource cache of its custom type's image resource.

A resource cache represents all information necessary to fetch a particular set of bits, i.e. the cache.

A resource cache always points to its resource_configs, and also specifies the version and params.

Note that resource_configs and resource_caches both point to each other. This loop is closed by a resource config that points to a base resource type at the end of the chain. This represents the case of a resource cache provided by a custom resource type, which in turn may be provided by another custom resource type, and so on, until they reach a base type.

A cache use is a join table between a resource cache and a 'user', which lets us know when the use is no longer needed. Think of it as a reference counter.

A use can be tied to a builds, in which case the use can be removed when the build is finished. A cache is in use by a build when the build is fetching it as an input. This is to prevent the garbage collector from reaping caches that are no longer desired in the long run, but are still in use by a build.

A use can be tied to a containers, in which case the use can be removed when the container goes away. A cache is in use by a container when it's being used as the container's image, either via image_resource or a custom resource type.

A build image resource cache is a join table between a build and a resource cache.

A build image resource cache is used for keeping caches that were used for an image_resource in a build, as part of the resource retention policy.

A resource config check session is used for periodic resource checking of resources and resource_types by pointing to their resource config. There is only one check container per resource config, even if it's defined in many pipelines.

A check session's expiry is tied to its worker's uptime, as a simple way to force balancing of check containers across workers more quickly after a deploy.

A worker resource config check session is a join table tying a resource config check session to a worker base resource type. When either go away, the worker resource config check session goes away. This is so that containers using an old base resource type are reaped and recreated.

An entry in the base resource types table represents a resource type which can be provided by any worker. It is used to keep resource_configs as general as possible, and not tied to particular workers. This in turn allows resource_caches to also be general, as is desired for Concourse's higher-level pipeline resource caching semantics.

An entry in the worker base resource type table represents a location on a particular worker where the rootFS for a specific version of a base_resource_type is found. When a worker registers, its base_resource_types are synced by first removing old versions and then inserting new ones.

A worker resource cache is a join between a resource cache and a worker base resource type.

A worker resource cache is automatically removed when its worker base resource type or resource cache is removed. This is used to automatically invalidate any caches that were fetched with an old version of a base resource type.

Runtime

The containers table represents the set of all containers across all the workers, and keeps track of their state such that no container is ever left behind on a worker.

Containers have a handful of really important attributes:

handle: string

The unique identifier of the container in Garden.

worker_name: string

The name of the worker where the container is located.

state: CREATING, CREATED, or DESTROYING

The stage in the lifecycle of the container. See Lifecycle

Containers can be one of four types, each with individual references to their relating object.

This signifies that the container is dependant on another container which is busy checking for the image this container will be based on. This is used in the case of custom resource types, or tasks with image_resource. This container will be in the CREATING state until the image is fetched later.

This signifies that the container is dependant on another container which is busy downloading the bits for the image this container will be based on. This is used in the case of custom resource types, or tasks with image_resource. This container will be in the CREATING state until the image is fetched.

build_id: fkey

If this container is for a build step, this column refers to the builds (id) this container is related to.

If this container is for running the check script for a resource, this column refers to a worker_resource_config_check_sessions (id).

The volumes table represents the set of all volumes across all the workers, and keeps track of their state such that no volume is ever left behind on a worker.

Volumes have a handful of really important attributes.

handle: string

The unique identifier of the volume in BaggageClaim.

worker_name: string

The name of the worker where the volume located.

state: CREATING, CREATED, or DESTROYING

The stage in the lifecycle of the volume. See Volumes.

container_id: fkey

The container this volume is associated with.

If this volume is used for a worker base resource type this column points to it.

If this volume is for a worker resource cache this column points to it.

If this volume is for a worker task cache this column points to it.

The workers table represents all of the workers that have registered with the Concourse cluster.

name: string

The unique name for the worker used to identify it in the cluster.

addr: string

The full address of the Garden server running for the worker.

state: RUNNING, STALLED, LANDING, LANDED, RETIRING

The current state of the worker. This keeps track of the worker's lifecycle in order to handle the lifecycle of workers and their ability to have new containers or volumes created on them.

The full address of the worker's baggageclaim server; used to store volumes on the worker.

The worker task caches table tracks all of the details of a task cache for a specific worker. Worker task caches are created for any task in a job for all of the paths specified task cache.

This table is used to identify worker task cache volumes by their id.