Branch Deployments in Dagster+#

Dagster+ provides out-of-the-box support for Continuous Integration (CI) with Branch Deployments.

Branch Deployments automatically create staging environments of your Dagster code, right in Dagster+. For every push to a branch in your git repository, Dagster+ will create a unique deployment, allowing you to preview the changes in the branch in real-time.

Understanding Branch Deployments#

• Overview
• Benefits
• Requirements
• Supported platforms
• Limitations
• Output handling
• Change Tracking

Overview#

Think of a branch deployment as a branch of your data platform, one where you can preview changes without impacting production or overwriting a testing environment.

Let's take a closer look:

1. In your git repository, a new branch is created off of main. In the example above, this branch is named feature-1.

2. Dagster+ is notified of the push and creates a branch deployment named feature-1. The branch deployment functions just like your production deployment of Dagster+, but contains the Dagster code changes from the feature-1 branch.

   In this example, the feature-1 branch deployment 'talks' to a cloned schema in a database. This is completely separate from the prod schema associated with the production deployment.

3. For every push to the feature-1 branch, the feature-1 branch deployment in Dagster+ is rebuilt and redeployed.

Benefits#

Now that you know how Branch Deployments work, why should you use them?

• Improved collaboration. Branch Deployments make it easy for everyone on your team to stay in the loop on the latest Dagster changes.
• Reduced development cycle. Quickly test and iterate on your changes without impacting production or overwriting a testing environment.

Requirements#

To use Branch Deployments, you'll need a Dagster+ account.

Supported platforms#

Branch Deployments can be used with any git or CI provider. However, setup is easiest with the Dagster GitHub app or Dagster Gitlab app as parts of the process are automated. Refer to the Setting up Branch Deployments section for more info.

Limitations#

Branch Deployments aren't currently supported for use in Dagster Open Source.

Output handling#

Output created from a branch deployment - such as a database, table, etc. - won't be automatically removed from storage once a branch is merged or closed. Refer to the Best practices section for info on how to handle this.

Change Tracking#

When a Branch Deployment is deployed, it compares the asset definitions in the branch deployment with the asset definitions in the main deployment. The Dagster UI will then mark the changed assets, making it easy to identify changes. Refer to the Branch Deployments Change Tracking guide for more information and examples.

Managing Branch Deployments#

• Setting up Branch Deployments
• Accessing a branch deployment

Setting up Branch Deployments#

There are currently two ways to set up Branch Deployments for Dagster+. In the table below:

• Platform - The name of the git/CI platform, which is also a link to a setup guide
• How it works - Summary of how Branch Deployments work with the platform
• May be a good fit if... - A high-level summary of when the platform may be a good fit

Accessing a branch deployment#

Once configured, branch deployments can be accessed:

From a GitHub pull requestIn Dagster+

Every pull request in the repository contains a View in Cloud link:

Clicking the link will open a branch deployment - or a preview of the changes - in Dagster+.

Best practices#

To ensure the best experience when using Branch Deployments, we recommend:

• Configuring jobs based on environment. Dagster automatically sets environment variables containing deployment metadata, allowing you to parameterize jobs based on the executing environment. Use these variables in your jobs to configure things like connection credentials, databases, and so on. This practice will allow you to use Branch Deployments without impacting production data.

• Creating jobs to automate output cleanup. As Branch Deployments don't automatically remove the output they create, you may want to create an additional Dagster job to perform the cleanup.

Want some help with implementation? Check out the Testing against production with Dagster+ Branch Deployments guide for a step-by-step look at implementing these best practices in your data pipelines.