Running Dagster locally#

In this guide, we'll walk you through how to run Dagster on your local machine using the dagster dev command.

Looking for installation help? Refer to the Dagster installation guide.

Understanding the dagster dev command#

The dagster dev command launches the Dagster webserver/UI and the Dagster daemon, allowing you to start a full deployment of Dagster from the command line.

This command should be run in a Python environment where the dagster and dagster-webserver packages are installed. Once started, the process should be kept running.

Locating your code#

Before you can start developing, you need to tell Dagster how to find the Python code containing your assets and jobs. There are a few ways to do this, which are outlined in the tabs below.

Note: If using an example Dagster project, or if you used the dagster CLI to create a project, you can run the dagster dev command in the same folder as the project to load the project code.

Dagster can load a file directly as a code location. In the following example, we used the -f argument to supply the name of the file:


dagster dev -f my_file.py

This command loads the definitions in my_file.py as a code location in the current Python environment.

You can also include multiple files at a time, where each file will be loaded as a code location:


dagster dev -f my_file.py -f my_second_file.py

Dagster can also load Python modules as code locations. When this approach is used, Dagster loads the definitions defined in the module passed to the command line.

We recommend defining a variable containing the Definitions object in a submodule named definitions inside the Python module. In practice, the submodule can be created by adding a file named definitions.py at the root level of the Python module.

As this style of development eliminates an entire class of Python import errors, we strongly recommend it for Dagster projects deployed to production.

In the following example, we used the -m argument to supply the name of the module and where to find the definitions:


dagster dev -m your_module_name.definitions

This command loads the definitions in the variable containing the Definitions object in the definitions submodule in the current Python environment.

You can also include multiple modules at a time, where each module will be loaded as a code location:


dagster dev -m your_module_name.definitions -m your_second_module.definitions

To load definitions without supplying command line arguments, you can use the pyproject.toml file. This file, included in all Dagster example projects, contains a tool.dagster section with a module_name variable:


[tool.dagster]
module_name = "your_module_name.definitions"  ## name of project's Python module and where to find the definitions
code_location_name = "your_code_location_name"  ## optional, name of code location to display in the Dagster UI

When defined, you can run this in the same directory as the pyproject.toml file:


dagster dev

Instead of this:


dagster dev -m your_module_name.definitions

You can also include multiple modules at a time using the pyproject.toml file, where each module will be loaded as a code location:


[tool.dagster]
modules = [{ type = "module", name = "foo" }, { type = "module", name = "bar" }]

Asset and run storage#

When running dagster dev, you may see log output that looks like this:


Using temporary directory /Users/rhendricks/tmpqs_fk8_5 for storage.

This indicates that any runs or materialized assets created during your session won't be persisted once the session ends. This can be useful when using Dagster for temporary local development or testing, when you don't care about the results being persisted.

To designate a more permanent home for your runs and assets, you can set the DAGSTER_HOME environment variable to a folder on your filesystem. Dagster will then use the specified folder for storage on all subsequent runs of dagster dev.

Local instance#

You can optionally use a dagster.yaml file to configure your Dagster instance - for example, to configure run concurrency limits or specify that runs should be stored in a Postgres database instead of on the filesystem.

If the DAGSTER_HOME environment variable is set, dagster dev will look for a dagster.yaml file in the DAGSTER_HOME folder. If DAGSTER_HOME is not set, dagster dev will look for that file from the folder where the command was run.

For the full list of options that can be set in the dagster.yaml file, refer to the Dagster instance documentation.

Detecting when you're running in `dagster dev`#

You may want to detect whether you're running locally. For example, you might want schedules or sensors to start in the RUNNING state in production but not in your local test deployment.

dagster dev sets the environment variable DAGSTER_IS_DEV_CLI to 1. You can detect whether you're in a local dev environment by checking for the presence of that environment variable.

Moving to production#

dagster dev is primarily useful for running Dagster for local development and testing. It isn't suitable for the demands of most production deployments. Most importantly, dagster dev does not include authentication or web security. Additionally, in a production deployment, you might want to run multiple webserver replicas, have zero downtime continuous deployment of your code, or set up your Dagster daemon to automatically restart if it crashes.

For information about deploying Dagster in production, refer to the Open Source Deployment guides.

You are viewing an unreleased or outdated version of the documentation