Skip to content

Running a Pipeline

To run a data ELT pipeline:

  1. Unpause the DAG for that pipeline.

    Note: Unpausing a scheduled DAG for the first time will trigger a run. To manually trigger a run, click the icon in the Actions column or near the upper righthand corner in subsequent DAG views.

    Unpause a DAG

  2. Click that DAG's name to enter its grid view.

    Grid View

  3. Click Graph to enter the DAG's graph view.

    High level DAG graph view

  4. If all tasks end successfully, the Status indicator will switch from running (light green) to success (dark green).

    a DAG after a successful run

Now that you have some data in your warehouse, you can explore that data in Superset.

Troubleshooting

Occassionally a task will fail. To investigate

  1. Click on the failed DAG run (will be red in the bar chart on the left) and click the Graph button in that DAG run's details.

    Select a failed DAG run and enter its graph view

  2. Identify and click the task that failed (it will have a red outline and the tooltip will show Status: failed).

    Click the failed task

  3. Click the Log button in the failed task instance's detail pop-up.

    Enter the Log view

  4. Review the output in the logs to see the error output. You'll probably have to debug the issue or raise the issue upstream before the DAG will run successfully.

    Unpause a DAG

Resources

See the Airflow UI documentation for more information about available views and interfaces.