Open Skills Logo
Open Skills

airflow-dag-patterns

Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

View Source
name:airflow-dag-patternsdescription:Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

Use this skill when

  • Creating data pipeline orchestration with Airflow

  • Designing DAG structures and dependencies

  • Implementing custom operators and sensors

  • Testing Airflow DAGs locally

  • Setting up Airflow in production

  • Debugging failed DAG runs

    • Do not use this skill when

  • You only need a simple cron job or shell script

  • Airflow is not part of the tooling stack

  • The task is unrelated to workflow orchestration

    • Instructions

  • Identify data sources, schedules, and dependencies.

  • Design idempotent tasks with clear ownership and retries.

  • Implement DAGs with observability and alerting hooks.

  • Validate in staging and document operational runbooks.

    • Refer to resources/implementation-playbook.md for detailed patterns, checklists, and templates.

    Safety

  • Avoid changing production DAG schedules without approval.

  • Test backfills and retries carefully to prevent data duplication.

    • Resources

  • resources/implementation-playbook.md for detailed patterns, checklists, and templates.