#data

Airflow

→ Trigger dags from external dag

This involves 3 dags:

1. the external dag to follow
2. the external dag to run
3. the trigger dag

the 2 formers are classical dags. both 1 and 3 MUST have the same schedule_interval, while the 2 should be @once. It is usefull to be able to run it manually.

trigger dag:

from airflow.sensors.external_task_sensor import ExternalTaskSensor
from airflow.operators.dagrun_operator import TriggerDagRunOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': airflow.utils.dates.days_ago(7),
    'email_on_failure': True,
    'email': [email]
}

dag = DAG(dagName, default_args=default_args, schedule_interval='0 21 * * *')

with dag:
    wait_for_parent = ExternalTaskSensor(
        task_id='wait_for_parent',
        external_dag_id='the-external-dag-id',
        external_task_id='the-external-dag-id.the-external-task-id'
    )
    
    run_task = TriggerDagRunOperator(
        task_id="run-task",
        trigger_dag_id="the-second-external-dag-id-to-run",
        execution_date=datetime.now()
    )
    
    wait_for_parent >> run_task

→ Kubernetes pod operator

• KPO airflow
• KPO astro
• KPO google

→ template vars reference

• vars can be used in template fields

→ debugging airflow

• debugging