Home › Apache Airflow › Apache Airflow DAGs AGENTS.md

Apache Airflow DAGs AGENTS.md

AGENTS.md for Apache Airflow projects: TaskFlow DAGs, fast parse-time code, idempotent tasks, explicit schedules and catchup, connections, small XComs, and DAG tests.

AGENTS.mdApache Airflow

Copy AGENTS.md

How to use

Save this as AGENTS.md in your repository root, commit it, and your agent will pick it up automatically. Edit the commands to match your project.

Note: Review and test before use. Provided as is, without warranty. See Terms.

# AGENTS.md: Apache Airflow

## Commands
- Run one DAG locally without the scheduler: `airflow dags test <dag_id> <logical_date>`; one task: `airflow tasks test <dag_id> <task_id> <logical_date>`.
- Tests: `pytest tests/` including a DAG import test (every file in `dags/` loads with no import errors and no cycles).
- Check the Airflow version in requirements/Dockerfile before writing imports: Airflow 3 uses `airflow.sdk` (`from airflow.sdk import dag, task`); Airflow 2 uses `airflow.decorators`. Follow what the repo already uses.

## DAG files
- DAG files are parsed constantly: no database queries, API calls, heavy imports or `Variable.get()` at module top level. Do that work inside tasks or with Jinja templates.
- Define DAGs with the TaskFlow API (`@dag`, `@task`) and explicit `dag_id`, `schedule`, `start_date` (static, timezone-aware), `catchup` (set it explicitly), `max_active_runs`, `tags` and `default_args` with `retries` and `retry_delay`.
- One DAG per file named after the `dag_id`. Shared code goes in a package under `include/` or `plugins/`, not copied between DAGs.

## Tasks
- Tasks are idempotent and deterministic for a given logical date or data interval: write to partitions keyed by the interval, use upserts or overwrite-partition, never "append everything since now".
- Use the data interval (`data_interval_start`/`data_interval_end`) for time windows, never `datetime.now()`.
- XCom carries small metadata (paths, IDs, counts), never datasets or dataframes. Large data goes to object storage or the warehouse.
- Push heavy compute to the system built for it (warehouse SQL, Spark, containers via `KubernetesPodOperator`); workers orchestrate.
- Use provider operators and hooks with connection IDs; credentials live in Airflow Connections or a secrets backend, never in code or Variables in plain text.
- Set `execution_timeout` on tasks that call external systems; use sensors in deferrable/reschedule mode instead of blocking a worker slot.
- Use assets/datasets for cross-DAG dependencies rather than sensors polling other DAGs, when the version supports it.

## Done when
DAG import test passes, `airflow dags test` runs the changed DAG for one interval, and reruns of the same interval produce the same result.