# AGENTS.md: Apache Airflow

## Commands
- Run one DAG locally without the scheduler: `airflow dags test <dag_id> <logical_date>`; one task: `airflow tasks test <dag_id> <task_id> <logical_date>`.
- Tests: `pytest tests/` including a DAG import test (every file in `dags/` loads with no import errors and no cycles).
- Check the Airflow version in requirements/Dockerfile before writing imports: Airflow 3 uses `airflow.sdk` (`from airflow.sdk import dag, task`); Airflow 2 uses `airflow.decorators`. Follow what the repo already uses.

## DAG files
- DAG files are parsed constantly: no database queries, API calls, heavy imports or `Variable.get()` at module top level. Do that work inside tasks or with Jinja templates.
- Define DAGs with the TaskFlow API (`@dag`, `@task`) and explicit `dag_id`, `schedule`, `start_date` (static, timezone-aware), `catchup` (set it explicitly), `max_active_runs`, `tags` and `default_args` with `retries` and `retry_delay`.
- One DAG per file named after the `dag_id`. Shared code goes in a package under `include/` or `plugins/`, not copied between DAGs.

## Tasks
- Tasks are idempotent and deterministic for a given logical date or data interval: write to partitions keyed by the interval, use upserts or overwrite-partition, never "append everything since now".
- Use the data interval (`data_interval_start`/`data_interval_end`) for time windows, never `datetime.now()`.
- XCom carries small metadata (paths, IDs, counts), never datasets or dataframes. Large data goes to object storage or the warehouse.
- Push heavy compute to the system built for it (warehouse SQL, Spark, containers via `KubernetesPodOperator`); workers orchestrate.
- Use provider operators and hooks with connection IDs; credentials live in Airflow Connections or a secrets backend, never in code or Variables in plain text.
- Set `execution_timeout` on tasks that call external systems; use sensors in deferrable/reschedule mode instead of blocking a worker slot.
- Use assets/datasets for cross-DAG dependencies rather than sensors polling other DAGs, when the version supports it.

## Done when
DAG import test passes, `airflow dags test` runs the changed DAG for one interval, and reruns of the same interval produce the same result.