apache/airflow
apache/airflow
Activity
Last release
Open issues
Open PRs
License
release notes
release notes
Published 8/16/2024
MinorContains breaking changesPreviously, when a DAG is paused or removed, incoming dataset events would still trigger it, and the DAG would run when it is unpaused or added back in a DAG file. This has been changed; a DAG's dataset schedule can now only be satisfied by events that occur when the DAG is active. While this is a breaking change, the previous behavior is considered a bug.
The behavior of time-based scheduling is unchanged, including the timetable part
of DatasetOrTimeSchedule.
try_number is no longer incremented during task execution (#39336)Previously, the try number (try_number) was incremented at the beginning of task execution on the worker. This was problematic for many reasons.
For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in
the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion.
Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again.
One consequence of this change is, if users were "manually" running tasks (e.g. by calling ti.run() directly, or command line airflow tasks run),
try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change.
/logout endpoint in FAB Auth Manager is now CSRF protected (#40145)The /logout endpoint's method in FAB Auth Manager has been changed from GET to POST in all existing
AuthViews (AuthDBView, AuthLDAPView, AuthOAuthView, AuthOIDView, AuthRemoteUserView), and
now includes CSRF protection to enhance security and prevent unauthorized logouts.
This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints.
([@skip](https://github.com/skip)_if, [@run](https://github.com/run)_if) to make it simple to apply whether or not to skip a Task. (#41116)This feature adds a decorator to make it simple to skip a Task.
Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured
to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for
more details. Note: This feature is still experimental. See documentation on Executor <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#using-multiple-executors-concurrently>_ for a more detailed description.
Airflow integrates Scarf to collect basic usage data during operation. Deployments can opt-out of data collection by setting the [usage_data_collection]enabled option to False, or the SCARF_ANALYTICS=false environment variable.
See FAQ on this <https://airflow.apache.org/docs/apache-airflow/stable/faq.html#does-airflow-collect-any-telemetry-data>_ for more information.
accessors to read dataset events defined as inlet (#39367)dag test (#40010)endDate in task instance tooltip. (#39547)accessors to read dataset events defined as inlet (#39367, #39893)run_if & skip_if decorators (#41116)renderedjson component (#40964)get_extra_dejson method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811)__getattr__ to task decorator stub (#39425)RemovedIn20Warning in airflow task command (#39244)db migrate error messages (#39268)suppress_and_warn warning (#39263)declarative_base from sqlalchemy.orm instead of sqlalchemy.ext.declarative (#39134)on_task_instance_failed access to the error that caused the failure (#38155)output_processor parameter to BashProcessor (#40843)never_fail in BaseSensor (#40915)start_date (#40878)external_task_group_id to WorkflowTrigger (#39617)BaseSensorOperator introduce skip_policy parameter (#40924)__init__ (#41086)OTel Traces (#40874)pydocstyle rules to pyproject.toml (#40569)pydocstyle rule D213 in ruff. (#40448, #40464)Dag.test() to run with an executor if desired (#40205)AirflowInternalRuntimeError for raise non catchable errors (#38778)pytest to 8.0+ (#39450)back_populates between DagScheduleDatasetReference.dag and DagModel.schedule_dataset_references (#39392)B028 (no-explicit-stacklevel) in core (#39123)ImportError to ParseImportError for avoid shadowing with builtin exception (#39116)SubDagOperator examples warnings (#39057)model_dump instead of dict for serialize Pydantic V2 model (#38933)ws from 7.5.5 to 7.5.10 in /airflow/www (#40288)filesystems and dataset-uris to "how to create your own provider" page (#40801)otel_on to True in example airflow.cfg (#40712)task_id from send_email to send_email_notification in taskflow.rst (#41060)release notes
Published 8/16/2024
MinorContains breaking changesPreviously, when a DAG is paused or removed, incoming dataset events would still trigger it, and the DAG would run when it is unpaused or added back in a DAG file. This has been changed; a DAG's dataset schedule can now only be satisfied by events that occur when the DAG is active. While this is a breaking change, the previous behavior is considered a bug.
The behavior of time-based scheduling is unchanged, including the timetable part
of DatasetOrTimeSchedule.
try_number is no longer incremented during task execution (#39336)Previously, the try number (try_number) was incremented at the beginning of task execution on the worker. This was problematic for many reasons.
For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in
the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion.
Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again.
One consequence of this change is, if users were "manually" running tasks (e.g. by calling ti.run() directly, or command line airflow tasks run),
try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change.
/logout endpoint in FAB Auth Manager is now CSRF protected (#40145)The /logout endpoint's method in FAB Auth Manager has been changed from GET to POST in all existing
AuthViews (AuthDBView, AuthLDAPView, AuthOAuthView, AuthOIDView, AuthRemoteUserView), and
now includes CSRF protection to enhance security and prevent unauthorized logouts.
This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints.
([@skip](https://github.com/skip)_if, [@run](https://github.com/run)_if) to make it simple to apply whether or not to skip a Task. (#41116)This feature adds a decorator to make it simple to skip a Task.
Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured
to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for
more details. Note: This feature is still experimental. See documentation on Executor <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#using-multiple-executors-concurrently>_ for a more detailed description.
Airflow integrates Scarf to collect basic usage data during operation. Deployments can opt-out of data collection by setting the [usage_data_collection]enabled option to False, or the SCARF_ANALYTICS=false environment variable.
See FAQ on this <https://airflow.apache.org/docs/apache-airflow/stable/faq.html#does-airflow-collect-any-telemetry-data>_ for more information.
accessors to read dataset events defined as inlet (#39367)dag test (#40010)endDate in task instance tooltip. (#39547)accessors to read dataset events defined as inlet (#39367, #39893)run_if & skip_if decorators (#41116)renderedjson component (#40964)get_extra_dejson method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811)__getattr__ to task decorator stub (#39425)RemovedIn20Warning in airflow task command (#39244)db migrate error messages (#39268)suppress_and_warn warning (#39263)declarative_base from sqlalchemy.orm instead of sqlalchemy.ext.declarative (#39134)on_task_instance_failed access to the error that caused the failure (#38155)output_processor parameter to BashProcessor (#40843)never_fail in BaseSensor (#40915)start_date (#40878)external_task_group_id to WorkflowTrigger (#39617)BaseSensorOperator introduce skip_policy parameter (#40924)__init__ (#41086)OTel Traces (#40874)pydocstyle rules to pyproject.toml (#40569)pydocstyle rule D213 in ruff. (#40448, #40464)Dag.test() to run with an executor if desired (#40205)AirflowInternalRuntimeError for raise non catchable errors (#38778)pytest to 8.0+ (#39450)back_populates between DagScheduleDatasetReference.dag and DagModel.schedule_dataset_references (#39392)B028 (no-explicit-stacklevel) in core (#39123)ImportError to ParseImportError for avoid shadowing with builtin exception (#39116)SubDagOperator examples warnings (#39057)model_dump instead of dict for serialize Pydantic V2 model (#38933)ws from 7.5.5 to 7.5.10 in /airflow/www (#40288)filesystems and dataset-uris to "how to create your own provider" page (#40801)otel_on to True in example airflow.cfg (#40712)task_id from send_email to send_email_notification in taskflow.rst (#41060)Apache Airflow - A platform to programmatically author, schedule, and monitor workflows