| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
AIP-40: Add Deferrable "Async" Tasks (#15389) This is the implementation of AIP-40, Deferrable "Async" Tasks (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929). The main changes are: - A new concept of a Trigger is introduced, as a small piece of asyncio code that can fire off events - There is a BaseTrigger and some time-related triggers under a new airflow.triggers package - There is a new Trigger database model and associated trigger table - Async versions of the various date/time sensors have been added which defer rather than poke. - There is a new persistent process (Job) called triggerer - It only runs on Python 3.7+ - It handles polling the database for which triggers need running, running them, and re-scheduling task instances whose triggers have fired events - If a trigger throws an exception or exits without firing an event, it logs why and marks dependent task instances as failed - It monitors the asyncio event loop with a watchdog task and alerts the user if anything is overrunning (i.e. not using await) and blocking the loop. - It is designed to run in parallel with itself in a highly-available manner, and also has built-in consistent-hash based partitioning (sharding) support - Task Instances have a new deferred state which indicates they are waiting on a trigger to run - The trigger they are waiting for is stored in a new trigger_id column, and a failure timeout is in a trigger_timeout column - The scheduler takes care of timing out task instances into the failed state - Deferral is triggered by raising the TaskDeferred exception, or calling self.defer on the TaskInstance which does the same thing. - A next_method and next_kwargs column are added to specify what a task instance/operator's execution entry point should be if it's not the default of execute(). They are currently only used by deferral, but have been written to be independent in case they are useful elsewhere. - Two new dependencies are added - jump-consistent-hash is a small MIT licensed library that implements a fast, consistent hash algorithm - pytest-asyncio is an Apache 2 licensed library that enables async tests to be written easily Changes that are deliberately not in here and will be in a future PR for them specifically: - UI warning when the triggerer is not running and you have deferred task instances - Updating Breeze to include triggerer in what it runs - Updating the Docker Compose files to include triggerer - Updating the Helm Chart to include triggerer - Some way of detecting/preventing DB access within triggers | 4 年前 | |
| 3 年前 | ||
Add triggers for ExternalTask (#29313) Contributes back two of the core Triggers from https://github.com/astronomer/astronomer-providers so that it can be used to create an operator /sensor or used within taskflow API | 3 年前 | |
Add FileTrigger (#29265) Contributes back one of the core Triggers from https://github.com/astronomer/astronomer-providers so that it can be used to create an operator /sensor or used within taskflow API | 3 年前 | |
convert moment with timezone to UTC instead of raising an exception (#29606) * convert moments with timezone to utc instead of raising an exception * test DateTimeTrigger with different timezones | 3 年前 | |
Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290) This PR applies PEP-563 to all core airflow Python files - i.e. those that we want to cherry-pick to v2-* branches in order to make cherry-picking for the upcoming 2.4* releases easier. There was a separate PR (#26289) with non-core changes that are not going to be cherry-picked.. This PR is a result of combining some of the 47 PRs reviewed and approved separately (otherwise it would have been unreviewable) The history of those PRs can be changed in: https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Afuture-annotations+is%3Aopen Relevant discussion: https://lists.apache.org/thread/81fr042s5d3v17v83bpo24tnrr2pp0fp Lazy consensus call: https://lists.apache.org/thread/l74nvjh8tgbtojllhwkcn7f8mfnlz4jq | 3 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 4 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 |